LE certs REFER NO CRL. Sendmail won't run, as a result

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: new.transbay.net

I ran this command: certbot certonly --standalone --cert-name MAIL -d new.transbay.net

It produced this output: the usual files, cert chain fullchain privkey.
I COULD NOT GET ANYTHING TO WORK UNLESS I COPIED THE R3 CERTIFICATE INTO MY OFFERED PEM FILES.
Worse than that, I determined by trial and error that MY offered key had to be the concatenation of: fullchain R3 privkey, in that order.
My offered cert = fullchain + theircert, in that order. What a mess.

I suggest that LE was originally directed at HTTP, and that shows up in being less savvy in dealing with (the third party nonsense surrounding) SMTP.

The failure is demonstrated in the logs:
May 31 06:16:57 server sm-mta[614051]: STARTTLS: x509 cert verify: depth=0 /CN=new.transbay.net, state=0, reason=unable to get certificate CRL
May 31 06:16:57 server sm-mta[614051]: STARTTLS: x509 cert verify: depth=1 /C=US/O=Let's Encrypt/CN=R3, state=0, reason=unable to get certificate CRL
May 31 06:16:57 server sm-mta[614051]: STARTTLS: x509 cert verify: depth=2 /O=Digital Signature Trust Co./CN=DST Root CA X3, state=0, reason=unable to get certificate CRL
May 31 06:16:57 server sm-mta[614051]: STARTTLS=server, relay=somewhere , version=TLSv1.3, verify=NOT, cipher=TLS_AES_128_GCM_SHA256, bits=128/128

My web server is (include version): The operating system my web server runs on is (include version):
Apache/2.4.41, Sendmail Version 8.15.2 on Ubuntu 20.04 LTS

My hosting provider, if applicable, is: A2 hosting, but I run it

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): rarely

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
certbot 0.40.0

I think this is more about sendmail's ancientness than it is an SMTP thing. Exim and Postfix are pretty easy to configure with Let's Encrypt certificates.

Something like this should work just fine on Ubuntu Focal for SMTPS and STARTTLS:

DAEMON_OPTIONS(`Port=smtp, Name=MTA')dnl
DAEMON_OPTIONS(`Port=smtps, Name=TLSMTA, M=s')dnl
DAEMON_OPTIONS(`Port=submission, Name=MSA, M=Ea')dnl
...
define(`confCACERT_PATH', `/etc/ssl/certs')dnl
define(`confCACERT', `/etc/letsencrypt/live/example.com/chain.pem')dnl
define(`confSERVER_CERT', `/etc/letsencrypt/live/example.com/cert.pem')dnl
define(`confSERVER_KEY', `/etc/letsencrypt/live/example.com/privkey.pem')dnl

I would try and see whether the above configuration gets rid of the CRL errors, but I do not think these CRL errors are fatal or affect sendmail's functionality.

For, Let's Encrypt doesn't even provide an CRL endpoint for the new.transbay.net certificate; only OCSP is provided for revocation checking for end-user certificates.

For the other two certs, CRL checking should work fine in fact.

What would be worrying is if there was an SSL verification error below these CRL errors.

Again, I avoid using sendmail, but this looks like a successful connection to me. The verify=NOT means that "a client certificate was not requested by us and we didn't try to verify one".

Is there anything below those log messages indicating a connection failure? Or do you have further symptoms, like the email failing to be delivered?

1 Like

I am cursed that I never use anything "easy to use" easily. I run postifx, but the config is absurd and mail misdelivers anyway. I agree sendmail has cobwebs since as you say the CRL file is superseded. It would be the entire point, for a CRL to be automatically consulted, 'hiding' them makes no sense until they're end-of-lifed. All the online notes&c presume the CRL info is in the certificate.

I note in passing seeing this just now: don't know how old it is:
"Sectigo removes CRL support in newly issued certificates - The Trustico® Blog"
One might hope OCSP support would already be in sendmail. Maybe it is.

I looked in vain for a DontBlameSendmail control to make it not care. I am not totally certain that per se is the show stopper but the error messages do complain loudly, and I don't remember seeing "warning:".

======

A bit ago I had the issue of Dovecot not connecting to sendmail 587, timing out saying (appx) "nobody ever talked to me". Others had noticed that but in reading re: dovecot determined it 'ought' to work, so sure enough sendmail logs say "No AUTH mechanisms" and that was because the cert failed.

But even after sendmail starts with no announced SSL complaints, an attempt to send email issues the complaints about CRL.
I'm damned if I do and damned if I don't. The system bitches about self-served certificates, but w/o LE I'd have to pay, and that's just silly. But, the other shoe: sendmail-hostile? Ouch.

The setup you cited is universally cited - and the certs are simply busted. I have the same stuff (except renaming to spare typing.)

define(confCACERT', /etc/letsencrypt/archive/MAIL/CAC')dnl # <= EDIT cert
define(confSERVER_CERT', /etc/letsencrypt/archive/MAIL/CRT')dnl # <= EDIT
define(confSERVER_KEY', /etc/letsencrypt/archive/MAIL/KEY')dnl # <= EDIT
define(confCLIENT_CERT', /etc/letsencrypt/archive/MAIL/CRT')dnl # <= EDIT
define(confCLIENT_KEY', /etc/letsencrypt/archive/MAIL/KEY')dnl # <= EDIT

root@server:/etc/letsencrypt/archive# ls MAIL
./ ../ 0c 0ch 0fc 0k CAC CRT KEY R3 key.0fc.R3.0k

Who knows why I wander into the twilight zone, but has anyone else reported needing to "build their own keys"?
Maybe I /am/ the last homo sapiens to try sendmail+dovecot. argh.

Yeah, the connection is made but (it at least at first) seems the transaction is failed because sendmail can't find the CRL reference in the offered cert. I have dumped the cert and no CRL info is in it. The error line ends in "0" as in "rc = 0", but I monitor the mail and it never makes it in.

If you are willing to help, I only hate postfix because it doesn't work "as advertised", for me at least, I'm probably misconfiguring it but never got any meaningful help. I migrated from sendmail and yes it should have been slam-dunk trivial, but between my /etc/aliases, /etc/virtusers, /etc/virthosts files, I ended up have to overdefine the crap out of all my tables to get it all to work as well as it does and still about 3-4 users can't get anything at all unless I create "userX" and divert their mail to that user, where I then have a program that appends the input to the correct file. (!?!?)

I should send excerpts of the various tables and you can say "ya, don't use that and don't say this."

Are you sure that this reported failure is fatal?

It is not fatal. when all else fails, hack the code. But in this case, no need:

** X509_VERIFY_CB -- verify callback
**
** Parameters:
** ctx -- x509 context
**
** Returns:
** accept connection?
** currently: always yes.
*/

static int
x509_verify_cb(ok, ctx)
int ok;
X509_STORE_CTX ctx;
{
if (ok == 0)
{
if (LogLevel > 13)
tls_verify_log(ok, ctx, "x509"); // I only see the complaint because I use LogLevel 14 to debug our milter.
if (ctx->error == X509_V_ERR_UNABLE_TO_GET_CRL)
{
ctx->error = 0;
return 1; /
override it */
}
}
return ok;
}

but then it means the logs are not helping disclose what's going on, why we see NOT:
STARTTLS=server, relay=c-68-34-7-299.hsd1.mi.comcast.net [68.34.7.299], version=TLSv1.3, verify=NOT, cipher=TLS_AES_128_GCM_SHA256, bits=128/128

Is your sendmail process able to access the CRL at http://x1.c.lencr.org/ ?

Well, thanks for that, and I now have the file from them.

define(`confCRL,bactick/etc/ssl/eAuuqRbQ')dnl # <= EDIT

this is kind of blunt but should work. Will see.
Wait. What did I feed the thing?

=================================================

no help per se: but was it a valid test?

May 31 10:55:28 server sm-mta[616106]: STARTTLS=server, relay=c-68-34-7-299.hsd1.mi.comcast.net [68.34.7.299], version=TLSv1.3, verify=NOT, cipher=TLS_AES_128_GCM_SHA256, bits=128/128
May 31 10:55:28 server sm-mta[616106]: STARTTLS=server, cert-subject=, cert-issuer=, verifymsg=ok
May 31 10:55:28 server sm-mta[616106]: AUTH warning: no mechanisms

Hi Eric, fancy seeing you here!

When I do

openssl s_client -starttls smtp -servername new.transbay.net new.transbay.net:25

everything looks completely correct to me (with openssl verifying your certificate correctly), so I think you've managed to set this up properly.

Everyone keeps saying "the connection seems to be fine", you included, so it seems there is something salient about that (fact?) that my ignoring is interfering with fixing the problem. I just don't see what it is.

Here is the syndrome:

  1. LE certs AS ISSUED by LE GIVE ERRORS and sendmail won't use them.
  2. By trial and error based on various ssl error messages:
    first for sanity I renamed LE's files, as 0c 0fc 0k 0ch = cert fullchain privkey chain, and then determine that I have to MODIFY LE's keys:
    Beyond that, I obtained the R3 cert as the cert of the top-level issuer.
    KEY and CERT are what I tell sendmail to use, and I create them as follows (+ = concat) KEY = 0fc + R3 + 0k and CERT = 0c + 0fc.
  3. See the summary at end since I won't be able to build the example I intended {dammit}

Witness

define(confCACERT', /etc/letsencrypt/archive/MAIL/CAC')dnl # <= EDIT cert (by now I forget what it contains.)
define(confSERVER_CERT', /etc/letsencrypt/archive/MAIL/0c')dnl # <= EDIT
define(confSERVER_KEY', /etc/letsencrypt/archive/MAIL/0k')dnl # <= EDIT
define(confCLIENT_CERT', /etc/letsencrypt/archive/MAIL/0c')dnl # <= EDIT
define(confCLIENT_KEY', /etc/letsencrypt/archive/MAIL/0k')dnl # <= EDIT

This is everyon's favorite setup, cert key cert key. Using THOSE data as LE provided.

make sendmail.cf
service sendmail restart

service sendmail restart
May 31 23:08:42 server sm-mta[616090]: NOQUEUE: stopping daemon, reason=signal
May 31 23:08:44 server su: (to smmsp) root on none
May 31 23:08:44 server su: pam_unix(su:session): session opened for user smmsp by (uid=0)
May 31 23:08:44 server systemd: pam_unix(systemd-user:session): session opened for user smmsp by (uid=0)
May 31 23:08:44 server su: pam_unix(su:session): session closed for user smmsp
May 31 23:08:44 server sm-mta[620051]: starting daemon (8.15.2): SMTP+queueing@00:10:00
May 31 23:08:44 server sm-mta[620051]: STARTTLS=server, Diffie-Hellman init, key=2048 bit (/)
May 31 23:08:44 server sm-mta[620051]: STARTTLS=server, init=1
May 31 23:08:44 server sm-mta[620051]: started as: /usr/sbin/sendmail-mta -Am -L sm-mta -bd -q10m

Ok, so no SSL complaints on INIT. BUT:

mailer says:

An error occurred while sending mail. The mail server responded:
5.7.0 Authentication required.
Please ensure that you are using the correct identity to send and that the used authentication method is correct. Verify that you are allowed to send via this SMTP server with your current credentials from your current network.

sendmail says:

May 31 23:11:16 server sm-mta[620115]: STARTTLS: x509 cert verify: depth=0 /CN=new.transbay.net, state=0, reason=unable to get certificate CRL
May 31 23:11:16 server sm-mta[620115]: STARTTLS: x509 cert verify: depth=1 /C=US/O=Let's Encrypt/CN=R3, state=0, reason=unable to get certificate CRL
May 31 23:11:16 server sm-mta[620115]: STARTTLS: x509 cert verify: depth=2 /O=Digital Signature Trust Co./CN=DST Root CA X3, state=0, reason=unable to get certificate CRL
May 31 23:11:16 server sm-mta[620115]: STARTTLS=server, relay=c-68-34-7-247.hsd1.mi.comcast.net [68.34.7.247], version=TLSv1.3, verify=NOT, cipher=TLS_AES_128_GCM_SHA256, bits=128/128
May 31 23:11:16 server sm-mta[620115]: STARTTLS=server, cert-subject=, cert-issuer=, verifymsg=ok
May 31 23:11:16 server sm-mta[620115]: AUTH warning: no mechanisms

My sendmail milter requires the sender of the message being sent to be authenticated, but it only logs success, not failure {laughs: so I should correct this decade-old oversight}, so that's why it's not saying "this mail is outta here because the sender is a ringer".

====================================================================================================================

Even here I have an unexpected result: these same (offered key and cert values) are now giving different errors than they used to: following this exact test (offered values === LE values) previously the complaint (I forget by now ...) was about missing values (signer cert, I think), that originally had me create keys based on some comment I ran across a couple days ago. An then, "All I changed between then and now," said the userland creature, "was to add this line
define(confCRL',/etc/ssl/eAuuqRbQ')dnl # <= EDIT
to the MC file." where the eAu thing is " the CRL at http://x1.c.lencr.org/" suggested by bruncsak above. [It's not working, as I am using it.]

The error seems to have "stabilized" as that CRL error, and /supposedly/ sendmail does not care about the CRL.
The bottom line (the QED I was striving for) is that sendmail is bailing on providing AUTH, and now from "a cause yet to be determined". Since it is NOT(?) the missing CRL that is the issue, this whole thread is irrelevant. {laughs}

===

Handsome as ever, I see. Banzai, etc. :slight_smile:

I fear my profile picture is from 2006 or something. :slightly_smiling_face:

The server side certificate seems to be working correctly. What about the client side certificate? What is your SMTP/STARTTLS client?

(Totally unrelated: the [single] IP address of your name servers: 85.187.158.191 is unreachable from multiple networks in Europe)

I'll look like Mr. Magoo before you do. :slight_smile:

For the first time, I found a remark that may explain things, and it's worth repeating here:
The issue is "can't authenticate", but the /SSL crap seems all working./
What if in fact the issue is that the user themself did not properly log in? In order to send?
The remark was to check that SASL is not lying in behalf of the user; but I would have
expected a "bad password" mention i the logs and I'm tailing both the auth and mail logs.

Duh, that should be easy to verify from the logs.

I'm using Thunderbird to talk to Dovecot. Sending email from me to me on the same machine is encountering the errors. TB cannot send anything on my behalf. SFAIK the "client side certificate" is irrelevant (on the server) and from the client is whatever TB (Mozilla) uses internally; I never gave that any thought but they're not at fault. On the server I state them to be the same as the server's. [I think they are not used in a simple SMTP submission, after the user is authenticated. I don't use a 'personal certificate'.] The cert being used while the thing is failing are from LE but I suspect now the certs are not at fault.

(Totally unrelated: the [single] IP address of your name servers: 85.187.158.191 is unreachable from multiple networks in Europe)

My firewalls. I permit 80/443, but block everything from some places, all of the PRC and chunks of space from "massive hosting or provider cos", eg digitalocean, orange.es, OVH.fr/.ca, psychz, liquidweb, t-mobile.de, yada-endless etc. and in fact all rented space on microsoft: and all of Romania: this is from two decades of fighting spam and being pissed off that nobody takes the issue seriously (enough to admit that MSware is the root cause, and move to end it by abandoning msft-ware.)

I review the stats and retain high-traffic rules. Be assured, if I had a 2048 core machine I would read everything. While the milter is a CPU hog, I don't want to burden it with 99.9%-likely-to-be-spam inspections; it (firewall) may throw babies out with the bathwater, but I always unblock anywhere as needed at user request.

Providers don't care at all about spam anymore, it seems. To have accepted it as "inevitable" and "what can you do" is a serious mistake. We have spam because of bot-nets and we have bot-nets because MSware has more bugs than Dr. Pepper has bubbles quarter after quarter (see CERT.org) world without end. We can in fact purge the Internet of spam/crime if people would just stop using leaky software. And don't give me the "but unix is attacked too" defense, because unix is TEN THOUSAND TIMES more secure than Windows; again, CERT says so, not me. Get all MSware off the net, and spam/crime will fall to near-zero within three months. We ISPs had nearly eliminated spam by eary-mid-1998, and THEN arose the bot-nets, allowing spammers to hide beneath layers of otherwise respectable people. If you had been using Linux/Unix since then, good for you, nobody ever got a virus FROM you.

To brag uselessly, the US Government awarded me $37,500 damages from Robert Soloway (I chased him off 5 consecutive Chinese servers in 1998.) The Chinese at the time might answer a complaint, but their national policy now seems to be to endorse and even support spammers: I got from a /spammer/ an email with the same Subject: line as /I/ used to complain to /his/ network, meaning they were in cahoots. Soloway never paid, but he did go to jail. Robert Braverman, of Oklahoma, had sued and was awarded $11 MILLION, so the guy was bad news.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.