Revoking certain certificates on March 4

A post was split to a new topic: Error processing CAA for domain

Your script does not retrieve the correct serialnumber when the server uses SNI. You need to indicate the servername:

openssl s_client -connect example.com:443 -showcerts -servername example.com </dev/null 2>/dev/null | openssl x509 -text -noout | grep -A 1 Serial\ Number | tr -d :

4 Likes

Thanks @kjo, that’s a good point.

1 Like

Good point @kjo. I believe recent versions of openssl actually do use the connect string to set the SNI field, but I agree that older versions do not, and we should have the more robust command.

3 Likes

A post was split to a new topic: Hit rate limit renewing certificates

A post was merged into an existing topic: Hit rate limit renewing certificates

When I first heard about this problem, at about 1600 UTC, I had not received an email. But i thought to check, so I ran a script and discovered one current certificate was affected.

Several hours later, when I was about to double check my script after reading kjo’s comment, I noticed that I had received an email at about 2000 UTC.

I suppose the emails are still being sent out, but there are rumors going around that if you don’t get an email, you aren’t affected. Please someone from Let’s Encrypt staff, if you can edit the original article, make it clear that non-receipt of an email is not equivalent to being unaffected. Even if you’ve verified that Let’s Encrypt has your contact details and the address doesn’t get sent to spam, the email might just still be on its way.

7 Likes

That's a great point. I'll get this change worked up now. FWIW, the notification mailer has completed it's run.

6 Likes

I’m not really sure if https://checkhost.unboundtest.com/ is working correctly, for one of my affected domains it’s showing The certificate currently available on [name] is OK ... but the file downloaded from https://letsencrypt.org/caaproblem/ is still showing missing CAA checking results for [name] at [datetime] and the certificate was last renewed almost a month ago (until now).

Edit: looks like the certificate was actually renewed a day after or so so it’s all good I guess

3 Likes

@vedranl Would you mind letting me know what the domains/certs in question are so I can check?

1 Like

Looks like I can’t send private messages, any way to send this in private?

1 Like

Maybe you renewed the certificate, so only the old one, not used anymore, is affected and will be revoked? (for example if you had the wrong configuration for certbot: renew-by-default which renew the certificate even when it's not yet needed)

1 Like

@Phil Is there any plan to push back the start time of the revocations since some people (like me) literally just received the email notice in the last 2 hours which is just over 4 hours before midnight UTC?

2 Likes

@fleish

I truly understand your sentiment, but I apologize that we must keep a strict deadline here or risk further baseline requirement violations. We definitely have an improvement to make in our notification emailer throughput. Unfortunately that doesn’t help at this very moment.

2 Likes

@Phil, the improvement to make is that in future, you should not waste away the five days of notification period on you deciding to make a tool to check which certificates were affected; you should have immediately informed the community and let them renew all of their certificates to resolve the issue.

Is there a place where we can see how your internal governance planned to handle an event such as this one? I think that the community is quite obviously telling you all that your process in this circumstance was wildly misguided and I want to know how you all plan to improve upon this in the future.

To be clear, the fact you had a bug is not the problem; the fact that you completely wasted the notification period is the problem.

6 Likes

Yeah, same here. Got an email 4.5h before 00:00 UTC. Glad I get off on abuse and risk. :blush:

2 Likes

There are 3 million certificates affected out of 116 million. Under which circumstances is a certificate affected?

I am asking because in my case, I got the email for my personal websites and account where every domain was affected. For my business I did not get an email, so I checked all domains against the list with affected serials and not one of them is affected (list contains ~2000 domains).

I am quite scared that my customers will contact us tomorrow and report that their domains are not working anymore. But I checked it so many times know with different ways (extracting the serial from the pem file, via openssl and searched the domain names in that file) and never got a result, so I am probably safe, but it still does not feel good.

Can you give us any more information why some domains are affected and some not? Is it only because of regular reissuance as you said in your first post? For my private websites I am using Traefik, so this webserver is then more regularly renewing the certs than my business which uses the the official letsencrypt client?

EDIT: The bug was confirmed on the 29th of February (according to the announcement). There have been some renewals in that timeframe since then, but in general the registration and renewal time frames are quite random, so the certificates do not have been renewed recently, there are still several from e.g. January.

2 Likes

Yes you are right, it was renewed 30 days before it was supposed to expire. Thanks again @yuriks for looking into this

3 Likes

@digilist only certificates containing more than one SAN (domain) and that had a specific timing requirement (authorization reused and 8 hours or more after the initial CAA check. See 2020.02.29 CAA Rechecking Bug for details.) are affected by this. If the serials for your certs are not in https://letsencrypt.org/caaproblem/ then they will not be revoked.

3 Likes

I added a new entry to the FAQ talking about reasons you may receive an email and yet not need to renew your certificate. This covers the problem experienced by @vedranl as well as some others. Thank you!

4 Likes