Thank for very much for your help. We work on a contingency plan as our CDN still hasn’t deployed the new certificate.
1 Like
A post was split to a new topic: Ability for Automated Notification of Revocations
The online hostname tool doesn’t accept multiple hostnames like it should, this is a great way to check domains in bulk.
For me I found a quick way was to get the server’s Account ID from URI field in /etc/letsencrypt/accounts/acme-v02.api.letsencrypt.org/directory/*/regr.json and then check the affected serials.txt for any affected account IDs (which can represent multiple domains at once):
grep ‘78311327’ caa-rechecking-incident-affected-serials.txt
Luckily, for me only 1 dev domain on a relatively new dev server was affected, and reissuing it within Plesk was a simple button click.
4 Likes
We’re using Akamai as well, as far as I can tell they’re holding up deployment of the refreshed certifcates on the network to be able to batch changes at once(edges would otherwise get many more updates than normal) and if these were to be processed in order, not all certificates would be deployed in time.
They’re targeting to have certificates deployed before revocation according to our account representative, we’ll have to see.
6 Likes
If you remove the renew-by-default, you can keep in running the command daily, as certbot will then only renew when the cert is about to expire.
It’s wise to run this daily, because if something fails, it’s better to retry the next day than next week.
3 Likes
@modemgeek this could be a problem with the forum substituting curly quotes (“like these”) for the ASCII 34 quotes " that bash expects.
2 Likes
If anyone is having capacity problems related to renewal with Let’s Encrypt, and if you know how to configure your ACME client, you can also consider the free “GO SSL” certificates from Buypass AS which are available from an ACME endpoint:
Many clients that work with Let’s Encrypt should also work with Buypass, although I don’t think a huge amount of compatibility testing has been done. If your client has a Let’s Encrypt ACME API endpoint hard-coded, it may be challenging to get it to request issuance from Buypass.
This is probably most relevant if you specifically know that you need to renew and your renewal is somehow being blocked by a Let’s Encrypt rate limit, or if you get an internal server error during your renewal. That shouldn’t be the case for most users here. Other kinds of validation and issuance problems are not very likely to be solved by getting a certificate from Buypass instead of Let’s Encrypt.
This forum probably can’t provide support for issuance and renewal issues with Buypass certificates, which should be directed instead to Buypass’s forums.
I wanted to mention this because I thought it could help a minority of people with a need to renew who are stuck on particular kinds of errors.
8 Likes
thanks. I caught that early on. I figured it out. It was the CR. I created the file on Windows and transferred to CentOS.
1 Like
My client https://github.com/bruncsak/ght-acme.sh tested and supports Buypass’ ACME service.
1 Like
I just read through the incident report and noticed these three lines:
2020-02-29 03:08 UTC: Let’s Encrypt engineer confirms bug.
2020-02-29 03:10 UTC: Let’s Encrypt SRE team disables issuance.
2020-02-29 05:22 UTC: Fixed version of Boulder is deployed (containing Pass authzModel by value, not reference by jsha · Pull Request #4690 · letsencrypt/boulder · GitHub). Issuance is re-enabled.
This is an absolutely amazing response time for both disabling issuance, and fixing/testing/re-enabling.
I know this is a stressful and frustrating situation, and many end-users are expectedly upset, but the emergency performance of the LetsEncrypt staff here is simply amazing and deserves commendation.
15 Likes
They seem to have failed at this. We’re past the deadline now and they still haven’t deployed.
(edit: this was a reply in regards to Akamai-managed LE certificates; apologies for any confusion. (they referred to Akamai in my post))
1 Like
EDIT: Think, I misread your comment and thought that you said le missed the deadline. Gonna leave this here, tho:
Is that really a bad thing tho? The fact that the deployment wouldn’t be at once was clearly stated. They got to have it deployed in 5.5 hrs tho.
Every minute won from deploying later are potentially also won for end users (especially those that use outlook) from not having that problem - and of course for their respective IT departs for renewing the cert.
1 Like
There’s an extension in coordination with LE.
1 Like
sorry for the confusion. I edited my post to reflect the unclear wording. I was referring to Akamai’s handling of their Akamai-managed LE certificates.
2 Likes
And Akamai is still not committing to be able to hit that extended deadline, currently.
1 Like
A post was split to a new topic: Are single-hostname certificates affected?
I received the same email as everyone, claiming a problem due to CAA. However, my domains don’t use CAA records, so there’s no chance of this bug affecting them. Did LE check the suspect certificates to see whether the domains use CAA before invoking? I don’t know what the usage of the CAA record is, but I suspect you could have reduced the revocations if you’d checked DNS beforehand.
2 Likes
I understand what you’re trying to say, but your domains were affected by this bug as it’s possible your lack of CAA records was not checked at time of issuance. That’s why they have to revoke the certs. They were issued in a noncompliant way.
7 Likes
Your Akamai account Rep is certainly giving you better updates than mine. My rep did mention just some time ago that they are more “positive” now that we will be able to get our certs deployed prior to the new deadline. Do you mind sharing any other updates that you get from Akamai. Are they looking to do this big bang at 7:59 PM EST 
2 Likes
Part of it comes from the messages in Luna. We had a couple of calls during the day (CET).
However this is typically a case of getting notified too late by LE, Akamai having a system in place that’s designed to renew certificates in a slower pace than currently required. I have to be honest, that current slowness is something I haven’t seen from it in the past 4 years. But it’s been rocksolid for us previously as well as support which has approached this matter for us and our customer proactively.
2 Likes