#HugOps@LE 
Thanks for the additional time to get replacement certs.
Thanks @JamesLE et al (including Network Solutions) for getting resolution to that problem which was blocking/slowing getting replacement certs for hostnames with worldnic NS.
We were able to get all ~1000 of our affected certs reissued prior to revocation. 
7 Likes
Our CDN (Akamai) is still overloaded 
The renewal didn’t progress for 14 hours !
I opened a ticket with their support team, but I’m really worried it may not renew before revocation.
4 Likes
Your options seem to be to trust your vendor to get it resolved or have a contingency plan where you get your own cert (from LE or any other CA) and know how to deploy it (to Akamai or another CDN).
2 Likes
I’m still working through issues renewing my cert which I have a separate thread on already.
In case i’m unable to renew my cert before it gets revoked id like to know what type of effect using a revoked cer will have on my email server and the users’ experience? Will send/receive communications continue but with warnings or will it just stop working altogether ?
This is the first time that I’m unsuccessful to renew the certs and would like to be able to warn the users until it gets fixed.
1 Like
Checking validity is done by OCSP, and as these responses are valid for 4/5 days, they may remain in the cache of your clients, or if OCSP stapling is enabled, your server.
So it’s quite unpredictable at which point users will see an error: somewhere between 0 and 4 days later.
At least in the case of Thunderbird, you’ll get a message like the one in the screenshot here: https://support.mozilla.org/en-US/questions/1041573
By the way the status page now says that revocation will start at 20:00 UTC.
3 Likes
I'm getting this error when running the script. Not sure what it means exactly as i'm not proficient in bash
./run_check.sh: line 15: syntax error near unexpected token done' ./run_check.sh: line 15: done < "$input"
1 Like
Thank for very much for your help. We work on a contingency plan as our CDN still hasn’t deployed the new certificate.
1 Like
The online hostname tool doesn’t accept multiple hostnames like it should, this is a great way to check domains in bulk.
For me I found a quick way was to get the server’s Account ID from URI field in /etc/letsencrypt/accounts/acme-v02.api.letsencrypt.org/directory/*/regr.json and then check the affected serials.txt for any affected account IDs (which can represent multiple domains at once):
grep ‘78311327’ caa-rechecking-incident-affected-serials.txt
Luckily, for me only 1 dev domain on a relatively new dev server was affected, and reissuing it within Plesk was a simple button click.
4 Likes
We’re using Akamai as well, as far as I can tell they’re holding up deployment of the refreshed certifcates on the network to be able to batch changes at once(edges would otherwise get many more updates than normal) and if these were to be processed in order, not all certificates would be deployed in time.
They’re targeting to have certificates deployed before revocation according to our account representative, we’ll have to see.
6 Likes
If you remove the renew-by-default, you can keep in running the command daily, as certbot will then only renew when the cert is about to expire.
It’s wise to run this daily, because if something fails, it’s better to retry the next day than next week.
3 Likes
@modemgeek this could be a problem with the forum substituting curly quotes (“like these”) for the ASCII 34 quotes " that bash expects.
2 Likes
If anyone is having capacity problems related to renewal with Let’s Encrypt, and if you know how to configure your ACME client, you can also consider the free “GO SSL” certificates from Buypass AS which are available from an ACME endpoint:
Many clients that work with Let’s Encrypt should also work with Buypass, although I don’t think a huge amount of compatibility testing has been done. If your client has a Let’s Encrypt ACME API endpoint hard-coded, it may be challenging to get it to request issuance from Buypass.
This is probably most relevant if you specifically know that you need to renew and your renewal is somehow being blocked by a Let’s Encrypt rate limit, or if you get an internal server error during your renewal. That shouldn’t be the case for most users here. Other kinds of validation and issuance problems are not very likely to be solved by getting a certificate from Buypass instead of Let’s Encrypt.
This forum probably can’t provide support for issuance and renewal issues with Buypass certificates, which should be directed instead to Buypass’s forums.
I wanted to mention this because I thought it could help a minority of people with a need to renew who are stuck on particular kinds of errors.
8 Likes
thanks. I caught that early on. I figured it out. It was the CR. I created the file on Windows and transferred to CentOS.
1 Like
I just read through the incident report and noticed these three lines:
2020-02-29 03:08 UTC: Let’s Encrypt engineer confirms bug.
2020-02-29 03:10 UTC: Let’s Encrypt SRE team disables issuance.
2020-02-29 05:22 UTC: Fixed version of Boulder is deployed (containing Pass authzModel by value, not reference by jsha · Pull Request #4690 · letsencrypt/boulder · GitHub). Issuance is re-enabled.
This is an absolutely amazing response time for both disabling issuance, and fixing/testing/re-enabling.
I know this is a stressful and frustrating situation, and many end-users are expectedly upset, but the emergency performance of the LetsEncrypt staff here is simply amazing and deserves commendation.
15 Likes
They seem to have failed at this. We’re past the deadline now and they still haven’t deployed.
(edit: this was a reply in regards to Akamai-managed LE certificates; apologies for any confusion. (they referred to Akamai in my post))
1 Like
EDIT: Think, I misread your comment and thought that you said le missed the deadline. Gonna leave this here, tho:
Is that really a bad thing tho? The fact that the deployment wouldn’t be at once was clearly stated. They got to have it deployed in 5.5 hrs tho.
Every minute won from deploying later are potentially also won for end users (especially those that use outlook) from not having that problem - and of course for their respective IT departs for renewing the cert.
1 Like
There’s an extension in coordination with LE.
1 Like
sorry for the confusion. I edited my post to reflect the unclear wording. I was referring to Akamai’s handling of their Akamai-managed LE certificates.
2 Likes