Today one of my websites started to show expired certificate error.
I think not every PC user knows how to click "skip" button in his browser, so I lost some visitors this morning.
The research revealed there are AGAIN incompatible changes of the protocol that caused
Error: LetsEncrypt challenge request 405
I was forced to change my APT sources list and then update web hosting panel.
Could you please permanently STOP making any changes that may break the stability of existing informational systems or services? If the client is issued, it must work until the service (Let's Encrypt) exists. I was lucky to find the proper update of my panel, but what will I do in case they will not issue an update in time?
The good form rules are, e.g. XMPP protool: server or client may support some extensions, but in any case without these extensions client will successfully connect to server and basic functionality will work stable for years (or decades). Other examples are SMTP, POP3 and IMAP.
I remember at least few cases when website starts to show certificate expired (or "date invalid") error due to changes in the ACME protocol, thus I decided to write here immediately. It usually happens once per 1-2 years, but sometimes it may cause significant problems, not just gone visitors during the morning.
The panel script (VestaCP) updates certificates some time forward, maybe even 30 days, I don't know. This is NOT a dns problem since minor dns outages may not due for a week or month. I've launched the update certificates cron task manually and saw 405 errors. Some forum posts from 2019 and 2020 tell it's due to new protocol and old client, so 405'th error happens enough often.
Today's certificate was until July 9th, e.g. issued on April 9th +- when it worked without problems.
So my suggestion is actual: any changes must be compatible with older clients to prevent outages, regardless of when they were announced.
Well, while I agree with this statement in general, I also think Let's Encrypt deserves some slack here. From the beginning it was clear there would be incompatibilities between the ACME API at the time and the final RFC, as @_az has already stipulated. Think of it this way: what option would you rather have:
A third ACME API in stead of the now disabled ACMEv1 and current ACMEv2 APIs? This would mean your client would need an update too;
Let's Encrypt waiting for the RFC become final until they started their public API? That would mean no certificates for you for YEARS!
Changes to the current API as implemented now.
I'm interested to hear which option you would prefer. Also note that "don't make a change" is not really an option obviously, as Let's Encrypt should adhere to the RFC 8555 as much as possible.
Some certificate renewal failure may occur, due to some unexpected condition. If you start early enough the renewal (30 days before the certificate expiry) and try to renew the certificate about twice a day automatically, it is very unlikely that the certificate is not going to be renewed due to transient error condition. However, you must configure a defence line: if a certificate is still not renewed two weeks before its expiry, it won't be still at the end of its lifetime. That situation is supposed to trigger a high priority alarm directly to the administrator to look around in the system to check what is going on. That gives two weeks time to the administrator to fix the ACME client or anything else, and no service interruption will occur.
With ACME client that is no more working (e.g. in my case) I may try to update certificate every hour, the resut will be the same (failure).
I'm going to add certificate expiration date to my uptime monitoring (contrary to current HEAD /robots.txt with no SSL). That is really nice idea to notify administrator 2-3 days before the expiration as it already must be updated.
I hope RFC will be final and the protocol will be stable in future. As an example I've provided XMPP and SMTP, they exist for years, there are many addons but the base is stable and all old clients may use it along with newest versions. In case of XMPP there could be no confirmation of delivery, or file / voice transfer. In case of SMTP there could be no StartTLS support, but it will work.
I wish Let's Encrypt developers good luck and successes in developing this unique service.
Please keep in mind that those protocols do not rely upon a third party providing a service. They also do not hinge upon the shifting nature of security, which is one of the primary reasons for the very change that drove this topic to be created. I'm fairly certain that you don't think your antivirus software from 1995 should not ever evolve, so why would you believe that technology at the core of the integrity of the internet would not evolve? I understand that you're upset about the breakage you've seen. I'm an ACME client author, so I get it, but I simply can't justify holding the expectation that the operation of my software should never change in the light of new concerns. It's just not practical.
I am 100% with you on this and would love for ACME (and other protocols) to be on their final and permanent stable version. Unfortunately it's not that simple.
Technology changes, our knowledge of the world increases, and requirements evolve. This fact means that protocols will have to undergo changes throughout their lifespan. Many of which are backwards compatible, but there will be ones that are backwards incompatible. There are ways to mitigate the effects of these, and I believe that Let's Encrypt has done an awesome job with them (see the ACMEv1 deprecation efforts).
With other protocols, enabling backwards compatibility is possible in a majority of cases. ACME is different since it involves trust. It just isn't right to even slightly risk compromising the security of the world to avoid an inconvenience.
Nonetheless, In my opinion it's no less than a miracle that the basic protocols that underpin the internet such as TCP, UDP and DNS haven't had to undergo a breaking change since flag day.
Linux is pretty sluggish with change outside of security problems
Apache has been a stalwart of stability
MySQL is also very mature
PHP was modernized to make it more energy efficient