After certificate renewal on multiple servers I administer (they have all been fixed manually now, so disclosing them here won't help), I'm still getting expired certificate errors when using tools like curl or wget to download something from those servers. It seems the ACME server is still sending the old ISRG root certificate, cross-signed by an expired DST Root CA X3 certificate rather than the self-signed new root.
Since tools like curl and wget aren't as smart as web browsers, they fail when they see the expired root.
Is there a way to reconfigure certbot somehow to issue chains with the new root rather than the old one ? Having to manually fix all chains after each renewal is pretty annoying.
Certbot is version 1.21 if that matters.
Let's Encrypt currently do use the chain ending in DST Root CA X3 (mainly motivated by compatibility with old Android devices).
If you wish to use the modern chain you need to use the --preferred-chain "ISRG Root X1" command line option or set the configuration option in your renewal configuration file preferred_chain = ISRG Root X1
In addition to what others said above, please consider running your operating system's updates AND/OR updating the operating system itself.
The behavior you mentioned should only affect fairly old versions of openssl, operating systems, and standard applications. Recent system releases, and the support updates for older systems (going back at least 5 years!), implement logic in OpenSSL (and other libraries) that will essentially ignore the expired DST Root and use some "short circuit" logic to build an alternate trust path to the ISRG Root.
What is "fairly old" ?
Updating the OS isn't always possible, for example I administer some older servers (servers which belong to customers of the company I work for) which run custom-built software which just won't work under newer OS versions without resorting to a ton of workarounds (chroot, LD_LIBRARY_PATH, etc.). The best I can do for those servers without recompiling CURL, OpenSSL, etc. is update the CA repository which usually works fine as long as the host server presents the correct root CA. And every time the LE certificate is renewed, I have to manually fix the chain.pem and fullchain.pem with correct root CAs.
What is the reason you use an expired certificate in the chain ? Old Android versions ?
Is it possible to reconfigure Certbot to use the correct root ? Or I will have to write a custom hook script to update the chain files with the correct root ?
For such outdated systems, IMHO, the very best anyone can do (to hyper-extend their life expectancy) is to P2V (virtualize) them. And then put the VM(s) behind proxies (both inbound and outbound).
Then you either have an old ACME client OR you aren't getting the most out of your current client.
There is an option --preferred-chain that can be used to make that change for you.
Thanks, I will try that.
Unfortunately, I do not own those servers. Majority of them aren't exposed directly to the interweb though, most are behind a VPN and a squid proxy.
Most major operating systems have backported the OpenSSL fix, and other affected libraries, to releases dating back many years. Unless the OS is very old, you shouldn't have to recompile anything, merely install updates. This is the recommended fix, as this change helped backport numerous security updates as well. IIRC, several distributions released backports to support the alternate chain building function for systems that were no longer in a standard support window, and had not received security fixes.