We are hosting school district websites using automated Let's Encrypt certificates (via an ACME client backend).
The Problem: Recently, several school districts have reported intermittent "root certificate errors" blocking access to their websites. Crucially, this only happens internally from within the school networks on devices running the Lightspeed Filtering Agent (an SSL-decrypting content filter). Outside the school network (or on devices without the filter), the websites load perfectly.
Our Findings:
The issues began immediately following recent certificate renewals.
The culprit seems to be the new Let's Encrypt certificate chains (specifically involving the newer ISRG root/intermediates) interacting poorly with older versions of the Lightspeed agent's trust store during SSL decryption.
While the school IT teams are looking into upgrading their Lightspeed agents, this is a slow process across multiple districts.
My Questions for the Community:
Is there a known workaround or "preferred chain" flag we can configure in our ACME client to force Let's Encrypt to issue certificates using the older, widely trusted legacy chain that won't trigger these Lightspeed agent errors?
What is the exact string/parameter we should pass to our ACME client (e.g., Certbot or similar) to target the most widely compatible chain for environments with strict/outdated SSL-decrypting middleboxes?
I think a lot depends on what exactly it is that that SSL middlebox isn't liking. The default chain that's provided chains up to ISRG Root X1 which is the same root as has been used all along. That is, the default settings are already the chain that's most compatible across most systems.
There is a possibility that the web servers aren't sending the full certificate chain. Could you either provide the domain of a failing web server or try connecting to https://valid.x1.test-certs.letsencrypt.org/ through an un-updated firewall just to make sure that it's an issue with your firewall.
It's not possible to change the chain and it is a bad idea anyway.
If there is nothing wrong with the certificate chain of your school's websites, it would mean that there is a bug in the filtering appliance that cannot handle the certificate chain.
In that case, you would not be able to visit a lot of websites, because soon all websites using let's encrypt certificates will start using the new, longer chain. And I think most of them already are.
Please check if the chain certificates are all properly installed in the webservers of your school's websites.
And if they are, you'll have no other choice then upgrading the filtering appliances if they contain a bug.
Thank you for the feedback everyone! Our partners believe they have fixed the issue with updating their filtering agent. I will come back to this thread if the issue persists and/or resurfaces.