The sites given above are all not sending any intermediate certificates, which is a bad configuration.
Without intermediates, building the chain is left as an exercise to clients. There is a version of the R3 intermediate (used by Let's Encrypt) which just expired a few minutes ago. Clients using this certificate for verification will show those errors.
For some clients clearing caches or restarting systems will be enough to fetch the new intermediates. However it would still be best practice, if the sites above were sending (correct) intermediates.
Ok. Well, I've never used intermediate certs before and until now it hasn't caused a problem.
I always wondered what they were anyway.
With that in mind, what are intermediate certificates and how to use them? (Looking it up as I'm asking)
This is because browsers are quite good at fixing server misconfigurations on the fly. They usually cache the intermediates or get them from somewhere else if servers omit them. But not all clients do this (e.g OpenSSL does not) and not all clients do it well.
For a certificate to be trusted, it must be signed by a trustworthy authority. This trust is anchored in so-called "root certificates". These are shipped with your browser. Root certificates sign intermediate certificates which in turn sign leaf certificates. Leaf certificates are those you use on your sites. This signing creates a chain of trust which makes the leaf trustworthy.
The intermediate certificates used by Let's Encrypt should be automatically fetched by your ACME client (acme.sh in your case). Your acme client puts both leaf + intermediate certificates in a file usually called "fullchain"(.cer).
Usually you just need to configure your webserver to use this file and your webserver will serve both leaf + intermediates. However, if you're running a very old version of Apache (or older versions of other webservers), you might need additional configuration to serve the intermediates.
The fact that you didn't have any problem doesn't mean others didn't have the same luck
What I'm wondering is what might have changed in the last couple of hours.
Blissfully getting away with not having set things up correctly (for years), and then suddenly discovering it's time to revisit and fix your ancient mistakes... that part I get.
I'm just curious what on earth might have caused this sudden change. I didn't download a new browser, the servers haven't been updated lately, all (as far as I can tell in this moment). But suddenly "intermediate certificates", which I've wondered about for years, "wtf are those anyway"? are suddenly demanding I actually answer that question.
As already stated above, there's a version of the R3 intermediate which expired about 1 hour 42 minutes ago. This is causing the issues you're seeing. (This type of intermediate expiry happens rarely, especially this scenario is new).
@ovunque What browser are you using? I just tried current Edge on Windows and current Chrome on latest Android and both showed your site starsandstrife.com just fine. So, they 'fixed' your missing chain as noted by @Nummer378 who also noted some browsers / clients may not do this well.
Because your server is missing the chain, your browser is choosing / using a missing piece that expired today.
Ok, that helps. Sorry I overlooked the prior note. I forced and update, used the "fullchain" where before I wasn't, and that seems to have done the trick (at least for 2 of those domains anyway. db dot trillionpictures dot com will be a little different no doubt.)
This trust is anchored in so-called "root certificates". These are shipped with your browser. Root certificates sign intermediate certificates which in turn sign leaf certificates. Leaf certificates are those you use on your sites. This signing creates a chain of trust which makes the leaf trustworthy.
I almost grasp what you're getting at here, but not quite. (Or maybe "almost" is overstating it). Makes me painfully aware that I'm blissfully unaware of the how all this works behind the scenes.
The intermediate certificates used by Let's Encrypt should be automatically fetched by your ACME client
Yes it does, but in this case I had not been using "fullchain.cer" which, if I'm understanding these notes correctly contains both the domain and intermediate certs.
(I honestly don't know what the average user/admin perspective on these things is exactly. I've studied these things, so it's easy to lose user perspective)
Yes that's correct. Your acme client fetches everything you need, but (web-)servers also need to be configured to use them. And because different webservers work differently, there's not always the singular "do it this way" answer (which is also why acme.sh gives you both the certificate with and without intermediates).
