Hello 
The certificates you've posted are:
The certificates you need are:
It's really weird that you have a leaf certificate signed by R3 (as you should) then the Let's Encrypt Authority X3 intermediate certificate (as you shouldn't). I surmise that your virtualmin needs its intermediate database updated.
I'm wondering if it might be an out-of-date certbot/certbot docker image:
# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
certbot/dns-linode latest 56cb3c466315 7 months ago 117MB
certbot/certbot latest 67cfe9e9c63f 7 months ago 95.5MB
hello-world latest bf756fb1ae65 21 months ago 13.3kB
The full chain is provided by Boulder (Let's Encrypt's CA software) to certbot (or whatever ACME client is being used) each time a certificate is generated. The ACME client doesn't/shouldn't keep a database of intermediate certificates. Since you have an R3-signed certificate, your ACME client must have been sent the current chain, which would include R3 and not include Let's Encrypt Authority X3. What you've currently presented is like buying a Honda vehicle then popping the hood to find a Toyota engine. We see this problem commonly with control panel and other site management software that does keep a database of intermediate certificates (that are often not updated). That does not appear to be the case here though. The problem may potentially be due to some custom script work as @swelljoe mentions below.
I just want to note that I don't think this script is generated by Virtualmin? I can't find any mention of it in our repos, and I don't think this is how renewal happens.
We don't ship an intermediate database. Up to Webmin 1.870 we did have a copy of the LE intermediate cert included (I think due to the minimal ACME_tiny client we used by default, I don't remember details), but that hasn't been the case for some time. I'm not sure how this system is getting the old one...we grab a new one on every renewal, as far as I know, and all modern systems should be using certbot for the cert and chain request, by default.
I want to be clear, I do not understand this issue (the cross-chain signing issue, which is still plaguing our users and lots of others), but I do know we stopped shipping the old intermediate chain in the distant past. It should not be an issue on up-to-date Virtualmin systems.
But, even with certificates generated after the expiration of that DST Root CA X3 cert, I still see it show up as a certification path (even though there is a valid certification path for newer clients), and I don't understand why. This exact same thing happens for certificates generated directly with certbot, without Virtualmin involved, so it is unrelated to Virtualmin (though OP does seem to have something wrong if they're getting the old intermediates, but it's not the case in my Virtualmin deployments, so something is unusual with OPs Virtualmin system or the way certs are being generated/used).
e.g. this is a cert issued after the expiration:
I can reproduce this same broken certification path on a certbot-generated certificate, as well, again with no Virtualmin or Webmin involved. In this case, Virtualmin isn't doing anything unique to cause this problem, and I don't know how to solve it, if certbot also generates a certificate that can fail for some clients.
I tested on an Ubuntu 16.04 system, where I can reliably reproduce the issue, and found that if I simply removed the DST_Root_CA_X3 certificate from the system CA bundle, a valid certification path could be found without it...but if it exists, requests fail.
That's not a broken chain, that's all correct and intended.
Let's Encrypt is intentionally serving a chain up to the now expired DST Root CA X3, because that helps with Android compatibility. All clients that are not Android shall use the path leading up to ISRG Root X1 for validation, ignoring the expired DST Root CA X3.
We're aware that there are some libraries (e.g older OpenSSL) that can't do this*, which is why Let's Encrypt offers an alternate chain which does not include the certificate leading up to DST Root CA X3 (instead terminating at ISRG Root X1). This alternate chain can be requested by the ACME client, if Android compatibility is not desired.
*Some libraries, e.g OpenSSL 1.0.2 have a workaround where simply removing DST Root CA X3 from the trust store makes them compatible with the default chain. This is also why some vendors (Ubuntu) have recently started shipping patches which remove DST Root CA X3.
OK, that answers my question/confusion. I assumed I was seeing a representative example of the problem when I was testing on Ubuntu 16.04, but it seems to be somewhat different problem with the same symptoms.
This also confirms that Virtualmin with Webmin in any version after 1.870 is doing the right thing WRT the chain (it's using whatever certbot gives it). So, I don't know why OP has the old intermediate chain...seems to be something in custom scripts.
Thanks for the confirmation of that @swelljoe. This information not only helps whittle down the issue here, but can be recycled in other cases in the future.
I just want to note that I don't think this script is generated by Virtualmin? I can't find any mention of it in our repos, and I don't think this is how renewal happens
I was unclear in my wording. The script is my own and it uses the virtualmin-supplied script virtualmin install-cert.
The reason for this is that I had a requirement for wild card domain names a while ago, which virtualmin's system didn't support. So I had to generate my own certificates. The script was a letsencrypt deploy hook script.
Where do I put the two files so that certbot will grab them:
Trying to figure out where certbot might be grabbing this stuff, I ran bash commands like:
cp /etc/letsencrypt/live/algasol.com/fullchain.pem .
split the two certs into fc1.pem and fc2.pem
locate Authority |egrep '\.(pem|crt)' >cafiles.log
locate CA |egrep '\.(pem|crt)' >>cafiles.log
wrote a perl script
cat difflength.pl
while(<>){
chomp $_;
$df = `diff fullchain.pem $_`;
$len = length($df);
print $len,' ',$_,"\n";
}
perl difflength.pl <cafiles.log >lensfc.log
did the same for fc1.pem and fc2.pem
for each file I took the 0 length diff full paths and put them into a files named {fc*}paths.log
then I wrote another perl script:
cat chompfilename.pl
while(<>){
s/[^\/]*$//;
print;
print "\n";
}
and finally
ls *paths* |xargs cat|perl chompfilename.pl |uniq
producing:
0 /snap/core20/1081/etc/ssl/certs/
0 /snap/core20/1026/etc/ssl/certs/
0 /snap/core18/2128/etc/ssl/certs/
0 /snap/core18/2074/etc/ssl/certs/
0 /snap/core/11743/etc/ssl/certs/
0 /snap/core/11606/etc/ssl/certs/
0 /snap/core20/1081/etc/ssl/certs/
0 /snap/core20/1026/etc/ssl/certs/
0 /snap/core18/2128/etc/ssl/certs/
0 /snap/core18/2074/etc/ssl/certs/
0 /snap/core/11743/etc/ssl/certs/
0 /snap/core/11606/etc/ssl/certs/
0 /snap/core20/1081/etc/ssl/certs/
0 /snap/core20/1026/etc/ssl/certs/
0 /snap/core18/2128/etc/ssl/certs/
0 /snap/core18/2074/etc/ssl/certs/
0 /snap/core/11743/etc/ssl/certs/
0 /snap/core/11606/etc/ssl/certs/
So I'm not sure what all these directories are but, unless as I suspected in my concern about the docker image for certbot I'm using, one of these directories is where I should copy the certificate authority files.
I also, after updating the docker certbot, forced renew and although there are new certs in the right places, even after a restart of the services I'm still getting the intermediate chain error.
Your leaf certificate and two intermediate certificates (R3 and ISRG Root X1) should already be in fullchain.pem produced by certbot. If you're only seeing your leaf certificate and the R3 intermediate certificate in fullchain.pem then you have certbot configured for the alternate/short chain instead of the default/long chain.
The phrase "certbot configured" would direct me to this documentation on certbot which has information on the configuration file (and command line options) but the word "authority" occurs only once and not in reference to a configurable parameter.
The only other sense in which the word "configured" may be applicable that I can think of is in terms of version alignments between packages included in the certbot environment and that brings me back to my question about the Docker image for certbot. I ended up going to the Docker image for certbot precisely to avoid version alignment problems. Indeed, the reason Docker is in such widespread use is that people want to avoid version alignment problems.
Should I start digging into the build spec for the Docker build for certbot?
The certbot parameter of concern here is --preferred-chain.
Or just call it a day and use another ACME client that can.
Does this look right?
# openssl verify -verbose -show_chain -CAfile /etc/letsencrypt/live/algasol.com/chain.pem /etc/letsencrypt/live/algasol.com/cert.pem
/etc/letsencrypt/live/algasol.com/cert.pem: OK
Chain:
depth=0: CN = *.algasol.com (untrusted)
depth=1: C = US, O = Let's Encrypt, CN = R3
depth=2: C = US, O = Internet Security Research Group, CN = ISRG Root X1
This is the command I ran (after some confusion and work trying to figure out exactly what string to pass to the --preferred-chain parameter):
docker run -it --rm --name certbot \ -v "/root/.linode_api:/root/.linode_api" \ -v "/etc/letsencrypt:/etc/letsencrypt" \ -v "/var/lib/letsencrypt:/var/lib/letsencrypt" \ -v "/var/log:/var/log" \ -v "/etc/letsencrypt/renewal-hooks/deploy" \ certbot/dns-linode renew --force-renewal --preferred-chain "ISRG Root X1"
But I'm still getting a broken chain diagnostic from "SSL Checker" after having deployed via the virtualmin install-cert command and restarting apache
That's because the webserver for algasol.com is still only serving the leaf certificate (and no intermediate certificates):
https://decoder.link/sslchecker/algasol.com/443
You need to fix your Apache configuration to actually serve the intermediate certificates.
OK, I went in and manually over-rode the virtualmin install-cert operation in the apache2 configuration for algasol.com:
#SSLCACertificateFile /home/algasol/ssl.ca
SSLCACertificateFile /etc/letsencrypt/live/algasol.com/chain.pem
So web access works now.
I'll have to do the same for dovecot etc. as well apparently.
Looking better. I'm not seeing the second intermediate certificate (ISRG Root X1) though, probably because of this:
It all depends upon which chain you want:
IPv4 and IPv6 are not identical:
openssl s_client -connect algasol.com:443 -servername algasol.com -4 | head
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = *.algasol.com
verify return:1
CONNECTED(00000005)
---
Certificate chain
0 s:CN = *.algasol.com
i:C = US, O = Let's Encrypt, CN = R3
1 s:C = US, O = Let's Encrypt, CN = R3
i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
---
openssl s_client -connect algasol.com:443 -servername algasol.com -6 | head
140715871445440:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../ssl/record/ssl3_record.c:332:
CONNECTED(00000005)
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 5 bytes and written 313 bytes
Verification: OK
---
IPv6 fails 
That's a great observation @rg305. I suspect that this may further indicate Apache configuration issues. Maybe incorrectly using IP:80 in the VirtualHosts instead of *:80 ?