I'm having a problem with certificate renewal for numerous hosts spread throughout my network. I keep getting certificate expiry notices and when I try to renew manually, at least one of the host names fail. There are perf sonar nodes for internal network testing and have multiple interfaces, names and IP addresses. I've verified that all are reachable and functioning from the Internet but the certificate renew failure from Lets Encrypt continues.
In this instance, the certificate was created with certbot version 0.38.0:
/usr/local/bin/certbot certonly --apache
--cert-name ps-brandonu.mrnet.mb.ca
-d ps-brandonu.mrnet.mb.ca
-d lat-brandonu.mrnet.mb.ca
-d lat6-brandonu.mrnet.mb.ca
-d bdw-brandonu.mrnet.mb.ca
-d bdw6-brandonu.mrnet.mb.ca
All of the above host names resolve to the same physical hardware but two interfaces on that hardware. What's weird is the ps-brandonu host is fine but the lat-brandonu fails - ps-brandonu is on exactly the same interface a lat-brandonu and shares the same IPv4 and IPv6 addresses. Stranger still, the IPv6 name also works so the problem seems to be very specifically "lat-brandonu" which has both A and AAAA DNS records.
Somewhere along the line the certificate seems to have dropped the lat-brandonu:
[root@lat-brandonu ~]# /usr/local/bin/certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Found the following certs:
Certificate Name: ps-brandonu.mrnet.mb.ca
Domains: ps-brandonu.mrnet.mb.ca bdw-brandonu.mrnet.mb.ca bdw6-brandonu.mrnet.mb.ca lat6-brandonu.mrnet.mb.ca
Expiry Date: 2021-02-28 17:22:24+00:00 (VALID: 79 days)
Certificate Path: /etc/letsencrypt/live/ps-brandonu.mrnet.mb.ca/fullchain.pem
Private Key Path: /etc/letsencrypt/live/ps-brandonu.mrnet.mb.ca/privkey.pem
An attempt to get the certificate again and add in lat-brandonu again doesn't seem to fix things:
[root@lat-brandonu ~]# /usr/local/bin/certbot certonly --apache
--cert-name ps-brandonu.mrnet.mb.ca
-d ps-brandonu.mrnet.mb.ca
-d lat-brandonu.mrnet.mb.ca
-d lat6-brandonu.mrnet.mb.ca
-d bdw-brandonu.mrnet.mb.ca
-d bdw6-brandonu.mrnet.mb.ca
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
You are updating certificate ps-brandonu.mrnet.mb.ca to include new domain(s):
- lat-brandonu.mrnet.mb.ca
You are also removing previously included domain(s):
(None)
Did you intend to make this change?
(U)pdate cert/(C)ancel: u
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for lat-brandonu.mrnet.mb.ca
Waiting for verification...
Challenge failed for domain lat-brandonu.mrnet.mb.ca
http-01 challenge for lat-brandonu.mrnet.mb.ca
Cleaning up challenges
IMPORTANT NOTES:
-
Congratulations! Your certificate and chain have been saved at:
/etc/letsencrypt/live/ps-brandonu.mrnet.mb.ca/fullchain.pem
Your key file has been saved at:
/etc/letsencrypt/live/ps-brandonu.mrnet.mb.ca/privkey.pem
Your cert will expire on 2021-03-10. To obtain a new or tweaked
version of this certificate in the future, simply run certbot
again. To non-interactively renew all of your certificates, run
"certbot renew" -
The following errors were reported by the server:
Domain: lat-brandonu.mrnet.mb.ca
Type: unauthorized
Detail: Invalid response from
http://lat-brandonu.mrnet.mb.ca/.well-known/acme-challenge/5PoBy2d8Wndel9Bs1vx_6szmmgebuC-o6ORPND75rZQ
[2001:410:2000:2002::101]: "\n\n404 Not
Found\n\nNot Found
\n<p"To fix these errors, please make sure that your domain name was
entered correctly and the DNS A/AAAA record(s) for that domain
contain(s) the right IP address. -
If you like Certbot, please consider supporting our work by:
Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate
Donating to EFF: https://eff.org/donate-le
So the renewal/update of the certificate, attempting to add lat-brandonu.mrnet.mb.ca fails. Is there any way to get a more detailed look at what that Failed Response means?
From a host out on the Internet I know the server is up. Using "nmap" to fetch http headers for lat-brandonu.mrnet.mb.ca for both IPv4 and IPv6:
Nmap done: 1 IP address (1 host up) scanned in 1.12 seconds
Starting Nmap 7.80 ( https://nmap.org ) at 2020-12-10 11:12 CST
Nmap scan report for lat6-brandonu.mrnet.mb.ca (2001:410:2000:2002::101)
Host is up (0.039s latency).
rDNS record for 2001:410:2000:2002::101: lat-brandonu.mrnet.mb.ca
PORT STATE SERVICE
80/tcp open http
| http-headers:
| Date: Thu, 10 Dec 2020 18:12:01 GMT
| Server: Apache
| Location: https://lat6-brandonu.mrnet.mb.ca/
| Content-Length: 242
| Connection: close
| Content-Type: text/html; charset=iso-8859-1
|
|_ (Request type: GET)
443/tcp open https
| http-headers:
| Date: Thu, 10 Dec 2020 18:12:01 GMT
| Server: Apache
| Strict-Transport-Security: max-age=31536000; includeSubDomains
| Protocols: h2 http/1.1
| Content-Security-Policy: default-src 'self' ; script-src 'self' 'unsafe-eval' 'unsafe-inline' ; img-src 'self' 'unsafe-inline' data: ; style-src 'self' 'unsafe-inline' ; connect-src *
| X-Frame-Options: DENY
| X-XSS-Protection: 1; mode=block
| X-Content-Type-Options: nosniff
| Connection: close
| Content-Type: text/html; charset=utf-8
|
|_ (Request type: HEAD)
This is an Apache 2.4 server running under Centos 7:
- Tue Nov 10 2020 CentOS Sources bugs@centos.org - 2.4.6-97.el7.centos
The application itself seems to be working fine and I can use all of the host names from a web browser, I just cannot update/renew part of the certificate. In looking at the debug log it appears some configuration changes are made to the apache configuration - but only for the lat-brandonu challenge. Could those be "broken?" somehow?
Something else I'm missing?