Certificate Renewal failures

I'm having a problem with certificate renewal for numerous hosts spread throughout my network. I keep getting certificate expiry notices and when I try to renew manually, at least one of the host names fail. There are perf sonar nodes for internal network testing and have multiple interfaces, names and IP addresses. I've verified that all are reachable and functioning from the Internet but the certificate renew failure from Lets Encrypt continues.

In this instance, the certificate was created with certbot version 0.38.0:
/usr/local/bin/certbot certonly --apache
--cert-name ps-brandonu.mrnet.mb.ca
-d ps-brandonu.mrnet.mb.ca
-d lat-brandonu.mrnet.mb.ca
-d lat6-brandonu.mrnet.mb.ca
-d bdw-brandonu.mrnet.mb.ca
-d bdw6-brandonu.mrnet.mb.ca

All of the above host names resolve to the same physical hardware but two interfaces on that hardware. What's weird is the ps-brandonu host is fine but the lat-brandonu fails - ps-brandonu is on exactly the same interface a lat-brandonu and shares the same IPv4 and IPv6 addresses. Stranger still, the IPv6 name also works so the problem seems to be very specifically "lat-brandonu" which has both A and AAAA DNS records.

Somewhere along the line the certificate seems to have dropped the lat-brandonu:

[root@lat-brandonu ~]# /usr/local/bin/certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log


Found the following certs:
Certificate Name: ps-brandonu.mrnet.mb.ca
Domains: ps-brandonu.mrnet.mb.ca bdw-brandonu.mrnet.mb.ca bdw6-brandonu.mrnet.mb.ca lat6-brandonu.mrnet.mb.ca
Expiry Date: 2021-02-28 17:22:24+00:00 (VALID: 79 days)
Certificate Path: /etc/letsencrypt/live/ps-brandonu.mrnet.mb.ca/fullchain.pem
Private Key Path: /etc/letsencrypt/live/ps-brandonu.mrnet.mb.ca/privkey.pem


An attempt to get the certificate again and add in lat-brandonu again doesn't seem to fix things:

[root@lat-brandonu ~]# /usr/local/bin/certbot certonly --apache
--cert-name ps-brandonu.mrnet.mb.ca
-d ps-brandonu.mrnet.mb.ca
-d lat-brandonu.mrnet.mb.ca
-d lat6-brandonu.mrnet.mb.ca
-d bdw-brandonu.mrnet.mb.ca
-d bdw6-brandonu.mrnet.mb.ca

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache


You are updating certificate ps-brandonu.mrnet.mb.ca to include new domain(s):

  • lat-brandonu.mrnet.mb.ca

You are also removing previously included domain(s):
(None)

Did you intend to make this change?


(U)pdate cert/(C)ancel: u
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for lat-brandonu.mrnet.mb.ca
Waiting for verification...
Challenge failed for domain lat-brandonu.mrnet.mb.ca
http-01 challenge for lat-brandonu.mrnet.mb.ca
Cleaning up challenges

IMPORTANT NOTES:

  • Congratulations! Your certificate and chain have been saved at:
    /etc/letsencrypt/live/ps-brandonu.mrnet.mb.ca/fullchain.pem
    Your key file has been saved at:
    /etc/letsencrypt/live/ps-brandonu.mrnet.mb.ca/privkey.pem
    Your cert will expire on 2021-03-10. To obtain a new or tweaked
    version of this certificate in the future, simply run certbot
    again. To non-interactively renew all of your certificates, run
    "certbot renew"

  • The following errors were reported by the server:

    Domain: lat-brandonu.mrnet.mb.ca
    Type: unauthorized
    Detail: Invalid response from
    http://lat-brandonu.mrnet.mb.ca/.well-known/acme-challenge/5PoBy2d8Wndel9Bs1vx_6szmmgebuC-o6ORPND75rZQ
    [2001:410:2000:2002::101]: "\n\n404 Not
    Found\n\n

    Not Found

    \n<p"

    To fix these errors, please make sure that your domain name was
    entered correctly and the DNS A/AAAA record(s) for that domain
    contain(s) the right IP address.

  • If you like Certbot, please consider supporting our work by:

    Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate
    Donating to EFF: https://eff.org/donate-le

So the renewal/update of the certificate, attempting to add lat-brandonu.mrnet.mb.ca fails. Is there any way to get a more detailed look at what that Failed Response means?

From a host out on the Internet I know the server is up. Using "nmap" to fetch http headers for lat-brandonu.mrnet.mb.ca for both IPv4 and IPv6:

Nmap done: 1 IP address (1 host up) scanned in 1.12 seconds
Starting Nmap 7.80 ( https://nmap.org ) at 2020-12-10 11:12 CST
Nmap scan report for lat6-brandonu.mrnet.mb.ca (2001:410:2000:2002::101)
Host is up (0.039s latency).
rDNS record for 2001:410:2000:2002::101: lat-brandonu.mrnet.mb.ca

PORT STATE SERVICE
80/tcp open http
| http-headers:
| Date: Thu, 10 Dec 2020 18:12:01 GMT
| Server: Apache
| Location: https://lat6-brandonu.mrnet.mb.ca/
| Content-Length: 242
| Connection: close
| Content-Type: text/html; charset=iso-8859-1
|
|_ (Request type: GET)
443/tcp open https
| http-headers:
| Date: Thu, 10 Dec 2020 18:12:01 GMT
| Server: Apache
| Strict-Transport-Security: max-age=31536000; includeSubDomains
| Protocols: h2 http/1.1
| Content-Security-Policy: default-src 'self' ; script-src 'self' 'unsafe-eval' 'unsafe-inline' ; img-src 'self' 'unsafe-inline' data: ; style-src 'self' 'unsafe-inline' ; connect-src *
| X-Frame-Options: DENY
| X-XSS-Protection: 1; mode=block
| X-Content-Type-Options: nosniff
| Connection: close
| Content-Type: text/html; charset=utf-8
|
|_ (Request type: HEAD)

This is an Apache 2.4 server running under Centos 7:

The application itself seems to be working fine and I can use all of the host names from a web browser, I just cannot update/renew part of the certificate. In looking at the debug log it appears some configuration changes are made to the apache configuration - but only for the lat-brandonu challenge. Could those be "broken?" somehow?

Something else I'm missing?

1 Like

In situations like these it's sometimes the case that there is some weirdness in the Apache configuration which Certbot stumbles upon, and fails to setup the challenge properly.

A good place to start is to post the output of:

sudo httpd -t -D DUMP_VHOSTS

One of the more common mistakes is if lat-brandonu.mrnet.mb.ca is mapped in more than a single VirtualHost.

3 Likes

Agreed, Apache will try to run at all costs (even under extremely poor configuration conditions).
So that is a great place to start looking.
If you don't see anything obvious, you might want to pay closer attention to the details.
Maybe that vhost config fails to handle the .well-known challenge requests as the others do (look for TYPOs, any missing line(s) or just recopy that code part from a known working vhost).
You can easily test this access by placing a test file in the expected challenge location and see if it can be reached from the Internet (and from both IPv6 and IPv4).
[You can also place another test file in a working vhost path for comparison]

1 Like

That was what I needed. I was able to see my configuration was not being processed first and didn't have the results I had expected:
irtualHost configuration:
*:80 is a NameVirtualHost
default server lat-brandonu.mrnet.mb.ca (/etc/httpd/conf.d/apache-toolkit_web_gui.conf:48)
port 80 namevhost lat-brandonu.mrnet.mb.ca (/etc/httpd/conf.d/apache-toolkit_web_gui.conf:48)
port 80 namevhost lat-brandonu.mrnet.mb.ca (/etc/httpd/conf.d/mrnet-perf-sonar-ssl.conf:54)
wild alias *-brandonu.mrnet.mb.ca
*:443 lat-brandonu.mrnet.mb.ca (/etc/httpd/conf.d/mrnet-perf-sonar-ssl.conf:73)

After renaming the file from mrnet-perf-sonar-ssl.conf to 00-mrnet-perf-sonar-ssl.conf and adding a SererName clause I got what it's supposed to be:
VirtualHost configuration:
*:443 ps-brandonu.mrnet.mb.ca (/etc/httpd/conf.d/00-mrnet-perf-sonar-ssl.conf:74)
*:80 is a NameVirtualHost
default server ps-brandonu.mrnet.mb.ca (/etc/httpd/conf.d/00-mrnet-perf-sonar-ssl.conf:54)
port 80 namevhost ps-brandonu.mrnet.mb.ca (/etc/httpd/conf.d/00-mrnet-perf-sonar-ssl.conf:54)
wild alias *-brandonu.mrnet.mb.ca
port 80 namevhost lat-brandonu.mrnet.mb.ca (/etc/httpd/conf.d/apache-toolkit_web_gui.conf:48)

Certificates now renew without a problem and the lat-brandonu alias was added back into the certificate.

Thank You.

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.