Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.
My domain is:
various, but one example is “ask.springshare.com”
I ran this command:
sudo certbot-auto certonly --webroot --webroot-path /springshare/www -d ask.springshare.com
It produced this output:
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for ask.springshare.com
Using the webroot path /springshare/www for all unmatched domains.
Waiting for verification…
Challenge failed for domain ask.springshare.com
http-01 challenge for ask.springshare.com
Cleaning up challenges
Some challenges have failed.
IMPORTANT NOTES:
-
The following errors were reported by the server:
Domain: ask.springshare.com
\n <meta http-equiv=\"X-UA-Compatible\" content=\"I"
Type: unauthorized
Detail: Invalid response from
http://ask.springshare.com/.well-known/acme-challenge/ZztvCEq56Cx7Gf8sruSxwEaITtjgPPp6o2gQ1hNYgHk
[54.83.125.71]: "\n<html lang=“en”>\n \nTo fix these errors, please make sure that your domain name was
entered correctly and the DNS A/AAAA record(s) for that domain
contain(s) the right IP address.
My web server is (include version):
nginx version: nginx/1.18.0
The operating system my web server runs on is (include version):
Centos7 (Linux 3.10.0-1062.18.1.el7.x86_64)
My hosting provider, if applicable, is:
AWS
I can login to a root shell on my machine (yes or no, or I don’t know):
yes
I’m using a control panel to manage my site (no, or provide the name and version of the control panel):
no
The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot):
certbot 1.3.0
We are using LE in a centralized fashion, where certificates are requested from a single server (called the cert-manager) and then distributed to one or more application servers. To make that work, all the app-servers are configured to forward (proxy_pass) challenge requests back to the cert-manager, where the request originated. Note that this process changes the hostname associated with the challenge, which originally came to ask.springshare.com/.well-known but is forwarded to the cert-manager as my-cert-manager-domain.com/.well-known
This approach has worked for quite some time, but as of yesterday all our challenges are failing. I used the debug command to “pause” the request process, and I was able to verify that the CONF file generated by LE / certbot for NGINX contains a “server_name” property that specifies which domain the challenge is listening on. E.g.
server{listen 80;server_name ask.springshare.com;root /var/lib/letsencrypt/http_01_nonexistent;location = /.well-known/acme-challenge/NdD8-_1VoRiX5XyB_NSpOxXjRY0E-LPcAGbY{default_type text/plain;return 200 NdD8-_1VoRiX5XyB_NSpOxXjRY0E-LPcAGbY.5RP4EzaOBQEqt_rqy_ZREzLnJBEq2sUTGeElaU6iryw;}}
As mentioned above, forwarding the request back to the cert-manager causes the hostname to change, so the clause generated by certbot is not matched, and the challenge fails. If I hand edit the CONF file to include a “server_name my-cert-manager-domain.com;” clause, the challenge will succeed. So I am pretty confident that the generated NGINX CONF include is the source of my problem.
I don’t see how our process ever could have worked if the server_name clause had been included in the NGINX CONF from the beginning. So I am wondering if the inclusion of the server_name clause is new for LE / certbot. If so, is there any work around that would allow our approach to work once more?
Thanks in advance for any help!