I am having issues with using certbot-auto on my server (nginx, ubuntu, details below) for renewing certificates, as it fails in verifying the domain (details below).
I can do the renewal properly by verifying my domain manually.
I have also copied the nginx config modified by certbot-auto while it was trying to verify the domain, and then went through the manual process using the same config (with modified challenge tokens), and that works as well.
This is the modification certbot makes to the nginx config during the challenge:
server {rewrite ^(/.well-known/acme-challenge/.*) $1 break; # managed by Certbot
listen 80;
server_name bf.ecomply.io;
location = /.well-known/acme-challenge/<challenge_url>{default_type text/plain;return 200 <challenge_token>;} # managed by Certbot
}
The issue seems to be that certbot-auto tries to fetch the proper response from https://bf.ecomply.io/.well-known/acme-challenge/<challenge_url> while the auto-generated modified config serves the appropriate challenge token on https://bf.ecomply.io/.well-known/acme-challenge/<challenge_url>.
Is this a bug in latest version of certbot (or the nginx plugin)? Or am I doing something wrong?
My domain is:
bf.ecomply.io
I ran this command:
sudo certbot
It produced this output:
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for bf.ecomply.io
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. bf.ecomply.io (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from https://bf.ecomply.io/.well-known/acme-challenge/<challenge_url> [82.165.97.140]: "<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><cen"
IMPORTANT NOTES:
- The following errors were reported by the server:
Domain: bf.ecomply.io
Type: unauthorized
Detail: Invalid response from
https://bf.ecomply.io/.well-known/acme-challenge/<challenge_url>
[82.165.97.140]: "<html>\r\n<head><title>502 Bad
Gateway</title></head>\r\n<body
bgcolor=\"white\">\r\n<center><h1>502 Bad
Gateway</h1></center>\r\n<hr><cen"
To fix these errors, please make sure that your domain name was
entered correctly and the DNS A/AAAA record(s) for that domain
contain(s) the right IP address.
My web server is (include version):
nginx/1.10.3 (Ubuntu)
The operating system my web server runs on is (include version):
Ubuntu 16.04.6 LTS
My hosting provider, if applicable, is:
I can login to a root shell on my machine (yes or no, or I don’t know):
yes
I’m using a control panel to manage my site (no, or provide the name and version of the control panel):
no
The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot):
There is no file bf.ecomply.io in the aforementioned folder.
Yes seems to be the case. Curious thing is I can interrupt the automatic process, and continue with doing it manually, and everything works. i.e. I can initiate the process, copy the modified nginx config (or just the modified segment I put in the original post), then do manual verification using that bit, and then verification works fine as the request is not redirected to https then.
Another curious thing that happens is that if I do as described, then renewing the certificates of a domain through the automatic process starts to work properly again.
P.S. I've had this issue on multiple domains on the same server, and I've fixed it for them via the aforementioned process. However I intentionally did not fix bf.ecomply.io to keep it as a viable test case for the aforementioned behavior.
Do you have any guesses as to why this is happening? I know I can renew certificates manually, however that takes way more time and energy for multiple domains, and since the automatic process always worked before, I am wondering what has happened here.
Last certificate is from 2019-05-10, there is no newer certificate visible.
If you use the standard nginx, a redirect is skipped because of the code certbot inserts. But your error message shows https, that's not possible. Use --webroot instead.
considering the usual 90days of validation, this matches the date I manually renewed the certificate on the domain.
That's what I meant. I can use certbot --manual, and use the code snippet that certbot inserts into nginx configs to properly conduct the verification. But for whatever reason certbot is unable to do this on its own.
Anyways --webroot also is not efficient. I suspect if this happens again I will add a script to gloss over the steps in the process that certbot fails at. Though still cannot understand why that is, i.e. is it a certbot bug or server misconfiguration.
Yes this is another domain name. It had the same issue, which I could work-around via described method (using certbot --manual and using the nginx config snippet inserted by certbot for verification)
bf.ecomply.io is another domain name on the same machine with the issue. I purposefully did not utilize the work-around I used for lapras.ecomply.io for it to be able to study the issue further.
The Let's Encrypt validator bot always starts with the http:// version, but is willing to follow a redirect to https:// if one is encountered. Do you have anything that would be generating such a redirect?
Could you share the Certbot output from a failed renewal attempt?
Not one that I could find. I checked all the nginx configs, and even tried manually verifying by just adding the code-snippet certbot generates in nginx configs, and succeeded, which should mean that there is no such redirect (or that in manual verification certbot DOES NOT follow redirects to https but in automatic verification it does, can that be the case?):
server {rewrite ^(/.well-known/acme-challenge/.*) $1 break; # managed by Certbot
listen 80;
server_name bf.ecomply.io;
location = /.well-known/acme-challenge/<challenge_url>{default_type text/plain;return 200 <challenge_token>;} # managed by Certbot
}
sure:
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for bf.ecomply.io
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. bf.ecomply.io (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from https://bf.ecomply.io/.well-known/acme-challenge/<challenge_url> [82.165.97.140]: "<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><cen"
IMPORTANT NOTES:
- The following errors were reported by the server:
Domain: bf.ecomply.io
Type: unauthorized
Detail: Invalid response from
https://bf.ecomply.io/.well-known/acme-challenge/<challenge_url>
[82.165.97.140]: "<html>\r\n<head><title>502 Bad
Gateway</title></head>\r\n<body
bgcolor=\"white\">\r\n<center><h1>502 Bad
Gateway</h1></center>\r\n<hr><cen"
To fix these errors, please make sure that your domain name was
entered correctly and the DNS A/AAAA record(s) for that domain
contain(s) the right IP address.
Isn't this the proxied server not responding properly? Anyhow I highly suspect it does not directly relate to the issue at hand since in that case it should have not been working with manual verification using certbot's auto-generated nginx configs.
Certbot isn't the one verifying the challenges—that's the Let's Encrypt bot, which doesn't run on your computer but in Let's Encrypt datacenters. They don't know¹ what kind of Certbot verification mode or method you're using.
Do you have web server logs and proxy logs related to these requests?
Âą Certbot does tell the Let's Encrypt service that it's Certbot and what authenticator it's running with, for debugging and statistical purposes, but Let's Encrypt doesn't routinely use this information for any purpose and it doesn't affect how challenges are validated.
There is no actual proxy server behind this. We took off the actual proxy a while back (thats why I'm using this domain for testing this issue to begin with). But I ran certbot again on the domain and these are nginx access and error logs respectively:
# Error logs
2019/08/02 11:28:56 [error] 9881#9881: *52330 open() "/usr/share/nginx/html/.well-known/acme-challenge/9VeqzLSNEuCrMXM3Z_BYLi74gq1PQiaOfQ5yypK48XI" failed (2: No such file or directory), client: 3.14.255.131, server: baylda-fragebogen.ecomply.io, request: "GET /.well-known/acme-challenge/9VeqzLSNEuCrMXM3Z_BYLi74gq1PQiaOfQ5yypK48XI HTTP/1.1", host: "bf.ecomply.io"
2019/08/02 11:28:56 [error] 9881#9881: *52331 open() "/usr/share/nginx/html/.well-known/acme-challenge/9VeqzLSNEuCrMXM3Z_BYLi74gq1PQiaOfQ5yypK48XI" failed (2: No such file or directory), client: 18.197.227.110, server: baylda-fragebogen.ecomply.io, request: "GET /.well-known/acme-challenge/9VeqzLSNEuCrMXM3Z_BYLi74gq1PQiaOfQ5yypK48XI HTTP/1.1", host: "bf.ecomply.io"
2019/08/02 11:28:56 [error] 9881#9881: *52332 open() "/usr/share/nginx/html/.well-known/acme-challenge/9VeqzLSNEuCrMXM3Z_BYLi74gq1PQiaOfQ5yypK48XI" failed (2: No such file or directory), client: 34.222.229.130, server: baylda-fragebogen.ecomply.io, request: "GET /.well-known/acme-challenge/9VeqzLSNEuCrMXM3Z_BYLi74gq1PQiaOfQ5yypK48XI HTTP/1.1", host: "bf.ecomply.io"
2019/08/02 11:28:57 [error] 9881#9881: *52333 open() "/usr/share/nginx/html/.well-known/acme-challenge/9VeqzLSNEuCrMXM3Z_BYLi74gq1PQiaOfQ5yypK48XI" failed (2: No such file or directory), client: 66.133.109.36, server: baylda-fragebogen.ecomply.io, request: "GET /.well-known/acme-challenge/9VeqzLSNEuCrMXM3Z_BYLi74gq1PQiaOfQ5yypK48XI HTTP/1.1", host: "bf.ecomply.io"
Seems like nginx is trying to fallback on some internal default configuration (I couldn't find the address /usr/share/nginx anywhere in actual nginx configuration files) as none of its rules apply to the incoming request.
I have also copied the nginx config modified by certbot-auto while it was trying to verify the domain, and then went through the manual process using the same config (with modified challenge tokens), and that works as well.
How did you get this configuration, and then how did you integrate it into your setup during the manual verification?
Well basically I ran certbot as usual (sudo certbot). While verification (which usually takes a few seconds) I copied the nginx config (e.g. /etc/nginx/sites-available/bf.ecomply.io), which during that period bears this additional segment:
server {rewrite ^(/.well-known/acme-challenge/.*) $1 break; # managed by Certbot
listen 80;
server_name bf.ecomply.io;
location = /.well-known/acme-challenge/<challenge_url>{default_type text/plain;return 200 <challenge_token>;} # managed by Certbot
}
then when the verification fails, I would run manual verification, and when certbot is waiting me to place the proper token on the proper path on my webserver, I would simply paste this snippet back into the same nginx config file, replacing the path and the token with the new values that certbot has provided.
Could we see the whole output of sudo certbot certificates?
I’m also kind of confused that you’re running sudo certbot rather than sudo certbot renew in order to renew your certificates. Does the latter work any better for you, perhaps?
that is mostly because the domains expire at different times and certbot renew, as it stands, does not work with renewal of certificates using domain names:
Currently, the renew verb is capable of either renewing all installed certificates that are due to be renewed or renewing a single certificate specified by its name. If you would like to renew specific certificates by their domains, use the certonly command instead. The renew verb may provide other options for selecting certificates to renew in the future.
renewing all domains at once is also not a good idea for my case since unforeseen issues during renewal process can lead to problematic certificates for domains that did not need a renewal in the first place (though I suspect the worst-case scenario should be certificates simply not being renewed, no piece of software has predictable behavior 100% of the time so its better be safe than sorry).
@joohoi, can you understand why certbot --nginx would fail to renew but manually copying the configuration file that it creates back into the nginx configuration would succeed with certbot --manual?
ok I just did the same operation on a couple of other domains on the same server with the same process and issues. I realized that for domain x.ecomply.io, going through the process as described above results in valid certificates in /etc/letsencrypt/live/x.ecomply.io. however, when editing the nginx config after the verification process (mainly to get the verification snippet out), I realized that the certificate was configured to be read from /etc/letsencrypt/ecomply.io beforehand (which I had to fix).
additionally, I noticed that while sudo certbot reads the nginx configs to list the domains and act accordingly, sudo certbot certonly --manual asks for the domain. is it possible that the issue is with certbot --nginx trying to read the nginx config and subsequently doing something according to the certificates located in /etc/letsencrypt/live/ecomply.io, which causes it to fail, while certbot --manual ignores all that and subsequently succeeds?
p.s. the content of /etc/letsencrypt/live/ecomply.io is also generated by certbot --nginx, though perhaps a relatively older version.