I ran this command: sudo certbot certonly --manual -d my.domain.com --dry-run
It produced this output:
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator manual, Installer None
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for my.domain.com
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
NOTE: The IP of this machine will be publicly logged as having requested this
certificate. If you're running certbot in manual mode on a machine that is not
your server, please ensure you're okay with that.
Are you OK with your IP being logged?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: y
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Create a file containing just this data:
8oBbsvmx4qhVIHdUCg-OLcyV_wrFa4TCBPdckI873Jg.mHUkueJuZCLXoeIboZRRwtEzPYfxXyii3Wml-LDIOzo
And make it available on your web server at this URL:
http://my.domain.com/.well-known/acme-challenge/8oBbsvmx4qhVIHdUCg-OLcyV_wrFa4TCBPdckI873Jg
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Press Enter to Continue
Waiting for verification...
Challenge failed for domain my.domain.com
http-01 challenge for my.domain.com
Cleaning up challenges
Some challenges have failed.
IMPORTANT NOTES:
- The following errors were reported by the server:
Domain: my.domain.com
Type: connection
Detail: xxx.xxx.xxx.xxx: Fetching
http://my.domain.com/.well-known/acme-challenge/8oBbsvmx4qhVIHdUCg-OLcyV_wrFa4TCBPdckI873Jg:
Error getting validation data
My web server is (include version): nginx/1.18.0 (Ubuntu 20.04)
I can login to a root shell on my machine.
I'm NOT using a control panel to manage my site.
The version of my client is: certbot 0.40.0
Issue:
curl http://my.domain.com/.well-known/...yields(52) Empty reply from server curl https://my.domain.com/.well-known/...yields8oBbsvmx....Wml-LDIOzo
Hypothesis:
port 80 must be blocked above web server level, some time after creating the initial cert. I have no access to those settings.
Question:
Is it possible to have certbot check https instead of http?
Nginx Config:
server {
server_name my.domain.com;
location ^~ /.well-known/acme-challenge/ {
alias /var/www/acme-challenge/;
allow all;
default_type "text/plain";
}
location / {
# First attempt to serve request as file, then
# as directory, then fall back to displaying a 404.
# try_files $uri $uri/ @rewrites;
proxy_pass http://xxx.xxx.xxx.xxx; # Not localhost
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
# other settings
}
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/my.domain.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/my.domain.com/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}
server {
if ($host = my.domain.com) {
return 301 https://$host$request_uri;
} # managed by Certbot
listen 80 proxy_protocol;
server_name my.domain.com;
return 404; # managed by Certbot
}
The HTTP->HTTPS redirect would be enough but requests to port 80 (HTTP) are being rejected by something.
You might try changing your
listen 80 proxy_protocol;
to just:
listen 80;
It doesn't look like you need proxy_protocol and maybe it is causing a problem.
We should really look at your entire nginx config. I would like to see if there is some other server block interfering (like one based on IP and not SNI).
Can you post the (long) output of this command. An upper case T is essential
Thanks everyone.. after a lot of trial and error, I have found my problem..
I found an immortal nginx that was blocking certbot from authenticating. It is immortal because after stopping the service and killing the process, it would start itself back up again.
After finding the magic bullet pkill nginx, I managed to allow cerbot to 2: Spin up a temporary webserver (standalone) and renew the certificate successfully.
Now I just need to figure out what inside nginx would stop certbot from verifying the challenge (error 400)..
It probably never actually stopped when all other nginx processes were stopped.
Those types of processes are usually referred to as zombie processes - ones that no longer can be controlled and continue to operate until killed OR the system is rebooted.
"Our implementation of the HTTP-01 challenge follows redirects, up to 10 redirects deep. It only accepts redirects to “http:” or “https:”, and only to ports 80 or 443. It does not accept redirects to IP addresses. When redirected to an HTTPS URL, it does not validate certificates (since this challenge is intended to bootstrap valid certificates, it may encounter self-signed or expired certificates along the way).
The HTTP-01 challenge can only be done on port 80. Allowing clients to specify arbitrary ports would make the challenge less secure, and so it is not allowed by the ACME standard."
As a historical note, it used to be possible to do an equivalent challenge starting on port 443, but it was discovered that this caused problems for some shared hosting setups (allowing one customer to obtain certificates for another customer), so that functionality was intentionally disabled. (To be more precise, there were two different versions of simple port 443 challenges, both of which ultimately had some unduly high risks for shared hosting environments; one of them was removed before Let's Encrypt even launched, while the other one was removed in 2019. Details of both are in section 4.3 of the Let's Encrypt academic paper.)
As others have mentioned, you can do the TLS-ALPN challenge on port 443 now, but it's kind of complicated and most people currently wouldn't be able to use it unless their web servers directly provide support for it.
.
If a process has the "Z" status in (h)top, only a reboot gets rid of it unfortunately..