My domain is: rm.jeremymouzin.com
I ran this command: sudo certbot --dry-run certonly --nginx -d rm.jeremymouzin.com -v
It produced this output:
Plugins selected: Authenticator nginx, Installer nginx
Certificate is due for renewal, auto-renewing...
Simulating renewal of an existing certificate for rm.jeremymouzin.com
Performing the following challenges:
http-01 challenge for rm.jeremymouzin.com
Waiting for verification...
Challenge failed for domain rm.jeremymouzin.com
http-01 challenge for rm.jeremymouzin.com
Certbot failed to authenticate some domains (authenticator: nginx). The Certificate Authority reported these problems:
Domain: rm.jeremymouzin.com
Type: unauthorized
Detail: 2001:4b98:dc0:43:f816:3eff:fe3d:54fb: Invalid response from https://rm.jeremymouzin.com/.well-known/acme-challenge/dPTXpAKiUWnt9j2PS-anPtHNDvOHrUYNnFGErlTScB8: 404
Hint: The Certificate Authority failed to verify the temporary nginx configuration changes made by Certbot. Ensure the listed domains point to this nginx server and that it is accessible from the internet.
Cleaning up challenges
Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
My web server is (include version): nginx/1.22.1
The operating system my web server runs on is (include version): Debian 12.5 bookworm
My hosting provider, if applicable, is: gandi.net
I can login to a root shell on my machine (yes or no, or I don't know): yes
I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no
The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 2.11.0 (installed via snapd)
The problem
I have a really weird problem: if I try to renew my certificate with the command mentioned before it doesn't work BUT if I try to renew with --dry-run AND --debug-challenges AND I wait for 5 seconds before typing ENTER the dry-run works! So I can't even renew it manually because there is no wait before validation (or maybe there is an option for that? anyway it should work without having to wait!)
Additional informations
Yesterday I read almost all posts in this forum that were relevant for my issue so here are some information you may ask me later.
In my DNS I have an A and AAAA records that points to my subdomain to the IP of my VPS and I have correctly configured nginx for both IPv4 and IPv6 (without forgetting to use listen [::]:80 for IPv6).
Here is my nginx config:
# Redirection from http:// to https://
server {
listen 80;
listen [::]:80;
server_name rm.jeremymouzin.com;
return 301 https://$host$request_uri;
}
server {
listen 443 ssl http2; # managed by Certbot
listen [::]:443 ssl http2;
ssl_certificate /etc/letsencrypt/live/rm.jeremymouzin.com/fullchain.pem; # managed by Certbot
ssl_trusted_certificate /etc/letsencrypt/live/rm.jeremymouzin.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/rm.jeremymouzin.com/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
root /var/www/rm.jeremymouzin.com;
server_name rm.jeremymouzin.com;
# Disable unwanted HTTP methods
if ($request_method !~ ^(GET|HEAD|POST)$) {
return 405;
}
# ModSecurity-nginx dynamic module
modsecurity on;
modsecurity_rules_file /etc/nginx/modsec_includes.conf;
location / {
proxy_pass http://localhost:3334;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_cache_bypass $http_upgrade;
}
location /security.txt {
return 301 https://$host/.well-known/security.txt;
}
}
I already tried to remove the modsecurity section and the # Disable unwanted HTTP methods, it still doesn't work.
I have NO geoblocking feature from firewall / whatever as you can see here: https://check-host.net/check-dns?host=rm.jeremymouzin.com&csrf_token=a971351a9e70b672b30369f098efc96a8160e88e
My ports 80 and 443 are opened and reachable from the internet:
$ nmap -p80,443 rm.jeremymouzin.com
Starting Nmap 7.92 ( https://nmap.org ) at 2024-06-08 07:09 CEST
Nmap scan report for rm.jeremymouzin.com (46.226.107.169)
Host is up (0.027s latency).
Other addresses for rm.jeremymouzin.com (not scanned): 2001:4b98:dc0:43:f816:3eff:fe3d:54fb
rDNS record for 46.226.107.169: xvm-107-169.dc0.ghst.net
PORT STATE SERVICE
80/tcp open http
443/tcp open https
Nmap done: 1 IP address (1 host up) scanned in 0.35 seconds
My certbot certificates:
Saving debug log to /var/log/letsencrypt/letsencrypt.log
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Found the following certs:
Certificate Name: rm.jeremymouzin.com
Serial Number: 4a8cd248bd22f1363e39d6f053ff6f299cd
Key Type: ECDSA
Domains: rm.jeremymouzin.com
Expiry Date: 2024-06-27 19:38:50+00:00 (VALID: 19 days)
Certificate Path: /etc/letsencrypt/live/rm.jeremymouzin.com/fullchain.pem
Private Key Path: /etc/letsencrypt/live/rm.jeremymouzin.com/privkey.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(I cut out other certificates for other domains that are not relevant here and that don't renew either with the same issue!)
I also checked the nginx temporary configuration file created by certbot while using the following command: sudo certbot --dry-run certonly --nginx -d rm.jeremymouzin.com -v --debug-challenge, and as soon as it asks me to type enter, I take a look at the subdomain nginx config file to check what's new, here is what it produces:
$ cat /etc/nginx/sites-available/rm.jeremymouzin.com
# Redirection from http:// to https://
server {rewrite ^(/.well-known/acme-challenge/.*) $1 break; # managed by Certbot
listen 80;
listen [::]:80;
server_name rm.jeremymouzin.com;
return 301 https://$host$request_uri;
location = /.well-known/acme-challenge/Y5alQPckDhfQdYXS2XrOONBswQ0Q7J1dL1HweJAJbtQ{default_type text/plain;return 200 Y5alQPckDhfQdYXS2XrOONBswQ0Q7J1dL1HweJAJbtQ.IPOnz7gQvri-q1k781JN7bi0RYDIHgcqeoOdhhVkgVM;} # managed by Certbot
}
server {rewrite ^(/.well-known/acme-challenge/.*) $1 break; # managed by Certbot
listen 443 ssl http2; # managed by Certbot
listen [::]:443 ssl http2;
ssl_certificate /etc/letsencrypt/live/rm.jeremymouzin.com/fullchain.pem; # managed by Certbot
ssl_trusted_certificate /etc/letsencrypt/live/rm.jeremymouzin.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/rm.jeremymouzin.com/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
root /var/www/rm.jeremymouzin.com;
server_name rm.jeremymouzin.com;
# Disable unwanted HTTP methods
if ($request_method !~ ^(GET|HEAD|POST)$) {
return 405;
}
# ModSecurity-nginx dynamic module
modsecurity on;
modsecurity_rules_file /etc/nginx/modsec_includes.conf;
location / {
proxy_pass http://localhost:3334;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_cache_bypass $http_upgrade;
}
location /security.txt {
return 301 https://$host/.well-known/security.txt;
}
location = /.well-known/acme-challenge/Y5alQPckDhfQdYXS2XrOONBswQ0Q7J1dL1HweJAJbtQ{default_type text/plain;return 200 Y5alQPckDhfQdYXS2XrOONBswQ0Q7J1dL1HweJAJbtQ.IPOnz7gQvri-q1k781JN7bi0RYDIHgcqeoOdhhVkgVM;} # managed by Certbot
}
It seems to add the needed stuff for the verification (location = /.well-known/acme-challenge/XXXX). In fact I can reach to my browser and go to the http URL http://rm.jeremymouzin.com/.well-known/acme-challenge/Y5alQPckDhfQdYXS2XrOONBswQ0Q7J1dL1HweJAJbtQ, it gives me the expected output! And I can see in the Network tab that it redirects correctly first to https as expected. (I also tried to remove the https redirection from the nginx config and reloaded nginx and retried to renew the cert to see if it could help, it did not).
The weird thing is that when I hit enter, the test is successful! As if it needed some time to get the verification done right! I tried with a 3 seconds delay and it doesn't work. It needs at least 5 seconds between the time it ask me this (to type enter):
[...]
Challenges loaded. Press continue to submit to CA.
The following URLs should be accessible from the internet and return the value
mentioned:
URL:
http://rm.jeremymouzin.com/.well-known/acme-challenge/Nuw-Z7DWAecUqGeIIF_OO82jsKsmRhS28zg9P0_Po6U
Expected value:
Nuw-Z7DWAecUqGeIIF_OO82jsKsmRhS28zg9P0_Po6U.IPOnz7gQvri-q1k781JN7bi0RYDIHgcqeoOdhhVkgVM
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Press Enter to Continue
and when I hit enter.
Maybe I issued too much certificates or something? (I didn't use the --dry-run argument when doing some configuration on my server for my project and I did issue several certificates for the same subdomain, sorry about that!), see crt.sh | rm.jeremymouzin.com. Could it be an issue?
I can't figure out what makes this delay necessary to make the renewal verification work?! I can't figure out what I did wrong.
Oh and I did try already also to make a /.well-known/acme-challenge/ subdirectory with some test.txt file in it, it's perfectly reachable from my browser, I tried to chmod 777 acme-challenge (why not) didn't work either.
Based on the nginx config, the file system should not be related to the verification process as it gets directly the location AND the result from the certbot added directive (location = /.well-known/acme-challenge/Y5alQPckDhfQdYXS2XrOONBswQ0Q7J1dL1HweJAJbtQ{default_type text/plain;return 200 Y5alQPckDhfQdYXS2XrOONBswQ0Q7J1dL1HweJAJbtQ.IPOnz7gQvri-q1k781JN7bi0RYDIHgcqeoOdhhVkgVM;}) but as I'm not an nginx expert I tried anyway, it didn't work.
I'm quite out of options and I hope you will point out something I did miss that is completely stupid but that will make everything work. Thanks for your help.
but I thought it would be a good idea to summarize everything I learned for others and to tell you all the things I tried already to avoid back and forth messages.