Certbot not renewing over nginx

My domain is: breeze.ggt.bz, co.ggt.bz, several other subdomains

I ran this command: sudo certbot renew

It produced this output:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/breeze.ggt.bz.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Renewing an existing certificate for breeze.ggt.bz

Certbot failed to authenticate some domains (authenticator: nginx). The Certificate Authority reported these problems:
  Domain: breeze.ggt.bz
  Type:   unauthorized
  Detail: During secondary validation: 45.137.206.17: Invalid response from https://nyc1.breeze.ggtyler.dev/: "<!DOCTYPE html><html><head><meta name=\"viewport\" content=\"width=device-width, initial-scale=1\"><title>About | BreezeWiki</title>"

Hint: The Certificate Authority failed to verify the temporary nginx configuration changes made by Certbot. Ensure the listed domains point to this nginx server and that it is accessible from the internet.

...

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/co.ggt.bz.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Renewing an existing certificate for co.ggt.bz

Certbot failed to authenticate some domains (authenticator: nginx). The Certificate Authority reported these problems:
  Domain: co.ggt.bz
  Type:   unauthorized
  Detail: 45.137.206.17: Invalid response from https://co.ggt.bz/.well-known/acme-challenge/4eDu__4siL33f42YysHEMx1PGeoI3NXtDQxg1xMgu_Q: 500

Hint: The Certificate Authority failed to verify the temporary nginx configuration changes made by Certbot. Ensure the listed domains point to this nginx server and that it is accessible from the internet.

Failed to renew certificate co.ggt.bz with error: Some challenges have failed.

My web server is (include version): nginx v1.18.0

The operating system my web server runs on is (include version): Ubuntu 22.04

My hosting provider, if applicable, is: Royale Hosting

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): 3.0.1


Here are the two configs for the domains I'm trying to renew. There are some others that aren't renewing but they have near identical configs and the same errors.

server {
    server_name breeze.ggt.bz;
    #error_log /var/log/nginx/ggt.bz.error.log;
    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/breeze.ggt.bz/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/breeze.ggt.bz/privkey.pem;
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
    location / {
        proxy_pass http://localhost:58008;
        proxy_redirect off;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_http_version 1.1;
    }
}
server {
    if ($host = breeze.ggt.bz) {
        return 301 https://$host$request_uri;
    }
    server_name breeze.ggt.bz;
    #error_log /var/log/nginx/ggt.bz.error.log;
    listen 80;
    return 301 https://$host$request_uri;
}
server {
    server_name co.ggt.bz;
    #error_log /var/log/nginx/ggt.bz.error.log;
    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/co.ggt.bz/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/co.ggt.bz/privkey.pem;
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
    location / {
        proxy_pass http://localhost:58008;
        proxy_redirect off;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_http_version 1.1;
    }
}
server {
    if ($host = co.ggt.bz) {
        return 301 https://$host$request_uri;
    }
    server_name co.ggt.bz;
    #error_log /var/log/nginx/ggt.bz.error.log;
    listen 80;
    return 301 https://$host$request_uri;
}

Hello @ggtyler,

Using the online tool Let's Debug yields results of "OK" here
https://letsdebug.net/breeze.ggt.bz/2323678?debug=y

But there is useful information in there, the redirects

HTTPCheck
Debug
Requests made to the domain
Request to: breeze.ggt.bz/45.137.206.17, Result: [Address=45.137.206.17,Address Type=IPv4,Server=nginx/1.18.0 (Ubuntu),HTTP Status=301,Number of Redirects=4,Final HTTP Status=200], Issue:
Trace:
@0ms: Making a request to http://breeze.ggt.bz/.well-known/acme-challenge/letsdebug-test (using initial IP 45.137.206.17)
@0ms: Dialing 45.137.206.17
@223ms: Server response: HTTP 301 Moved Permanently
@223ms: Received redirect to https://breeze.ggt.bz/.well-known/acme-challenge/letsdebug-test
@223ms: Dialing 45.137.206.17
@563ms: Server response: HTTP 302 Found
@563ms: Received redirect to https://ggt.bz/breeze
@598ms: Dialing 45.137.206.17
@935ms: Server response: HTTP 302 Found
@935ms: Received redirect to https://breeze.ggtyler.dev
@1172ms: Dialing 45.137.206.17
@1509ms: Server response: HTTP 301 Moved Permanently
@1509ms: Received redirect to https://nyc1.breeze.ggtyler.dev/
@1543ms: Dialing 45.137.206.17
@1886ms: Server response: HTTP 200 OK 

Starting here http://breeze.ggt.bz/.well-known/acme-challenge/letsdebug-test ends up several redirects later here https://nyc1.breeze.ggtyler.dev/. Then returns a response of HTTP 200 OK for something likely nonexistent 404 Not Found - HTTP | MDN

Edit

And a screen shot of this online tool https://www.redirect-checker.org/ output

1 Like

The nginx authenticator inserts temp code into your server block to avoid redirects.

Yet, we see a different URL in the error which means there was a redirect.

Similarly, with your co.ggt.bz domain it shows an HTTPS URL in the error so your server also redirected that domain.

The nginx config you show us does not look like it is the one replying to HTTP requests for the IP address in your DNS.

Would you show the output of this:

curl -4 https://ifconfig.io

And also at least all the server blocks from this. An upper case T is essential

sudo nginx -T
1 Like

Yes, ggt.bz is a shortener that redirects to [server].[service].ggtyler.dev. This also occurs on one of my other config that handles a redirect:

server {
    listen 80;
    listen [::]:80;
    listen 443 ssl;
    listen [::]:443 ssl;

    server_name iv.ggtyler.dev;
    access_log off;

    ssl_certificate /etc/letsencrypt/live/iv.ggtyler.dev/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/iv.ggtyler.dev/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

    return 301 https://nyc1.iv.ggtyler.dev$request_uri; 
}
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/iv.ggtyler.dev.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Renewing an existing certificate for iv.ggtyler.dev

Certbot failed to authenticate some domains (authenticator: nginx). The Certificate Authority reported these problems:
  Domain: iv.ggtyler.dev
  Type:   unauthorized
  Detail: 45.137.206.17: Invalid response from https://nyc1.iv.ggtyler.dev/sorry.html: "<!DOCTYPE html><html lang=\"en\"> <head><meta charset=\"UTF-8\"><meta name=\"viewport\" content=\"width=device-width\"><link rel=\"icon\" "

Hint: The Certificate Authority failed to verify the temporary nginx configuration changes made by Certbot. Ensure the listed domains point to this nginx server and that it is accessible from the internet.

Failed to renew certificate iv.ggtyler.dev with error: Some challenges have failed.

So does certbot not handle redirects properly?

Certbot isn't the one following the redirects. The Let's Encrypt auth server does. Redirects are not ideal but are supported: Challenge Types - Let's Encrypt

But, please see my post describing that with the --nginx plugin authenticator we shouldn't be seeing redirects at all.

2 Likes

Yes, sorry, you posted your reply right before I did mine.

IP is easy: 45.137.206.17

I have a lot of server blocks though. For some reason, it's not letting me write to a file?

➜  ~ sudo nginx -T >> nginx-blocks.log 
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
1 Like

Ah, nevermind, the output just looked like it wasn't working.
nginx-blocks.txt (261.5 KB)

1 Like

Try running this and upload the log from /var/log/letsencrypt/letsencrypt.log

We can probably tell what is happening from that

sudo certbot renew --dry-run --cert-name co.ggt.bz

Are you sure the incoming HTTP (port 80) requests actually arrive at that nginx?

Was nginx running prior to running Certbot?

1 Like
  1. Yes, running sudo lsof -i :80 shows nginx listening on that port.
  2. Yes.

letsencrypt.txt (39.2 KB)

1 Like

Oh, I see there is a VERY large number of server blocks :slight_smile:

The Certbot --nginx option makes a temp change to your nginx config and then reloads nginx asynchronously. By default, it only waits 1 second and then tells the LE server to make the challenge.

If your nginx takes longer than 1s to reload it won't have the effective changes by the time the challenge arrives from the LE server and will fail. Which is what I think is happening here.

Try this and show result

sudo certbot renew --dry-run --cert-name co.ggt.bz --nginx-sleep-seconds 5
2 Likes

Yes, haha. I share the server between 4 other people and I have a multitude on stuff running on there, plus the shortening script, plus redirects from the regular subdomains to the server-specific subdomain.

With that being said, I'm still getting the same result:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/co.ggt.bz.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Simulating renewal of an existing certificate for co.ggt.bz

Certbot failed to authenticate some domains (authenticator: nginx). The Certificate Authority reported these problems:
  Domain: co.ggt.bz
  Type:   unauthorized
  Detail: 45.137.206.17: Invalid response from https://co.ggt.bz/.well-known/acme-challenge/yv0mdw_wk7lJkK2-H50qMj_LUI-ZKQYJG9_0Oxg486k: 500

Hint: The Certificate Authority failed to verify the temporary nginx configuration changes made by Certbot. Ensure the listed domains point to this nginx server and that it is accessible from the internet.

Failed to renew certificate co.ggt.bz with error: Some challenges have failed.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
All simulated renewals failed. The following certificates could not be renewed:
  /etc/letsencrypt/live/co.ggt.bz/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

Tried it with 15 seconds too.

1 Like

Hmm. Can you upload the /var/log/letsencrypt/letsencrypt.log from one of these renew failures with extra sleep?

Perhaps there is something in the nginx conf that is confusing Certbot's nginx authenticator.

I will be away for awhile but can look at the log later. Or maybe someone else will see something

2 Likes

Sure (done with 60 seconds): letsencrypt.2.txt (44.1 KB)

And hey, no worries :) It's New Years after all.

We actually go out tomorrow night :slight_smile:

Oh, I think it is because of the stray return 301 ... that is not inside a location block.

Remove the line below your listen 80; because the 'if' statement handles redirect anyway. The return not inside a location block is overriding the return 200 ... that is inside a location block.

This is your server block from the log after the temp change. Just fyi

server {rewrite ^(/.well-known/acme-challenge/.*) $1 break; # managed by Certbot
    if ($host = co.ggt.bz) {
        return 301 https://$host$request_uri;
    }
    server_name co.ggt.bz;
    #error_log /var/log/nginx/ggt.bz.error.log;
    listen 80;
    return 301 https://$host$request_uri;
location = /.well-known/acme-challenge/Y33Yh1Fei8gHfyT-DaXEYohmZccjQM_g734xVAaSN04{default_type text/plain;return 200 Y33Yh1Fei8gHfyT-DaXEYohmZccjQM_g734xVAaSN04.lbrqEaZlbnGQ1RxX4qFxjhlB_b96-zY8DLtOTQvYWZI;} # managed by Certbot

}

3 Likes

Looks like that worked :D

Now I'll just have to remove that from 20ish configs... joy...

Oh yeah, before I forget, that error was also happening on iv.ggtyler.dev and that had a different config that I posted earlier.

1 Like

Actually nevermind, I did absolutely nothing to that nginx config and it renews now (??)

Anyways thank you ^^

3 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.