Auto authorisation fails but manual authorisation using --debug-challenges works

Help
1

I have redacted real URL's. However anyone willing to help in private I can share real URL's. Hopefully this is a good start to diagnose. Thanks.

Small note:
This exact same command has been running for over 2 years. But we started facing the authorisation issue recently (1-2 months ago). Only thing that has changed is the no. of sites (currently 152, each site having a subdomain and a main domain).

Letsencrypt Error:

The issue is Letsencrypt authorisation fails when we run this command:
certbot certonly --nginx -d test4.giftcommerce.com -d www.test4.giftcommerce.com --redirect --expand --force-renewal --dry-run

It works when we run it with --debug-challenges and restart nginx manually
certbot certonly --nginx -d test4.giftcommerce.com -d www.test4.giftcommerce.com --redirect --expand --force-renewal --dry-run --debug-challenges

Points to note:

  1. The exact same command works when we add --debug-challenges. Before selecting the "Press enter to continue", we need to restart nginx. And also once certbot command runs successfully, nginx has to be restarted again. Then SSL is installed successfully.

  2. DNS IP is set correctly for both subdomain and main domain.

  3. This exact same command has been running for over 2 years. But we started facing the authorisation issue recently (1-2 months ago). Only thing that has changed is the no. of sites (currently 152, each site having a subdomain and a main domain).

  4. Created this folder manually but seems like Letsencrypt authorisation process generates random file within acme-challenge folder every time.
    Test file created: /var/www/sites/site7/public/.well-known/acme-challenge/test
    URL: https://test4.giftcommerce.com/.well-known/acme-challenge/test

This is the output when I run the command.

root@giftcommerce:/opt/gt# certbot certonly --nginx -d test4.giftcommerce.com -d www.test4.giftcommerce.com --redirect --expand --force-renewal --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Simulating a certificate request for test4.giftcommerce.com and www.test4.giftcommerce.com
Certbot failed to authenticate some domains (authenticator: nginx). The Certificate Authority reported these problems:
Domain: test4.giftcommerce.com
Type: unauthorized
Detail: Invalid response from http://test4.giftcommerce.com/ [178.79.159.104]: "\n\n<html lang="en">\n\n\n\n\n <meta charset="utf-8">\n <meta name="viewport" content="width=device-width, initia"
Domain: www.test4.giftcommerce.com
Type: unauthorized
Detail: Invalid response from http://test4.giftcommerce.com/ [178.79.159.104]: "\n\n<html lang="en">\n\n\n\n\n <meta charset="utf-8">\n <meta name="viewport" content="width=device-width, initia"
Hint: The Certificate Authority failed to verify the temporary nginx configuration changes made by Certbot. Ensure the listed domains point to this nginx server and that it is accessible from the internet.
Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

Server:
Linux giftcommerce.com 4.9.0-17-amd64 #1 SMP Debian 4.9.290-1 (2021-12-12) x86_64 GNU/Linux
Certbot version: certbot 1.24.0
Recently certbot version was updated as per this doc: Certbot Instructions | Certbot

Nginx version: nginx/1.18.0
PHP version: 7.4.27

SSL installation logic:

MySQL DB stores list of sites with their domains and subdomains.
PHP script is scheduled in cron, which checks all domain/subdomain to install. Certbot command is run from within PHP code for those sites and DB is updated based on success/error status.

Eg. site_1 domain (owned by username 'user1') is set as "testdomain-abc.com" in DB. Cron script would pick it and run the below command (for both subdomain and main domain):
certbot certonly --nginx -d user1.giftcommerce.com -d www.user1.giftcommerce.com -d testdomain-abc.com -d www.testdomain-abc.com --redirect --expand --force-renewal --dry-run

Site structure:

site_1
- nginx.conf (site1.giftcommerce.com, testdomain-abc.com)
site_2
- nginx.conf (site2.giftcommerce.com, testdomain-pqr.com)
.....
site_150
- nginx.conf (site150.giftcommerce.com, testdomain-xyz.com)

Thanks a lot.

2

do not use --force-renewal.

It looks like your script is trying to do what certbot already does by itself, check what certificates are expiring and try to renew them.

I can only suggest something is broken there, and if you do not want to restart nginx you should use the webroot plugin instead (you should make all websites share a webroot for the /.well-known/acme-challenge directory), kinda like:

server {
        listen 80;
        listen [::]:80;
        server_name _;

        location /.well-known/acme-challenge/ {
                root /var/www/acme;
        }

        location / {
                return 301 https://$host$request_uri;
        }
}
1 Like
3

A quick try is to use this on your command. The nginx plug-in may need to wait longer with larger nginx configs before continuing to the challenge

--nginx-sleep-seconds NGINX_SLEEP_SECONDS
                        Number of seconds to wait for nginx configuration
                        changes to apply when reloading. (default: 1)
5 Likes
4

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.