Getting urn:ietf:params:acme:error:unauthorized error while creating new certificate

Help
21

We may be getting closer to finding the cause.

What command do you use to start nginx? Is your nginx installed in the default location for Debian 12 and started with the default config?

Because your log shows that Certbot made the correct changes to the server block for this domain. Yet, your access log shows your server redirecting the first HTTP request that Let's Encrypt server sent you. You sent it to HTTPS://... LE Server followed that redirect and you sent it to HTTPS home page of that server. That, of course, is not what LE Server needs to see so it fails the request. Notice the URL in the error message is your home page URL.

So, the changes Certbot made are not being applied to your running nginx system. We have tried a very long sleep-seconds so I doubt that a longer wait would help. When the 1s default is not enough usually 2s or 3s is more than enough.

We should see if you have multiple nginx systems running. This can happen in rare cases where one is running under systemd control and the other "natively".

What do these show

sudo ps -eF | grep nginx
sudo systemctl status -l --no-pager nginx
4 Likes
22

I use systemctl restart nginx to start nginx and yes it is installed at the default location for Debian 12 which is /etc/nginx

sudo ps -eF | grep nginx

root     1990573       1  0 21665 49384   3 03:12 ?        00:00:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
www-data 1990574 1990573  0 21981 59648   0 03:12 ?        00:00:51 nginx: worker process
www-data 1990575 1990573  0 21981 59924   3 03:12 ?        00:00:51 nginx: worker process
www-data 1990576 1990573  0 21981 59588   3 03:12 ?        00:00:51 nginx: worker process
www-data 1990577 1990573  0 21981 60064   3 03:12 ?        00:00:50 nginx: worker process
root     2052306 2052279  0  1584  2156   1 05:42 pts/0    00:00:00 grep nginx

systemctl status -l --no-pager nginx

● nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; preset: enabled)
     Active: active (running) since Thu 2025-05-15 03:12:10 UTC; 2h 31min ago
       Docs: man:nginx(8)
    Process: 1990571 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
    Process: 1990572 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
   Main PID: 1990573 (nginx)
      Tasks: 5 (limit: 9484)
     Memory: 210.2M
        CPU: 3min 27.681s
     CGroup: /system.slice/nginx.service
             ├─1990573 "nginx: master process /usr/sbin/nginx -g daemon on; master_process on;"
             ├─1990574 "nginx: worker process"
             ├─1990575 "nginx: worker process"
             ├─1990576 "nginx: worker process"
             └─1990577 "nginx: worker process"

May 15 03:12:09 localhost systemd[1]: Starting nginx.service - A high performance web server and a reverse proxy server...
May 15 03:12:10 localhost systemd[1]: Started nginx.service - A high performance web server and a reverse proxy server.

I think i sort of might have hit upon the root cause.

Today i re-ran certbot certificates which you had shared yesterday, and after successfull run, when i scrolled up and saw through the complete logs, i see plenty of errors saying...

Renewal configuration file /etc/letsencrypt/renewal/some-subdomain.floristtouch.com.conf produced an unexpected error: expected /etc/letsencrypt/live/some-subdomain.floristtouch.com/cert.pem to be a symlink. Skipping.

Possibly this might have been caused while migrating servers. Although i would have expected scp to migrate symlinks correctly, but looking at that error, maybe i was mistaken and should have instead taken a tar.gz or rsync for migrating the /etc/letsencrypt folder from the old server to the new one.

As today morning, i again tried to run the same command that failed yesterday after removing all other virtualhosts/server blocks from the server...

certbot renew --dry-run --cert-name www.theflowershop.ie

and it renewed successfully! Infact since dry-run succeeded, i actually went ahead and force renewed it and even that went through successfully.

Now i think its just time for me to clean up some junk and create some automation to force auto renew domains which are failing and cleanup the /etc/letsencrypt/live and /etc/letsencrypt/renewal directories.

Will keep you posted on how things proceed again when i get the maintenance window tomorrow early morning UTC hours.

Appreciate all your help and continous support in coordinating with me to get to the root of this issue.

23

The lost symlinks most likely were lost when migrating. Good that you know how to fix them now. I believe scp can preserve symlinks but requires specific settings.

Glad that renewed but I would have expected it to if that was the only server block in nginx. We had an earlier test with just one server block and it worked. It was only when adding all your others it was a problem.

The symlinks need fixing regardless so definitely do that. But, if that was the reason for the problem with your full nginx config the error would be different. Which is why I didn't have you check that earlier.

Still, I will cross my fingers that fixing the symlinks will get all of them working :slight_smile:

One more note ... you shouldn't have to "force" any of them. If --dry-run works then just try without. If it is within 30 days of expiration the cert will renew. No need to force renew certs that have more life than that left.

To check all of them do: sudo certbot renew --dry-run

2 Likes
24

Now is the strangest part... I fixed each and every broken symlink. Now there are no more symlink errors, yet when i go to renew a domain, it gives the same error... :frowning:

root@localhost:/etc/nginx# certbot renew --dry-run --cert-name www.theflowershop.ie
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/www.theflowershop.ie.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Simulating renewal of an existing certificate for www.theflowershop.ie

Certbot failed to authenticate some domains (authenticator: nginx). The Certificate Authority reported these problems:
  Domain: www.theflowershop.ie
  Type:   unauthorized
  Detail: 178.79.159.104: Invalid response from https://theflowershop.ie/: "\n<!DOCTYPE html>\n<html lang=\"en\">\n\n\n\n<head>\n  <meta charset=\"utf-8\">\n  <meta name=\"viewport\" content=\"width=device-width, initia"

Hint: The Certificate Authority failed to verify the temporary nginx configuration changes made by Certbot. Ensure the listed domains point to this nginx server and that it is accessible from the internet.

Failed to renew certificate www.theflowershop.ie with error: Some challenges have failed.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
All simulated renewals failed. The following certificates could not be renewed:
  /etc/letsencrypt/live/www.theflowershop.ie/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
25

Not so strange. It would have been stranger had that worked. The symlinks had to be fixed anyway.

Had the symlinks been the full cause of the problem we would have seen a different error much earlier.

I don't have much time right now and need time to think how best to proceed. We always have the --webroot conversion which would work. But, you wanted to avoid the extra steps needed for that. That --nginx doesn't work for you is unlucky and may be very time-consuming to find the root cause.

1 Like
26

@nishantsworld

I am going to contact one of the Certbot developers offline and see if they have any suggestions.

In the meantime, you were able to try running with just one server block and that worked (both times).

Are you able to easily try running with, say, 100 server blocks? Trying either a full dry-run renew or just try one or two from that group of 100 with --cert-name.

sudo certbot renew --dry-run
or
sudo certbot renew --dry-run --cert-name X

If that works try again with, say, 1000 and retry some renew --dry-run tests.

Maybe this will help identify what is happening.

A quick recap for other volunteers ...

This is a very large nginx config with about 600 certificates and 2500 server blocks. They are using --nginx authenticator and the letsencrypt.log shows the correct updates being made to the correct server block. Yet, the reply to the LE server is as if the changes did not take affect (confirmed by looking at the access log). Adding nginx-sleep-seconds of 300 did not help. Reducing their nginx to just one server block works fine. It is only when all are active does it fail.

The whole system was working before migrating to Debian 12 / nginx 1.22.1

3 Likes
27

For trying, i'll again have to wait till early UTC hours tomorrow, but will update you with the output of this strategy, once i try the same.

My gutt feeling is that there is some error in just one of those 600+ server blocks which is not related to nginx syntax or certbot, but is causing the rest of the SSL generations/renewals to fail. I just need to figure out which is that one mole.

Really appreciate your continuous support on fixing this issue.

28

I agree. Have you checked the nginx error log(s) to see if there are any errors related to "reload"?

Your system must be starting otherwise nothing would work. But, your system behaves as if the reload that Certbot does is not working.

If you get a failure with a small number of server blocks it would be helpful to get an upload of the config.txt from this command:

sudo nginx -T >config.txt

An upper case T is essential. It will be the full active nginx config. Hopefully you can reproduce the problem with a small number of server blocks :slight_smile: Given our experience we may be able to see likely candidates that are causing this. Or, possibly even run that exact nginx config on one of our test servers.

1 Like
29

this problem does indeed seem tricky. i can try to offer some help, but unfortunately the timing here isn't great. i have some personal vacation coming up and then the entire certbot team at EFF is on a work retreat for most of next week

with that said, here are some ideas i have:

  1. what's the diff of /etc/nginx comparing the files on your new and to your old system? to help catch any weirdness that may have been previously introduced by copying, maybe try copying the files from the new server to the old one and diffing them there if that's possible? a command to do the diff once you had the files on the same machine would be something like diff -r <path to old /etc/nginx> <path to new /etc/nginx>. when you do that, does anything stand out? including the output of the diff in a post here would probably also be useful
  2. once you find a command like certbot renew --dry-run --cert-name <your cert name> that fails, try playing with a command like certbot certonly --dry-run --cert-name <your cert name> --nginx --verbose --debug-challenges. this should cause certbot to set up the challenge(s) in nginx and then output something like:
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Challenges loaded. Press continue to submit to CA.

The following URLs should be accessible from the internet and return the value
mentioned:

URL:
http://example.com/.well-known/acme-challenge/XI9IIkY8lVSrTXdZ8ZeynlKVHtcxEHFi9fFxPkpaU7g
Expected value:
XI9IIkY8lVSrTXdZ8ZeynlKVHtcxEHFi9fFxPkpaU7g.TH1GWUYMSm4Y00wkiubydR1yG_OlYx7GIyDDOdnaf4k
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Press Enter to Continue

having certbot pause like this after setting up the challenges would allow you to query your server yourself and inspect your nginx config to hopefully help you figure out why nginx isn't serving the correct challenge response

to help manage expectations, i and likely the rest of the certbot team won't be able to help with this again for at least a week. i'm sorry that events happened to line up this way, but i hope this helps at least a little bit for now in tracking down the source of the problem here :crossed_fingers:

5 Likes
30

IMHO you should not be doing this with --nginx. The plugin is commonly known to not perform well at that scale, and it's just a pain to admin.

Speaking as someone who has designed and deployed ACME for several SaaS/PaaS systems, personally, I would do the following in your situation :

  • Use the --standalone plugin with a ProxyPass onto a higher port (e.g. --http-01-port=8080 ). Many people here perfer webroot, but I think ProxyPass is simpler.

  • each domain gets it's own file under /etc/nginx/sites-available

  • Use a shared nginx include/macro to handle this. Your blocks simply have something like:

    include /etc/nginx/macros/acme-public.conf

  • And that file contains a generic proxypass....

File: acme-public.conf

location  /.well-known/acme-challenge  {
    proxy_set_header  X-Real-IP  $remote_addr;
    proxy_set_header  X-Forwarded-For  $proxy_add_x_forwarded_for;
    proxy_set_header  X-Forwarded-Proto  $scheme;
    proxy_set_header  Host  $host;
    proxy_pass  http://127.0.0.1:8080;
}

Each block would also be upgraded to:

  1. Reference another macro that sets the ssl params:

    include /etc/nginx/macros/ssl_config.conf

Which has something like:

## SSL - Core
# generated 2025-03-25, Mozilla Guideline v5.7, nginx 1.27.3, OpenSSL 3.4.0, intermediate config, no OCSP
# https://ssl-config.mozilla.org/#server=nginx&version=1.27.3&config=intermediate&openssl=3.4.0&ocsp=false&guideline=5.7

ssl_session_cache shared:MozSSL:10m;  # about 40000 sessions
ssl_session_timeout  1d;

# intermediate configuration
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ecdh_curve X25519:prime256v1:secp384r1;
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305;
ssl_prefer_server_ciphers off;

# add_header  Access-Control-Allow-Origin  http://127.0.0.1;
add_header  Access-Control-Allow-Origin  *;

# HSTS (ngx_http_headers_module is required) (63072000 seconds)
add_header Strict-Transport-Security "max-age=63072000" always;

ssl_dhparam "/etc/openresty/macros/dhparam";

and

  1. Manually declare the SSL certs in your file:
ssl_certificate = /etc/letsencrypt/active/cert/cert.pem;
ssl_certificate_key = /etc/letsencrypt/active/cert/cert.pkey;

So in the end, each domain has a file that looks like:

server {
    listen  80;
    server_name  {NAME};
    include  /etc/nginx/macros/acme-public.conf;
    include  /etc/nginx/macros/logging-http.conf;
    root  /var/www/sites/NULL;  # empty directory
    location / {
        return 301 https://$host$request_uri;
    }
}

server {
    listen  443 ssl;
    server_name  {NAME};
    include  /etc/nginx/macros/ssl_config.conf;
    include  /etc/nginx/macros/logging-https.conf;
    root  {ROOT};  # actual root
    ssl_certificate  /etc/letsencrypt/active/cert/fullchain.pem;
    ssl_certificate_key  /etc/letsencrypt/active/cert/pkey.pem;
}

A few benefits of this approach:

  • certbot works way faster, there are no compatibility issues with nginx parsing
  • everything delegated to macros (include files), so you can make changes quickly
  • you can easily use a script to enroll domains - I write simple tools to generate and update the sites-available files as needed.

If you go for a route like this, there are 3 steps to onboarding a domain:

  • create a 80 block; reload;
  • get a cert;
  • create a 443 block; reload

You can also have the 443 block reference a generic cert, then regenerate it. Unfortunately, you can't easily create a "stub" file where Certbot will eventually place the cert, because Certbot will think it's a conflict. I typically handle the 443 update in a post-hook.

You can certainly keep with your strategy. I just think you've outgrown the utility of the nginx plugin and would be better off with a more scalable approach.

7 Likes
31

Actually at the beginning of this incident, there were some warnings see on running nginx -T, however those i had fixed them prior to reaching out to you here itself.

32

Thanks for offering to help even through your vacation...

The current situation is such...

  1. There were missing symlinks, which i manually fixed
  2. Post that, i began moving individual server block into a new vhost.d folder and began renewing certs there. Doing this individually per block has now worked for around 200 odd server blocks
  3. I will need to continue to do this excercise till i find the rogue server block which is causing the rest of the server blocks to fail renewing (which coincidentally still didn't strike me amongst the 200 out of 600 that i renewed)

Once i find that rogue one, i will run the command which you shared and report back the findings...

certbot certonly --dry-run --cert-name --nginx --verbose --debug-challenges

This could take a while as i get only 1 hour every morning to do this activity during maintenance hours since it is a production server, but will get to the root of it before closing this thread so it helps somebody else facing the same issue.

1 Like
33

Thanks. I will try this suggestion, coz i know one thing for sure is that we are growing as a company and since you say we have already outgrown --nginx plugin, we rather plan to scale things the right way.

34

If you are trying to find out if it's just one rogue server block, may I suggest doing binary search instead of one by one? That is, move half of the untested server blocks into the new vhost.d folder. If it breaks, put half of them back and try again. If it worked, add half of the remaining ones and try again.

6 Likes
35

Thanks for the suggestion. True, that'll make things move faster.

3 Likes
36

I found one rogue server block and ran the command you shared certbot certonly --dry-run --cert-name --nginx --verbose --debug-challenges. Here is the output...

Plugins selected: Authenticator nginx, Installer nginx

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
An RSA certificate named hummingbird-flowers.floristtouch.com already exists. Do
you want to update its key type to ECDSA?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(U)pdate key type/(K)eep existing key type: k
Certificate not due for renewal, but simulating renewal for dry run
Simulating renewal of an existing certificate for hummingbird-flowers.floristtouch.com and www.hummingbird-flowers.floristtouch.com
Performing the following challenges:
http-01 challenge for hummingbird-flowers.floristtouch.com
http-01 challenge for www.hummingbird-flowers.floristtouch.com

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Challenges loaded. Press continue to submit to CA.

The following URLs should be accessible from the internet and return the value
mentioned:

URL:
http://hummingbird-flowers.floristtouch.com/.well-known/acme-challenge/FPxYuXD8wiZ952HlWOMIninO6oUfW7Z34XPk48famhY
Expected value:
FPxYuXD8wiZ952HlWOMIninO6oUfW7Z34XPk48famhY.kLJvLrJue3-9wIFIzTlLurUOBgAJevtoK8JWm3B81_E

URL:
http://www.hummingbird-flowers.floristtouch.com/.well-known/acme-challenge/Oe2qdrXsXGqweCtRWs7_EcvPpZ7qyE1Z2ToXeHTGw9I
Expected value:
Oe2qdrXsXGqweCtRWs7_EcvPpZ7qyE1Z2ToXeHTGw9I.kLJvLrJue3-9wIFIzTlLurUOBgAJevtoK8JWm3B81_E
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Press Enter to Continue
Waiting for verification...
Challenge failed for domain hummingbird-flowers.floristtouch.com
Challenge failed for domain www.hummingbird-flowers.floristtouch.com
http-01 challenge for hummingbird-flowers.floristtouch.com
http-01 challenge for www.hummingbird-flowers.floristtouch.com

Certbot failed to authenticate some domains (authenticator: nginx). The Certificate Authority reported these problems:
  Domain: hummingbird-flowers.floristtouch.com
  Type:   unauthorized
  Detail: 178.79.159.104: Invalid response from https://hummingbird-flowers.floristtouch.com/: "\n<!DOCTYPE html>\n<html lang=\"en\">\n\n\n\n<head>\n  <meta charset=\"utf-8\">\n  <meta name=\"viewport\" content=\"width=device-width, initia"

  Domain: www.hummingbird-flowers.floristtouch.com
  Type:   unauthorized
  Detail: 178.79.159.104: Invalid response from https://hummingbird-flowers.floristtouch.com/: "\n<!DOCTYPE html>\n<html lang=\"en\">\n\n\n\n<head>\n  <meta charset=\"utf-8\">\n  <meta name=\"viewport\" content=\"width=device-width, initia"

Hint: The Certificate Authority failed to verify the temporary nginx configuration changes made by Certbot. Ensure the listed domains point to this nginx server and that it is accessible from the internet.

Cleaning up challenges
Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

Note
After certbot generated the challenges and waited on me to hit "Enter", i manually tried to open http://hummingbird-flowers.floristtouch.com/.well-known/acme-challenge/FPxYuXD8wiZ952HlWOMIninO6oUfW7Z34XPk48famhY in the browser, however it simply redirected me to http://hummingbird-flowers.floristtouch.com/

I've attached the rogue nginx file, although it looks pretty standard when compared to the others that are working, with the only difference being that in the server block, the domains hummingbirdflowers.co.uk and www.hummingbirdflowers.co.uk are no longer existing, but anyways certbot seems to be renewing only the subdomains and not the main domains in the server_block
nginx_test.txt (1.9 KB)

So i am unable to see why would this server block corrupt the others.

Note, this was working on the old server running nginx v1.18 and certbot v4.0.0. (the only difference is that on the new server nginx is v1.22.1)

1 Like
37

That is in line with what we saw earlier. It is as if Certbot's nginx reload does not work for some odd reason. I doubt it's actually that and is instead some other odd quirk.

We'll have to wait for Certbot team to look at it.

My suggestion is that since you now just have one failing server block we could switch that to use the --webroot method. That won't require Certbot to parse, change, and reload your nginx.

It's up to you how to proceed. But, the below would get his server block working at the risk of making reproducing the problem harder.

sudo certbot certonly --cert-name X --webroot -w /var/www/html/public -d D1 -d D2

Put your actual names in place of X, D1 and D2.

This server block is using a default root folder. Is that common? The other ones I see were using a unique folder. I don't see how that would matter for this problem. Just curiuos

2 Likes
38

I think i am not really in a hurry to move to the webroot way of working, and giving up on the nginx module completely yet.

I have altered the default root directory for that particular domain for testing purposes.

39

Fair.

Did you know that HTTPS requests to that domain are using some other certificate? Possibly by nginx not using that server block to process the request:

See: SSL Checker

3 Likes
40

Have you checked below? maybe new server needs different

If a large number of server names are defined, or unusually long server names are defined, tuning the server_names_hash_max_size and server_names_hash_bucket_size directives at the http level may become necessary

2 Likes