Cerbot with NGINX failes with Status 403 and unauthorized

My domain is:
http://log.cx-networks.com

I ran this command:
sudo certbot --nginx

It produced this output:

Requesting a certificate for log.cx-networks.com

Certbot failed to authenticate some domains (authenticator: nginx). The Certificate Authority reported these problems:
  Domain: log.cx-networks.com
  Type:   unauthorized
  Detail: Invalid response from http://log.cx-networks.com/.well-known/acme-challenge/nUROT5jdwv881d6ddgE7L99j5HWQewTFExlS46d2IhU [88.198.212.191]: "<!DOCTYPE html>\n<html lang=\"en\">\n    <head>\n        <meta charset=\"UTF-8\">\n        <meta http-equiv=\"X-UA-Compatible\" content=\"I"

Hint: The Certificate Authority failed to verify the temporary nginx configuration changes made by Certbot. Ensure the listed domains point to this nginx server and that it is accessible from the internet.

Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

My web server is (include version):
nginx version: nginx/1.18.0 (Ubuntu)

The operating system my web server runs on is (include version):
Ubuntu 20.04.3 LTS

My hosting provider, if applicable, is:
Hetzner

I can login to a root shell on my machine (yes or no, or I don't know):
yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
certbot 1.25.0

Hello,
I have seen every now and then in our monitoring tool an error from our server that the service snap.certbot.renew throws an error. The certificate would have expired now, so I took care of it today. In the process, I noticed that the certbot can't renew the cert via an ACME challenge. I have added the error message above. In the log I could see that Certbot writes the necessary location blocks to the configuration.

2022-04-04 13:08:13,598:DEBUG:certbot_nginx._internal.parser:Writing nginx conf tree to /etc/nginx/sites-enabled/log.cx-networks.com:
server {rewrite ^(/.well-known/acme-challenge/.*) $1 break; # managed by Certbot


    server_name log.cx-networks.com;

    location / {
        proxy_set_header Host $http_host;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Server $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Graylog-Server-URL http://$server_name/;
        proxy_pass       http://192.168.200.5:9000;
    }
location = /.well-known/acme-challenge/qpYLAwnq27nu-bxzmjy4PSicef9nt6R2uHeTMFvoNB4{default_type text/plain;return 200 qpYLAwnq27nu-bxzmjy4PSicef9nt6R2uHeTMFvoNB4.jFCw7yBESEJIuV5lpW
yih8K5KZ4rvxLk0HRzaLHxLS0;} # managed by Certbot
}

In the nginx logs I can also see that the request from Let's Encrypt arrives at the server and is returned with a status of 200.

3.67.34.92 - - [04/Apr/2022:13:08:14 +0200] "GET /.well-known/acme-challenge/qpYLAwnq27nu-bxzmjy4PSicef9nt6R2uHeTMFvoNB4 HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

Still, unfortunately, a 403 - Forbidden response comes from Let's Encrypt and the certificates cannot be issued.

2022-04-04 13:08:15,943:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Mon, 04 Apr 2022 11:08:15 GMT
Content-Type: application/json
Content-Length: 1257
Connection: keep-alive
Boulder-Requester: 49544698
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: 0002hIxYzV4vNH0zjA3D7HfEtMSuDogIiyKpj8BwCXueaaI
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "log.cx-networks.com"
  },
  "status": "invalid",
  "expires": "2022-04-11T11:08:13Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "invalid",
      "error": {
        "type": "urn:ietf:params:acme:error:unauthorized",
        "detail": "Invalid response from http://log.cx-networks.com/.well-known/acme-challenge/qpYLAwnq27nu-bxzmjy4PSicef9nt6R2uHeTMFvoNB4 [88.198.212.191]: \"\u003c!DOCTYPE html\u
003e\\n\u003chtml lang=\\\"en\\\"\u003e\\n    \u003chead\u003e\\n        \u003cmeta charset=\\\"UTF-8\\\"\u003e\\n        \u003cmeta http-equiv=\\\"X-UA-Compatible\\\" content=\\\"
I\"",
        "status": 403
      },
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/2089003918/7C-49A",
      "token": "qpYLAwnq27nu-bxzmjy4PSicef9nt6R2uHeTMFvoNB4",
      "validationRecord": [
        {
          "url": "http://log.cx-networks.com/.well-known/acme-challenge/qpYLAwnq27nu-bxzmjy4PSicef9nt6R2uHeTMFvoNB4",
          "hostname": "log.cx-networks.com",
          "port": "80",
          "addressesResolved": [
            "88.198.212.191"
          ],
          "addressUsed": "88.198.212.191"
        }
      ],
      "validated": "2022-04-04T11:08:14Z"
    }
  ]
}

I have tried the following until now:

  • complete fresh reinstallation nginx server
  • using the default configuration for the nginx server
    But still it always comes back an Unauthorized response.

I have already searched many forums, but unfortunately did not find an answer. I would be happy if someone here can help me.

Thanks a lot and greetings,
Leo

1 Like

Welcome to the community @leo.kuenne and thanks for the great initial post

You should see several (currently 4) successful challenge requests in your nginx logs. You only show 1. These requests will be identical except they originate from different parts of the world. Some firewalls block based on geographic requests. Or, may block repeated attempts thinking they are ddos attacks. Could this be happening with your system?

4 Likes

Hi @MikeMcQ,

thanks a lot for your quick reply! I can only spot two request that I would relate to the same acme-challenge try.

18.196.102.134 - - [04/Apr/2022:12:55:49 +0200] "GET /.well-known/acme-challenge/xizKiDVHSiX6Mv01b2wdv3jvOQVQr5jdbVK5QwCjue8 HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

18.196.102.134 - - [04/Apr/2022:13:07:47 +0200] "GET /.well-known/acme-challenge/7NxArNaz8An6ZmMRj-tkUfVgajEnxN0F4szxhbHfHkg HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
3.67.34.92 - - [04/Apr/2022:13:08:14 +0200] "GET /.well-known/acme-challenge/qpYLAwnq27nu-bxzmjy4PSicef9nt6R2uHeTMFvoNB4 HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

3.67.34.92 - - [04/Apr/2022:13:13:12 +0200] "GET /.well-known/acme-challenge/KvJ-dsu14bIlYe3wRCubCcfYzksTMAI0YYw1LZN8zHI HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

We are blocking non german ips from accessing the server that is true.

I temporarily disabled geo-blocking in the firewall, but only three requests came through and in fact the certificate was then created and successfully deployed. Thanks a lot!!

Is there a way / setting to request only requests from Germany from Let's Encrpyt? I had searched for a list of IPs from Let's Encrpyt before but understandably they don't exist.

Otherwise I have to use a different way of verification

1 Like

Good news on getting the cert. Usually the production system challenges from 3 US locations and 1 German IP (this may change any time). Successful challenges are also cached for 30 days. So, you probably saw the 3 new US challenges.

There is no way to change where LE makes the requests from. Challenging from different origins is key to ensuring you have control of the domain name.

There is the DNS challenge. This will require an API for your DNS provider to add/remove the needed TXT records.

If you have API access to your firewall perhaps open/close the firewall just before/after the renewal. There are hooks in certbot or use your own script for the renew

5 Likes

Too bad. But understandable!

Yes, in the meantime I have looked at the DNS challange. I think it will come down to the hooks though!

Thanks a lot for your help!!!

3 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.