Certbot, nginx, OL8, 'failed to authenticate' - invalid response from acme-challenge

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: apex-test.dnv.com

I ran this command: sudo certbot --nginx

It produced this output:

Certbot failed to authenticate some domains (authenticator: nginx). The Certificate Authority reported these problems:
  Domain: apex-test.dnv.com
  Type:   unauthorized
  Detail: Invalid response from http://apex-test.dnv.com/.well-known/acme-challenge/Dba3tk5zWH670COWWirTQo8dY9gcdi9KwWdbgkaE3Xk [51.124.73.17]: "<html><head><title>Request Rejected</title></head><body>The requested URL was rejected. Please consult with your administrator.<"

My web server is (include version): nginx 1.14

The operating system my web server runs on is (include version): Oracle Linux 8

My hosting provider, if applicable, is: NA

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): NO

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):certbot 1.23.0

Certbox was installed following the instructions for Fedora (i.e it's a snap installation).

nginx.conf:

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

# Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    # Load modular configuration files from the /etc/nginx/conf.d directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/nginx/conf.d/*.conf;

    server {
        listen       80 default_server;
        listen       [::]:80 default_server;
        server_name  _ apex-test.dnv.com;
        root         /usr/share/nginx/html/;

        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;

        }
}

My site returns the default index.html file from HTTP.

The log has things like:

2022-02-09 16:10:22,067:DEBUG:certbot.crypto_util:Creating CSR: /etc/letsencrypt/csr/0000_csr-certbot.pem
2022-02-09 16:10:22,072:DEBUG:acme.client:JWS payload:
b'{\n  "identifiers": [\n    {\n      "type": "dns",\n      "value": "apex-test.dnv.com"\n    }\n  ]\n}'
2022-02-09 16:10:22,074:DEBUG:acme.client:Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/new-
order:
{
2022-02-09 16:10:22,397:DEBUG:certbot_nginx._internal.parser:Writing nginx conf tree to /etc/nginx/nginx.conf:
# For more information on configuration, see:
#   * Official English Documentation: http://nginx.org/en/docs/
#   * Official Russian Documentation: http://nginx.org/ru/docs/

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

# Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

http {
include /etc/letsencrypt/le_http_01_cert_challenge.conf;
server_names_hash_bucket_size 128;
    server {rewrite ^(/.well-known/acme-challenge/.*) $1 break; # managed by Certbot


        listen       80 default_server;
        listen       [::]:80 default_server;
        server_name  azurewe1548.azureexternal.dnvgl.com apex-test.dnv.com;
location = /.well-known/acme-challenge/Dba3tk5zWH670COWWirTQo8dY9gcdi9KwWdbgkaE3Xk{default_type text/plain;return 200 Dba3tk5zWH670COWWirTQo8dY9gcdi9KwWdbgkaE3Xk.mgkz5RfepOAFO-EBIRAn4sOvQVOfk28q9lsLA_cmpc0;} # managed by Certbot

}

and

2022-02-09 16:10:24,748:INFO:certbot._internal.auth_handler:Challenge failed for domain apex-test.dnv.com
2022-02-09 16:10:24,748:INFO:certbot._internal.auth_handler:http-01 challenge for apex-test.dnv.com
2022-02-09 16:10:24,748:DEBUG:certbot._internal.display.obj:Notifying user:
Certbot failed to authenticate some domains (authenticator: nginx). The Certificate Authority reported these problems:
  Domain: apex-test.dnv.com
  Type:   unauthorized
  Detail: Invalid response from http://apex-test.dnv.com/.well-known/acme-challenge/Dba3tk5zWH670COWWirTQo8dY9gcdi9KwWdbgkaE3Xk [51.124.73.17]: "<html><head><title>Request Rejected</title></head><body>The requested URL was rejected. Please consult with your administrator.<"

Hint: The Certificate Authority failed to verify the temporary nginx configuration changes made by Certbot. Ensure the listed domains point to this nginx server and that it is accessible from the internet.

2022-02-09 16:10:24,750:DEBUG:certbot._internal.error_handler:Encountered exception:
Traceback (most recent call last):
  File "/var/lib/snapd/snap/certbot/1788/lib/python3.8/site-packages/certbot/_internal/auth_handler.py", line 106, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, best_effort)

LetsDebug doesn't see an issue.

My access and error logs don't seem to show any incoming traffic as a result of the certbot action.

Hi @AndyH and welcome to the LE community forum :slight_smile:

Including vhost config files within a server block doesn't make any sense.

2 Likes

It seems to be the default configuration in RHEL.

There are no files in the directory and I don't believe that they are meant to support virtual hosts.

Very strange and a potential for misconfiguration.
If the folder is empty and you don't plan on putting any files there, then I would comment that line out.

1 Like

I can see nginx being restarted/reloaded twice within a couple of seconds, which makes me think that the first reload is when it places the challenge and the second when it returns the original configuration.

I was expecting a '.well-known' directory to be populated, but that's not happening. I think that the 'location' configuration is meant to return the challenge response without resorting to file access?

You might want to try the --webroot authentication method.
[no web server modification nor restart required]

1 Like

--webroot and --standalone both fail.

It's as if the remote process isn't trying to contact my server!

This looks like an error message from a WAF.

2 Likes

Correct. The --nginx plug-in works exactly like that.

2 Likes

This was in the temp change made by the plug-in. Does the azure name make sense to you? Are you using a load balancer or CDN?

2 Likes

The machine is known as azurewe1548.azureexternal.dnvgl.com within our network (we have a lump of MS Azure maintained as an internal resource).

apex-test.dnv.com is the 'Internet' name assigned to the server and the one that I'm trying to get the certificate for.

The server is sat behind a load balancer.

I'll look into that. I wonder if the load balancer/firewall is blocking 'bots' or letsencrypt addresses? What user agent would certbot show as?

Something like this:
Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)

There will be up to 4 attempts in sequence from various points around the globe (currently AWS regions). Note these are NOT certbot requests. These are requests coming from the Let's Encrypt server which is trying to validate you have control of that domain name. Certbot is the initiating client of this process (there are numerous clients).

Ah, the plot thickens. Load Balancers require some special handling with the HTTP challenge. Each server behind the LB must be able to respond to the challenge. That is often difficult so other approaches are needed.

2 Likes

OK... so we have a load balancer/firewall for the company.

This should accept apex-test.dnv.com and forward/redirect to the appropriate host which in this case is a single machine known internally as azure1548.azureexternal.dnvgl.com.

I'll check with the network guys as to whether this redirection is to a hostname or to its IP address - I believe that could be an issue with the HTTP challenge process?

Understood re: certbot client vs Let's Encrypt server.

If the load-balancer is already instructed to connect with only one internal system, we can pretty much rule that part out. Even if caching is involved, each challenge request is always unique; So that rules that part out as well.

Now the firewall (FW) is a whole other issue.
Maybe the FW logs can shed some light into why the challenge requests are failing.

Clearly NOT a normal nginx response.

2 Likes

Agree if Load Balancer only sends to one server that is not an issue. But, something seems amiss with nginx conf.
From first post:

From log of temp changes by certbot:

The first server block did not contain the azure name. Was that just an editing job on your part @AndyH

Also, there was a root statement in the first post but not in the temp changes. Was that also just lost in editing?

I ask to ensure we are looking at the same conf as the active nginx.

That said, a firewall or other system (not nginx) seems blocking requests. I just wonder if that is the only other factor though.

2 Likes

Correct.
That's why I pointed towards --webroot.

Although I missed:

Are there any logs on those failures?
Can you show the command used with --webroot ?

2 Likes
 sudo certbot certonly  --webroot -w /usr/share/nginx/html/ -v --test-cert -d apex-test.dnv.com
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Requesting a certificate for apex-test.dnv.com
Performing the following challenges:
http-01 challenge for apex-test.dnv.com
Using the webroot path /usr/share/nginx/html for all unmatched domains.
Waiting for verification...
Challenge failed for domain apex-test.dnv.com
http-01 challenge for apex-test.dnv.com

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
  Domain: apex-test.dnv.com
  Type:   unauthorized
  Detail: Invalid response from http://apex-test.dnv.com/.well-known/acme-challenge/Hk0Xm9BRE-83pZkE43qZYXTbAqUCSh1Oqb5TJP7RqXc [51.124.73.17]: "<html><head><title>Request Rejected</title></head><body>The requested URL was rejected. Please consult with your administrator.<"

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.
==> /var/log/nginx/error.log <==
2022/02/10 07:04:26 [error] 303783#0: *21869 open() "/usr/share/nginx/html/.well-known/acme-challenge/Hk0Xm9BRE-83pZkE43qZYXTbAqUCSh1Oqb5TJP7RqXc" failed (2: No such file or directory), client: 10.234.2.54, server: _, request: "GET /.well-known/acme-challenge/Hk0Xm9BRE-83pZkE43qZYXTbAqUCSh1Oqb5TJP7RqXc HTTP/1.1", host: "apex-test.dnv.com"

==> /var/log/nginx/access.log <==
10.234.2.54 - - [10/Feb/2022:07:04:26 +0000] "GET /.well-known/acme-challenge/Hk0Xm9BRE-83pZkE43qZYXTbAqUCSh1Oqb5TJP7RqXc HTTP/1.1" 404 169 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15" "-"

letsencrypt.log

2022-02-10 07:03:57,937:DEBUG:certbot._internal.plugins.selection:Single candidate plugin: * webroot
Description: Place files in webroot directory
Interfaces: Authenticator, Plugin
Entry point: webroot = certbot._internal.plugins.webroot:Authenticator
Initialized: <certbot._internal.plugins.webroot.Authenticator object at 0x7fd5d2b82730>
Prep: True
2022-02-10 07:03:57,937:DEBUG:certbot._internal.plugins.selection:Selected authenticator <certbot._internal.plugins.webroot.Authenticator object at 0x7fd5d2b82730> and installer None
2022-02-10 07:03:57,937:INFO:certbot._internal.plugins.selection:Plugins selected: Authenticator webroot, Installer None2022-02-10 07:03:57,956:DEBUG:certbot._internal.main:Picked account: <Account(RegistrationResource(body=Registration(key=None, contact=(), agreement=None, status=None, terms_of_service_agreed=None, only_return_existing=None, external_account_binding=None), uri='https://acme-staging-v02.api.letsencrypt.org/acme/acct/43538758', new_authzr_uri=None, terms_of_service=None), fcd5dd31f9ddf18378300b419889e931, Meta(creation_dt=datetime.datetime(2022, 2, 9, 16, 10, 15, tzinfo=), creation_host='azure1548.azureexternal.dnvgl.com', register_to_eff=None))>
2022-02-10 07:03:57,956:DEBUG:acme.client:Sending GET request to https://acme-staging-v02.api.letsencrypt.org/directory2022-02-10 07:03:57,956:DEBUG:acme.client:Sending GET request to https://acme-staging-v02.api.letsencrypt.org/directory.2022-02-10 07:03:57,958:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): acme-staging-v02.api.letsencrypt.org:443
2022-02-10 07:03:58,435:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "GET /directory HTTP/1.1" 200 822
2022-02-10 07:03:58,436:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Thu, 10 Feb 2022 07:03:58 GMT
Content-Type: application/json
Content-Length: 822
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
"keyChange": "https://acme-staging-v02.api.letsencrypt.org/acme/key-change",

2022-02-10 07:03:59,237:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Thu, 10 Feb 2022 07:03:59 GMT
Content-Type: application/json
Content-Length: 819
Connection: keep-alive
Boulder-Requester: 43538758
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: 0002ONu2v3ZD1BQ36BUI58bfK3IgB-Tice9BBz1Y2rWaTlY
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "apex-test.dnv.com"
  },
  "status": "pending",
  "expires": "2022-02-17T07:03:59Z",
  "challenges": [
    {
      "type": "http-01",
2022-02-10 07:04:00,533:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Thu, 10 Feb 2022 07:04:00 GMT
Content-Type: application/json
Content-Length: 1234
Connection: keep-alive
Boulder-Requester: 43538758
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: 0001CnyhQdZkKGc3NuV5NizAdwofG1fOC-46Erp2SUsZ7O0
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "apex-test.dnv.com"
  },
  "status": "invalid",
  "expires": "2022-02-17T07:03:59Z",
  "challenges": [
2022-02-10 07:04:00,534:DEBUG:certbot._internal.display.obj:Notifying user:
Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:  Domain: apex-test.dnv.com
  Type:   unauthorized
  Detail: Invalid response from http://apex-test.dnv.com/.well-known/acme-challenge/Hk0Xm9BRE-83pZkE43qZYXTbAqUCSh1Oqb5TJP7RqXc [51.124.73.17]: "<html><head><title>Request Rejected</title></head><body>The requested URL was rejected. Please consult with your administrator.<"

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

etc.

Looking at the logs - the webserver logs seems to show activity 30 seconds after certbot completes? I hadn't noticed that before.

I can see the calls made to the .well-known URL as shown in my initial enquiry. As these were a couple of hours after I'd gone to bed, I assume that they are from community testers rather than Lets Encrypt.

I re-ran the "certbot --nginx -v" and then grepped the webserver logs - nothing appeared.

To clarify any confusion over the nginx configuration - I removed the azure1548 reference after generating the initial report here. I've also changed a location setting to try and remove any caching issues:

    include /etc/nginx/conf.d/*.conf;

    server {
        listen       80 default_server;
        listen       [::]:80 default_server;
        server_name  _ apex-test.dnv.com;
        root         /usr/share/nginx/html/;

        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;

        location \ {
                allow all;
                expires -1;
        }
    }
}

Please place a test text file in the expected challenge location:
Like:
/usr/share/nginx/html/.well-known/acme-challenge/Test_File-1234

And let's see if it can be reached via the Internet.
http://apex-test.dnv.com/.well-known/acme-challenge/Test_File-1234

1 Like