Certbot standalone insists on IPv6, not able to bind to IPv4, fails authorization procedure

Help
1

I want to use certbot standalone on a custom port 8080. I configured Nginx to receive /.well-known/acme-challenge/ requests on port 80 and proxy them to 8080. But when I run certbot, it binds to this port on IPv6 which I don't have, and fails the verification.

Relevant Nginx part:

location /.well-known/acme-challenge/ {
    proxy_pass http://127.0.0.1:8080/.well-known/acme-challenge/;
}

Certbot command:

certbot certonly -v -vv --standalone --http-01-port 8080 --non-interactive --agree-tos --email "webmaster@example.com" -d "example.com" --dry-run

Relevant output:

Performing the following challenges:
http-01 challenge for example.com
Successfully bound to :8080 using IPv6
Certbot wasn't able to bind to :8080 using IPv4, this is often expected due to the dual stack nature of IPv6 socket implementations.
Waiting for verification...

What I tried:

  • Searched, of course, but topics I find are somehow irrelevant or do not help. Lots of people are - simply - misunderstanding the purpose of --http-01-port believing it will force the ACME challenge to use the port they want; it won't. But that clutters up the search results
  • Looked for a way to force Certbot to use IPv4, apparently there's none (Certbot, force IPv4?)
  • Disabled IPv6 on my system completely, did not help
  • Checked the actual place in the source code exactly where the error message comes from, found it saying "If Python is compiled without IPv6…", looked for a way to disable IPv6 for Python, nope

Port 80 (which is where the external connection is expected) on my server is open and has been working all the time for non-standalone verifications. I have also tried the config and the command pasted above on a clean, newly created Google Cloud VPS and it worked fine.

Something must be wrong with my main server.

# certbot --version
certbot 0.31.0

# cat /etc/debian_version
10.13

# nginx -V
nginx version: nginx/1.22.0
built by gcc 8.3.0 (Debian 8.3.0-6) 
built with OpenSSL 1.1.1d  10 Sep 2019 (running with OpenSSL 1.1.1n  15 Mar 2022)
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-g -O2 -fdebug-prefix-map=/data/builder/debuild/nginx-1.22.0/debian/debuild-base/nginx-1.22.0=. -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie'
2

What's the error message after this?

The IPv4/IPv6 message is likely to be a red herring. It usually means that Certbot has bound both address families in a single socket binding.

See whether the subsequent challenge is a timeout, or connection refused, or something else, is the more useful information.

5 Likes
3

The IPv4/IPv6 message is likely to be a red herring. It usually means that Certbot has bound both address families in a single socket binding.

Thank you for the heads up! Then it could have very well been something else. Having seen the wasn't able to bind part, I ignored where it complains about 404. I will have another take on revisiting my Nginx config first thing tomorrow then.

What's the error message after this?

Short:

certbot certonly 
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for example.com
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. example.com (http-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: 1.2.3.4: Fetching http://example.com/.well-known/acme-challenge/challenge-challenge: Error getting validation data

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: example.com
   Type:   connection
   Detail: 1.2.3.4: Fetching
   http://example.com/.well-known/acme-challenge/challenge-challenge:
   Error getting validation data

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.

Long:

certbot -v certonly 
Storing nonce: NONCE
Performing the following challenges:
http-01 challenge for example.com
Successfully bound to :8080 using IPv6
Certbot wasn't able to bind to :8080 using IPv4, this is often expected due to the dual stack nature of IPv6 socket implementations.
Waiting for verification...
JWS payload:
b'{\n  "resource": "challenge",\n  "type": "http-01"\n}'
Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/123456789/Sf45ERgs:
{
  "protected": "hash-protected",
  "signature": "hash-signature",
  "payload": "hash-payload"
}
https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/chall-v3/123456789/Sf45ERgs HTTP/1.1" 200 193
Received response:
HTTP 200
Server: nginx
Date: Wed, 30 Nov 2022 21:00:00 GMT
Content-Type: application/json
Content-Length: 193
Connection: keep-alive
Boulder-Requester: 78073534
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index", <https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/123456789>;rel="up"
Location: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/123456789/Sf45ERgs
Replay-Nonce: another-nonce
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "type": "http-01",
  "status": "pending",
  "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/123456789/Sf45ERgs",
  "token": "token"
}
Storing nonce: another-nonce
JWS payload:
b''
Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/123456789:
{
  "protected": "hash-protected",
  "signature": "hash-signature",
  "payload": ""
}
https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/123456789 HTTP/1.1" 200 1073
Received response:
HTTP 200
Server: nginx
Date: Wed, 30 Nov 2022 21:00:00 GMT
Content-Type: application/json
Content-Length: 1073
Connection: keep-alive
Boulder-Requester: 78073534
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: yet-another-nonce
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "example.com"
  },
  "status": "invalid",
  "expires": "2022-12-07T21:00:00Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "invalid",
      "error": {
        "type": "urn:ietf:params:acme:error:connection",
        "detail": "1.2.3.4: Fetching http://example.com/.well-known/acme-challenge/token: Error getting validation data",
        "status": 400
      },
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/123456789/Sf45ERgs",
      "token": "token",
      "validationRecord": [
        {
          "url": "http://example.com/.well-known/acme-challenge/token",
          "hostname": "example.com",
          "port": "80",
          "addressesResolved": [
            "1.2.3.4"
          ],
          "addressUsed": "1.2.3.4"
        }
      ],
      "validated": "2022-11-30T21:00:00Z"
    }
  ]
}
Storing nonce: yet-another-nonce
Reporting to user: The following errors were reported by the server:

Domain: example.com
Type:   connection
Detail: 1.2.3.4: Fetching http://example.com/.well-known/acme-challenge/token: Error getting validation data

To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address. Additionally, please check that your computer has a publicly routable IP address and that no firewalls are preventing the server from communicating with the client. If you're using the webroot plugin, you should also verify that you are serving files from the webroot path you provided.
Encountered exception:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/certbot/auth_handler.py", line 82, in handle_authorizations
    self._respond(aauthzrs, resp, best_effort)
  File "/usr/lib/python3/dist-packages/certbot/auth_handler.py", line 168, in _respond
    self._poll_challenges(aauthzrs, chall_update, best_effort)
  File "/usr/lib/python3/dist-packages/certbot/auth_handler.py", line 239, in _poll_challenges
    raise errors.FailedChallenges(all_failed_achalls)
certbot.errors.FailedChallenges: Failed authorization procedure. example.com (http-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: 1.2.3.4: Fetching http://example.com/.well-known/acme-challenge/token: Error getting validation data

Calling registered functions
Cleaning up challenges
Stopping server at :::8080...
Exiting abnormally:
Traceback (most recent call last):
  File "/usr/bin/certbot", line 11, in <module>
    load_entry_point('certbot==0.31.0', 'console_scripts', 'certbot')()
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 1365, in main
    return config.func(config, plugins)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 1250, in certonly
    lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 121, in _get_and_save_cert
    lineage = le_client.obtain_and_enroll_certificate(domains, certname)
  File "/usr/lib/python3/dist-packages/certbot/client.py", line 410, in obtain_and_enroll_certificate
    cert, chain, key, _ = self.obtain_certificate(domains)
  File "/usr/lib/python3/dist-packages/certbot/client.py", line 353, in obtain_certificate
    orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
  File "/usr/lib/python3/dist-packages/certbot/client.py", line 389, in _get_order_and_authorizations
    authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
  File "/usr/lib/python3/dist-packages/certbot/auth_handler.py", line 82, in handle_authorizations
    self._respond(aauthzrs, resp, best_effort)
  File "/usr/lib/python3/dist-packages/certbot/auth_handler.py", line 168, in _respond
    self._poll_challenges(aauthzrs, chall_update, best_effort)
  File "/usr/lib/python3/dist-packages/certbot/auth_handler.py", line 239, in _poll_challenges
    raise errors.FailedChallenges(all_failed_achalls)
certbot.errors.FailedChallenges: Failed authorization procedure. example.com (http-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: 1.2.3.4: Fetching http://example.com/.well-known/acme-challenge/token: Error getting validation data
Failed authorization procedure. example.com (http-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: 1.2.3.4: Fetching http://example.com/.well-known/acme-challenge/token: Error getting validation data

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: example.com
   Type:   connection
   Detail: 1.2.3.4: Fetching
   http://example.com/.well-known/acme-challenge/token:
   Error getting validation data

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.
4

This is a fallback error that Let's Encrypt spits out when there is an unspecified networking or HTTP protocol error.

You can exclude the --standalone binding issue by just trying something like:

certbot certonly -d example.com --webroot -w /tmp --dry-run

and I predict that it will result in the same error message.

You can try get a more specific error messaging by trying to curl your website from an unrelated external server, or run your site through something like letsdebug.net.

You can also try run the --standalone command with --debug-challenges, which will pause Certbot before it submits the challenge response to Let's Encrypt. You can then go into the Certbot log, find the /.well-known/acme-challenge/xxx URL, and try curl it yourself from an external server to see what happens.

6 Likes
5

I have a small script that can help debug LetsEncrypt's proxy routes:

it's hardcoded to port 7201, but one can edit that easily. i'll release a customizable version.

it requires a pip install pyramid.

5 Likes
6

The error was indeed elsewhere and --debug-challenges was the key to finding it. You helped tremendously. Thank you!

3 Likes
7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.