Certbot certificate validation failing with timeout when all seems to be correct

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: test.33.foh.house

I ran this command:docker compose -f docker-data/docker-compose.yml run --rm --entrypoint "
certbot -v certonly --webroot -w /var/www/certbot
$staging_arg
$email_arg
$domain_args
--rsa-key-size $rsa_key_size
--agree-tos
--force-renewal" certbot

It produced this output: the failing portion is this: 2024-04-10 20:04:28,761:DEBUG:acme.client:Storing nonce: JLFMYkWoixlQ2nDp1PzJ-n0JKbTQh1pCbchhrARkis0H4iO2TIY
2024-04-10 20:04:31,765:DEBUG:acme.client:JWS payload:
b''
2024-04-10 20:04:31,771:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/authz-v3/336931346037:
{
"protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2FjY3QvMTY2MTc5Njg3NyIsICJub25jZSI6ICJKTEZNWWtXb2l4bFEybkRwMVB6Si1uMEpLYlRRaDFwQ2JjaGhyQVJraXMwSDRpTzJUSVkiLCAidXJsIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2F1dGh6LXYzLzMzNjkzMTM0NjAzNyJ9",
"signature": "K0u8RSjvtUkIOvxvPA10Byiz3J1fe9hm3kMqgLMaOITHUPAguNXNDAgDkcT04kAeNSUc4IFoHYiZJyE1JFcXciR_P7KjZ_h7lkA1xrkhwpOmKhlVifQ4ANge9ws86M6QIges7_KXN2MIwvHvOEflre1IkSnNfTft8UznLLTrHyZZrDKYkiUMX-OVtdPyGf8_4FG5ODU8tDStwv7q2-e2PRqCXzPTqLW8zLAt51G2Py-2RMFXBFSUr5mvsAwURdGWvwGLxEf08No8Z8-A4XcXkXboF6Vz7zfMqMUBI7xhv31GKWhHYHcGftRcFCt0ik0bT2rkslCZOV1jT69WxTeBbz8qfg5-WB-fa5L_thWiiJgqoUQD53h5YebC0s3Nxy-as4mmKakg3YWxZ0N9bncaTrCUyAVu1XWHvYyyM6BcWNdMC2YaxWsARPdLON1ZvvqCSF59UKmtMYIhZNljNqb_WcSFF8x5GtmwvQtbDjFnF-FL8QIXhMZkwyrL5ayQWdON45EdmXYYxyj7QF5vffwHXi4IweGBouMJHMhDy8zCX_JqnWQx-fvsxVgqSLaaV1iTlYNhstI_ODdLIkce5i-50EiUO6FAMRCXMJV6fKIfCEdeDb0SsZEg6SCD5yTTfi5rzMKSdN2H1bwlWL4neAJ3Bf1IdeT06ogXC8Dc42u0pl4",
"payload": ""
}
2024-04-10 20:04:31,840:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/336931346037 HTTP/1.1" 200 1207
2024-04-10 20:04:31,840:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Wed, 10 Apr 2024 20:04:31 GMT
Content-Type: application/json
Content-Length: 1207
Connection: keep-alive
Boulder-Requester: 1661796877
Cache-Control: public, max-age=0, no-cache
Link: https://acme-v02.api.letsencrypt.org/directory;rel="index"
Replay-Nonce: Xs-N5fZgM0_ozZPHH7luboq3vvlTCmNSGBCvS8rnDhhgyMcQZRc
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
"identifier": {
"type": "dns",
"value": "test.33.foh.house"
},
"status": "invalid",
"expires": "2024-04-17T20:04:18Z",
"challenges": [
{
"type": "http-01",
"status": "invalid",
"error": {
"type": "urn:ietf:params:acme:error:connection",
"detail": "During secondary validation: 104.143.78.83: Fetching http://test.33.foh.house/.well-known/acme-challenge/tB5dl7_LERfVwg0yhJGTZgof4pA631_uyRUfSP65owY: Timeout during connect (likely firewall problem)",
"status": 400
},
"url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/336931346037/KGkwew",
"token": "tB5dl7_LERfVwg0yhJGTZgof4pA631_uyRUfSP65owY",
"validationRecord": [
{
"url": "http://test.33.foh.house/.well-known/acme-challenge/tB5dl7_LERfVwg0yhJGTZgof4pA631_uyRUfSP65owY",
"hostname": "test.33.foh.house",
"port": "80",
"addressesResolved": [
"104.143.78.83"
],
"addressUsed": "104.143.78.83",
"resolverAddrs": [
"A:10.1.12.81:31390",
"AAAA:10.1.12.88:20140"
]
}
],
"validated": "2024-04-10T20:04:18Z"
}
]
}
2024-04-10 20:04:31,841:DEBUG:acme.client:Storing nonce: Xs-N5fZgM0_ozZPHH7luboq3vvlTCmNSGBCvS8rnDhhgyMcQZRc
2024-04-10 20:04:31,841:INFO:certbot._internal.auth_handler:Challenge failed for domain test.33.foh.house
2024-04-10 20:04:31,841:INFO:certbot._internal.auth_handler:http-01 challenge for test.33.foh.house
2024-04-10 20:04:31,841:DEBUG:certbot._internal.display.obj:Notifying user:
Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
Domain: test.33.foh.house
Type: connection
Detail: During secondary validation: 104.143.78.83: Fetching http://test.33.foh.house/.well-known/acme-challenge/tB5dl7_LERfVwg0yhJGTZgof4pA631_uyRUfSP65owY: Timeout during connect (likely firewall problem)

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

2024-04-10 20:04:31,842:DEBUG:certbot._internal.error_handler:Encountered exception:
Traceback (most recent call last):
File "/opt/certbot/src/certbot/certbot/_internal/auth_handler.py", line 108, in handle_authorizations
self._poll_authorizations(authzrs, max_retries, max_time_mins, best_effort)
File "/opt/certbot/src/certbot/certbot/_internal/auth_handler.py", line 212, in _poll_authorizations
raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.

2024-04-10 20:04:31,842:DEBUG:certbot._internal.error_handler:Calling registered functions
2024-04-10 20:04:31,842:INFO:certbot._internal.auth_handler:Cleaning up challenges
2024-04-10 20:04:31,842:DEBUG:certbot._internal.plugins.webroot:Removing /var/www/certbot/.well-known/acme-challenge/tB5dl7_LERfVwg0yhJGTZgof4pA631_uyRUfSP65owY
2024-04-10 20:04:31,842:DEBUG:certbot._internal.plugins.webroot:All challenges cleaned up
2024-04-10 20:04:31,842:DEBUG:certbot._internal.log:Exiting abnormally:
Traceback (most recent call last):
File "/usr/local/bin/certbot", line 33, in
sys.exit(load_entry_point('certbot', 'console_scripts', 'certbot')())
File "/opt/certbot/src/certbot/certbot/main.py", line 19, in main
return internal_main.main(cli_args)
File "/opt/certbot/src/certbot/certbot/_internal/main.py", line 1873, in main
return config.func(config, plugins)
File "/opt/certbot/src/certbot/certbot/_internal/main.py", line 1600, in certonly
lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
File "/opt/certbot/src/certbot/certbot/_internal/main.py", line 143, in _get_and_save_cert
lineage = le_client.obtain_and_enroll_certificate(domains, certname)
File "/opt/certbot/src/certbot/certbot/_internal/client.py", line 517, in obtain_and_enroll_certificate
cert, chain, key, _ = self.obtain_certificate(domains)
File "/opt/certbot/src/certbot/certbot/_internal/client.py", line 428, in obtain_certificate
orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
File "/opt/certbot/src/certbot/certbot/_internal/client.py", line 496, in _get_order_and_authorizations
authzr = self.auth_handler.handle_authorizations(orderr, self.config, best_effort)
File "/opt/certbot/src/certbot/certbot/_internal/auth_handler.py", line 108, in handle_authorizations
self._poll_authorizations(authzrs, max_retries, max_time_mins, best_effort)
File "/opt/certbot/src/certbot/certbot/_internal/auth_handler.py", line 212, in _poll_authorizations
raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.
2024-04-10 20:04:31,843:ERROR:certbot._internal.log:Some challenges have failed.

My web server is (include version): nginx 1.15-alpine docker container

The operating system my web server runs on is (include version): Ubuntu 22.04.3 LTS

My hosting provider, if applicable, is: NA

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): Not sure it is from certbot/certbot image from docker hub

I've verified that the challenge file is created and accessible using curl while it is temporarily there. My nginx logs show successful requests to the challenge file and sometimes one unsuccessful request that looks like its trying to access the file with all lowercase and failing because the file has some uppercase. I have a test.html file in the challenge folder that I can access successfully through the browser. I can't find any reason for the failure but it is consistent.

Welcome @abaile3312

Do you have a firewall blocking certain IP addresses or geographic regions?

Because Let's Encrypt servers will validate an HTTP Challenge from various places around the world.

The "Secondary Validation" in the error indicates it was one or more of the remote locations that failed.

3 Likes

My host is a vm clone of one that was working just fine. the ip address was changed to match my domain dns record ip address. As far as I can tell, there is no firewall blocking it. I tried using dig from a bunch of dns servers and all were successful except DNS.WATCH timed out a couple times before succeeding.

It is not the DNS lookup that fails but the HTTP request from the LE Server to your domain.

You said you saw some challenges in your nginx access log. You should be seeing 4 or 5 successful requests to satisfy the HTTP challenge.

Can you show those entries?

4 Likes

I can't access that site so something is there

4 Likes

here is the latest nginx log during the certbot command: 18.117.109.109 - - [10/Apr/2024:20:04:18 +0000] "GET /.well-known/acme-challenge/tB5dl7_LERfVwg0yhJGTZgof4pA631_uyRUfSP65owY HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
34.214.118.238 - - [10/Apr/2024:20:04:18 +0000] "GET /.well-known/acme-challenge/tB5dl7_LERfVwg0yhJGTZgof4pA631_uyRUfSP65owY HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
23.178.112.206 - - [10/Apr/2024:20:04:19 +0000] "GET /.well-known/acme-challenge/tB5dl7_LERfVwg0yhJGTZgof4pA631_uyRUfSP65owY HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"

the previous one with the file not found where it was trying to access it in all lowercase is no longer in the log because the container was restarted.

here are a bunch coming from I don't know where that are failing:
2024/04/10 21:44:54 [error] 9#9: *935 "/etc/nginx/html/index.html" is not found (2: No such file or directory), client: 44.206.245.162, server: test.33.foh.house, request: "HEAD / HTTP/1.1", host: "test.33.foh.house"
44.206.245.162 - - [10/Apr/2024:21:44:54 +0000] "HEAD / HTTP/1.1" 404 0 "-" "curl/7.81.0" "-"
2024/04/10 21:44:55 [error] 9#9: *936 "/etc/nginx/html/index.html" is not found (2: No such file or directory), client: 44.206.245.162, server: test.33.foh.house, request: "HEAD / HTTP/1.1", host: "test.33.foh.house"
44.206.245.162 - - [10/Apr/2024:21:44:55 +0000] "HEAD / HTTP/1.1" 404 0 "-" "curl/7.81.0" "-"
2024/04/10 21:44:56 [error] 9#9: *937 "/etc/nginx/html/index.html" is not found (2: No such file or directory), client: 44.206.245.162, server: test.33.foh.house, request: "HEAD / HTTP/1.1", host: "test.33.foh.house"
44.206.245.162 - - [10/Apr/2024:21:44:56 +0000] "HEAD / HTTP/1.1" 404 0 "-" "curl/7.81.0" "-"
2024/04/10 21:44:56 [error] 9#9: *938 "/etc/nginx/html/index.html" is not found (2: No such file or directory), client: 44.206.245.162, server: test.33.foh.house, request: "HEAD / HTTP/1.1", host: "test.33.foh.house"
44.206.245.162 - - [10/Apr/2024:21:44:56 +0000] "HEAD / HTTP/1.1" 404 0 "-" "curl/7.81.0" "-"
2024/04/10 21:44:57 [error] 9#9: *939 "/etc/nginx/html/index.html" is not found (2: No such file or directory), client: 44.206.245.162, server: test.33.foh.house, request: "HEAD / HTTP/1.1", host: "test.33.foh.house"
44.206.245.162 - - [10/Apr/2024:21:44:57 +0000] "HEAD / HTTP/1.1" 404 0 "-" "curl/7.81.0" "-"
2024/04/10 21:45:04 [error] 9#9: *940 open() "/var/www/certbot/.well-known/acme-challenge/tB5dl7_LERfVwg0yhJGTZgof4pA631_uyRUfSP65owY" failed (2: No such file or directory), client: 17.58.58.20, server: test.33.foh.house, request: "GET /.well-known/acme-challenge/tB5dl7_LERfVwg0yhJGTZgof4pA631_uyRUfSP65owY HTTP/1.1", host: "test.33.foh.house"
17.58.58.20 - - [10/Apr/2024:21:45:04 +0000] "GET /.well-known/acme-challenge/tB5dl7_LERfVwg0yhJGTZgof4pA631_uyRUfSP65owY HTTP/1.1" 404 154 "-" "AppleNewsBot" "-"
2024/04/10 21:45:04 [error] 9#9: *941 open() "/var/www/certbot/.well-known/acme-challenge/tB5dl7_LERfVwg0yhJGTZgof4pA631_uyRUfSP65owY:" failed (2: No such file or directory), client: 17.58.58.21, server: test.33.foh.house, request: "GET /.well-known/acme-challenge/tB5dl7_LERfVwg0yhJGTZgof4pA631_uyRUfSP65owY: HTTP/1.1", host: "test.33.foh.house"
17.58.58.21 - - [10/Apr/2024:21:45:04 +0000] "GET /.well-known/acme-challenge/tB5dl7_LERfVwg0yhJGTZgof4pA631_uyRUfSP65owY: HTTP/1.1" 404 154 "-" "AppleNewsBot" "-"
2024/04/10 21:46:32 [error] 9#9: *972 open() "/var/www/certbot/.well-known/acme-challenge/tB5dl7_LERfVwg0yhJGTZgof4pA631_uyRUfSP65owY:" failed (2: No such file or directory), client: 17.58.57.20, server: test.33.foh.house, request: "GET /.well-known/acme-challenge/tB5dl7_LERfVwg0yhJGTZgof4pA631_uyRUfSP65owY: HTTP/1.1", host: "test.33.foh.house"
17.58.57.20 - - [10/Apr/2024:21:46:32 +0000] "GET /.well-known/acme-challenge/tB5dl7_LERfVwg0yhJGTZgof4pA631_uyRUfSP65owY: HTTP/1.1" 404 154 "-" "AppleNewsBot" "-"
2024/04/10 21:46:32 [error] 9#9: *973 open() "/var/www/certbot/.well-known/acme-challenge/tB5dl7_LERfVwg0yhJGTZgof4pA631_uyRUfSP65owY" failed (2: No such file or directory), client: 17.58.57.20, server: test.33.foh.house, request: "GET /.well-known/acme-challenge/tB5dl7_LERfVwg0yhJGTZgof4pA631_uyRUfSP65owY HTTP/1.1", host: "test.33.foh.house"
17.58.57.20 - - [10/Apr/2024:21:46:32 +0000] "GET /.well-known/acme-challenge/tB5dl7_LERfVwg0yhJGTZgof4pA631_uyRUfSP65owY HTTP/1.1" 404 154 "-" "AppleNewsBot" "-"
2024/04/10 21:46:32 [error] 9#9: *974 open() "/var/www/certbot/.well-known/acme-challenge/tB5dl7_LERfVwg0yhJGTZgof4pA631_uyRUfSP65owY:" failed (2: No such file or directory), client: 17.58.62.5, server: test.33.foh.house, request: "GET /.well-known/acme-challenge/tB5dl7_LERfVwg0yhJGTZgof4pA631_uyRUfSP65owY: HTTP/1.1", host: "test.33.foh.house"
17.58.62.5 - - [10/Apr/2024:21:46:32 +0000] "GET /.well-known/acme-challenge/tB5dl7_LERfVwg0yhJGTZgof4pA631_uyRUfSP65owY: HTTP/1.1" 404 154 "-" "AppleNewsBot" "-"
2024/04/10 21:46:33 [error] 9#9: *975 open() "/var/www/certbot/.well-known/acme-challenge/tB5dl7_LERfVwg0yhJGTZgof4pA631_uyRUfSP65owY:" failed (2: No such file or directory), client: 18.206.225.94, server: test.33.foh.house, request: "GET /.well-known/acme-challenge/tB5dl7_LERfVwg0yhJGTZgof4pA631_uyRUfSP65owY: HTTP/1.1", host: "test.33.foh.house"
18.206.225.94 - - [10/Apr/2024:21:46:33 +0000] "GET /.well-known/acme-challenge/tB5dl7_LERfVwg0yhJGTZgof4pA631_uyRUfSP65owY: HTTP/1.1" 404 154 "-" "Slackbot-LinkExpanding 1.0 (+https://api.slack.com/robots)" "-"
2024/04/10 21:51:33 [error] 9#9: *1048 open() "/var/www/certbot/.well-known/acme-challenge/tB5dl7_LERfVwg0yhJGTZgof4pA631_uyRUfSP65owY:" failed (2: No such file or directory), client: 17.58.58.21, server: test.33.foh.house, request: "GET /.well-known/acme-challenge/tB5dl7_LERfVwg0yhJGTZgof4pA631_uyRUfSP65owY: HTTP/1.1", host: "test.33.foh.house"
17.58.58.21 - - [10/Apr/2024:21:51:33 +0000] "GET /.well-known/acme-challenge/tB5dl7_LERfVwg0yhJGTZgof4pA631_uyRUfSP65owY: HTTP/1.1" 404 154 "-" "AppleNewsBot" "-"
2024/04/10 21:51:37 [error] 9#9: *1052 open() "/var/www/certbot/.well-known/acme-challenge/tB5dl7_LERfVwg0yhJGTZgof4pA631_uyRUfSP65owY:" failed (2: No such file or directory), client: 17.58.58.20, server: test.33.foh.house, request: "GET /.well-known/acme-challenge/tB5dl7_LERfVwg0yhJGTZgof4pA631_uyRUfSP65owY: HTTP/1.1", host: "test.33.foh.house"
17.58.58.20 - - [10/Apr/2024:21:51:37 +0000] "GET /.well-known/acme-challenge/tB5dl7_LERfVwg0yhJGTZgof4pA631_uyRUfSP65owY: HTTP/1.1" 404 154 "-" "AppleNewsBot" "-"
2024/04/10 21:51:37 [error] 9#9: *1053 open() "/var/www/certbot/.well-known/acme-challenge/tB5dl7_LERfVwg0yhJGTZgof4pA631_uyRUfSP65owY" failed (2: No such file or directory), client: 17.58.58.24, server: test.33.foh.house, request: "GET /.well-known/acme-challenge/tB5dl7_LERfVwg0yhJGTZgof4pA631_uyRUfSP65owY HTTP/1.1", host: "test.33.foh.house"
17.58.58.24 - - [10/Apr/2024:21:51:37 +0000] "GET /.well-known/acme-challenge/tB5dl7_LERfVwg0yhJGTZgof4pA631_uyRUfSP65owY HTTP/1.1" 404 154 "-" "AppleNewsBot" "-"
2024/04/10 21:51:37 [error] 9#9: *1054 open() "/var/www/certbot/.well-known/acme-challenge/tB5dl7_LERfVwg0yhJGTZgof4pA631_uyRUfSP65owY:" failed (2: No such file or directory), client: 17.58.58.22, server: test.33.foh.house, request: "GET /.well-known/acme-challenge/tB5dl7_LERfVwg0yhJGTZgof4pA631_uyRUfSP65owY: HTTP/1.1", host: "test.33.foh.house"
17.58.58.22 - - [10/Apr/2024:21:51:37 +0000] "GET /.well-known/acme-challenge/tB5dl7_LERfVwg0yhJGTZgof4pA631_uyRUfSP65owY: HTTP/1.1" 404 154 "-" "AppleNewsBot" "-"

You blocked everything outside USA (from keycdn ping test) but there are a few watchpoint in other continent

5 Likes

Those are the primary LE center and 2 secondary ones based in USA.

Two missing are outside the USA. You really should check your comms devices for a geographic based firewall

Many of the log recordfs on your second post are just junk. See "curl" or "AppleNewsBot" in the user-agent.

5 Likes

I've tried the keycdn ping test and am getting the same result. Everything outside the us is blocked but I tried it using the domain of the server I cloned as well and it is the same thing. They are all blocked but the let's encrypt certificate was created just fine on that server. So why is it a problem now? Its only a few months difference.

And I don't need anyone outside the US accessing the site, should that prevent me from getting an ssl certificate? that doesn't make any sense.

5 Likes

There was a recent change adding remote validation centers

If you are not willing or able to open port 80 for the HTTP Challenge you could use the DNS Challenge instead. Unless of course you run your own DNS Servers and block access by geography but that is unusual.

5 Likes

Yes, I am trying the DNS route. Thank You.

3 Likes