Renew certificate failed due to secondary validation

Help
1

Hello!
Dedicated server could not renew the certificate by certbot 1.27.0.
The details are below (I changed sensitive data to "mydomain.com", "my.ip.address" and "my_token". I can send this data to Letsencrypt staff).

certbot error:

Domain: mydomain.com
  Type:   connection
  Detail: During secondary validation: my.ip.address: Fetching http://mydomain.com/.well-known/acme-challenge/my_token: Timeout during connect (likely firewall problem)

Web server access log shows two times success GET my_token by Let's Encrypt validation server:

[US][mydomain.com] 54.245.188.205 - [31/May/2022:17:38:56 +0300] "GET /.well-known/acme-challenge/my_token HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-" [Boardman, United States]
[US][mydomain.com] 66.133.109.36 - [31/May/2022:17:38:56 +0300] "GET /.well-known/acme-challenge/my_token HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-" [-, United States]

The file /.well-known/acme-challenge/my_token is actually created during cerbot working, I have checked it.

The site http://mydomain.com is accessible across all the world (I checked it too).
The site is alive and has many visitors.
The server is located in EU (Hetzner, Germany).
Of course, 80 and 443 ports are opened at firewall.
Test with some file http://mydomain.com/.well-known/acme-challenge/some_file_for_test works well.

What's else may be wrong?

letsencrypt.log fragments concerning my_token

2022-05-31 17:38:49,739:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/114566122636 HTTP/1.1" 200 800
2022-05-31 17:38:49,740:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Tue, 31 May 2022 14:38:49 GMT
Content-Type: application/json
Content-Length: 800
Connection: keep-alive
Boulder-Requester: 379456920
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: 0002elZtyPZTNeEGJQyrK5QDbr3jByJw4zHTrcPDI1_fDto
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "mydomain.com"
  },
  "status": "pending",
  "expires": "2022-06-07T14:38:46Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/114566122636/1MnokQ",
      "token": "my_token"
    },
    {
      "type": "dns-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/114566122636/9cXdyg",
      "token": "my_token"
    },
    {
      "type": "tls-alpn-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/114566122636/rWaWGQ",
      "token": "my_token"
    }
  ]
}

...

2022-05-31 17:38:53,439:DEBUG:certbot._internal.plugins.webroot:Attempting to save validation to /home/ac/www/.well-known/acme-challenge/my_token

...

2022-05-31 17:38:56,219:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "POST /acme/chall-v3/114566122636/1MnokQ HTTP/1.1" 200 187
2022-05-31 17:38:56,220:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Tue, 31 May 2022 14:38:56 GMT
Content-Type: application/json
Content-Length: 187
Connection: keep-alive
Boulder-Requester: 379456920
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index", <https://acme-v02.api.letsencrypt.org/acme/authz-v3/114566122636>;rel="up"
Location: https://acme-v02.api.letsencrypt.org/acme/chall-v3/114566122636/1MnokQ
Replay-Nonce: 0002OHlzxVzTs88GU5RrgT53xAUgJAqNCFA16iXrp02Fr9o
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "type": "http-01",
  "status": "pending",
  "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/114566122636/1MnokQ",
  "token": "my_token"
}
2022-05-31 17:38:56,220:DEBUG:acme.client:Storing nonce: 0002OHlzxVzTs88GU5RrgT53xAUgJAqNCFA16iXrp02Fr9o
2022-05-31 17:38:56,220:DEBUG:acme.client:JWS payload:
b'{}'

...

2022-05-31 17:39:04,393:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/114566122636 HTTP/1.1" 200 800
2022-05-31 17:39:04,393:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Tue, 31 May 2022 14:39:04 GMT
Content-Type: application/json
Content-Length: 800
Connection: keep-alive
Boulder-Requester: 379456920
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: 0001mZE5131_TmS-dDqCu_AYAV3Aet5J8LdRf_odBvVyeGw
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "mydomain.com"
  },
  "status": "pending",
  "expires": "2022-06-07T14:38:46Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/114566122636/1MnokQ",
      "token": "my_token"
    },
    {
      "type": "dns-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/114566122636/9cXdyg",
      "token": "my_token"
    },
    {
      "type": "tls-alpn-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/114566122636/rWaWGQ",
      "token": "my_token"
    }
  ]
}

...

2022-05-31 17:39:13,516:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/authz-v3/114566122636:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2FjY3QvMzc5NDU2OTIwIiwgIm5vbmNlIjogIjAwMDI1YXFiQ3E5ZWxJUU5DTW9tYmxZcHpSRHdER3ZldU9VeW9PZ2VZa0FmM0lFIiwgInVybCI6ICJodHRwczovL2FjbWUtdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9hdXRoei12My8xMTQ1NjYxMjI2MzYifQ",
  "signature": "N86vbdLA-BjNYqsZYnWpDUCFocUFllimCWu2-VKWn2TOQUJdeDorshnXwMG9iNKwmvvXV8Ru7rGiIKmA8O6A14cllZNGWKLotRArsTshwZhEHgrLTZvTmbhCHd6nyIh4autrEVxp5b5uwaE18LrVIfzvgU-asdlQp9yJGZwj9f-8Wps20x0SCllSj_pWXmdrNtx1_KPyyJawiir7pRqbnvZiFIHlDgpQwHv694Cw6LOHx8l3nBZogxlmNKElYTpHH7eD64elDPNEbxOiS0hDXl2_Z0ENAzxv-j6j638DkRA2UIvZKalr4mC6i6yQ5gZQmoMiww-QLMyqozJfjYmh2g",
  "payload": ""
}
2022-05-31 17:39:13,667:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/114566122636 HTTP/1.1" 200 1088
2022-05-31 17:39:13,668:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Tue, 31 May 2022 14:39:13 GMT
Content-Type: application/json
Content-Length: 1088
Connection: keep-alive
Boulder-Requester: 379456920
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: 00019y9nDVAAyppVQAoYOtB998W41vlUYNOhD0JZoNNvhI8
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "mydomain.com"
  },
  "status": "invalid",
  "expires": "2022-06-07T14:38:46Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "invalid",
      "error": {
        "type": "urn:ietf:params:acme:error:connection",
        "detail": "During secondary validation: my.ip.address: Fetching http://mydomain.com/.well-known/acme-challenge/my_token: Timeout during connect (likely firewall problem)",
        "status": 400
      },
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/114566122636/1MnokQ",
      "token": "my_token",
      "validationRecord": [
        {
          "url": "http://mydomain.com/.well-known/acme-challenge/my_token",
          "hostname": "mydomain.com",
          "port": "80",
          "addressesResolved": [
            "my.ip.address"
          ],
          "addressUsed": "my.ip.address"
        }
      ],
      "validated": "2022-05-31T14:38:56Z"
    }
  ]
}
2022-05-31 17:39:13,668:DEBUG:acme.client:Storing nonce: 00019y9nDVAAyppVQAoYOtB998W41vlUYNOhD0JZoNNvhI8
2022-05-31 17:39:13,669:DEBUG:acme.client:JWS payload:

...

  Domain: mydomain.com
  Type:   connection
  Detail: During secondary validation: my.ip.address: Fetching http://mydomain.com/.well-known/acme-challenge/my_token: Timeout during connect (likely firewall problem)

...

2022-05-31 17:39:22,207:DEBUG:certbot._internal.plugins.webroot:Removing /home/mydomain.com/www/.well-known/acme-challenge/my_token

...

2022-05-31 17:39:22,213:ERROR:certbot._internal.renewal:Failed to renew certificate mydomain.com with error: Some challenges have failed.
2022-05-31 17:39:22,213:DEBUG:certbot._internal.renewal:Traceback was:
Traceback (most recent call last):
  File "/snap/certbot/2035/lib/python3.8/site-packages/certbot/_internal/renewal.py", line 484, in handle_renewal_request
    main.renew_cert(lineage_config, plugins, renewal_candidate)
  File "/snap/certbot/2035/lib/python3.8/site-packages/certbot/_internal/main.py", line 1541, in renew_cert
    renewed_lineage = _get_and_save_cert(le_client, config, lineage=lineage)
  File "/snap/certbot/2035/lib/python3.8/site-packages/certbot/_internal/main.py", line 129, in _get_and_save_cert
    renewal.renew_cert(config, domains, le_client, lineage)
  File "/snap/certbot/2035/lib/python3.8/site-packages/certbot/_internal/renewal.py", line 344, in renew_cert
    new_cert, new_chain, new_key, _ = le_client.obtain_certificate(domains, new_key)
  File "/snap/certbot/2035/lib/python3.8/site-packages/certbot/_internal/client.py", line 441, in obtain_certificate
    orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
  File "/snap/certbot/2035/lib/python3.8/site-packages/certbot/_internal/client.py", line 493, in _get_order_and_authorizations
    authzr = self.auth_handler.handle_authorizations(orderr, self.config, best_effort)
  File "/snap/certbot/2035/lib/python3.8/site-packages/certbot/_internal/auth_handler.py", line 106, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, best_effort)
  File "/snap/certbot/2035/lib/python3.8/site-packages/certbot/_internal/auth_handler.py", line 206, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.

2022-05-31 17:39:22,214:DEBUG:certbot._internal.display.obj:Notifying user: 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2022-05-31 17:39:22,215:ERROR:certbot._internal.renewal:All renewals failed. The following certificates could not be renewed:
2022-05-31 17:39:22,215:ERROR:certbot._internal.renewal:  /etc/letsencrypt/live/mydomain.com/fullchain.pem (failure)
2022-05-31 17:39:22,215:DEBUG:certbot._internal.display.obj:Notifying user: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2022-05-31 17:39:22,215:DEBUG:certbot._internal.log:Exiting abnormally:
Traceback (most recent call last):
  File "/snap/certbot/2035/bin/certbot", line 8, in <module>
    sys.exit(main())
  File "/snap/certbot/2035/lib/python3.8/site-packages/certbot/main.py", line 19, in main
    return internal_main.main(cli_args)
  File "/snap/certbot/2035/lib/python3.8/site-packages/certbot/_internal/main.py", line 1744, in main
    return config.func(config, plugins)
  File "/snap/certbot/2035/lib/python3.8/site-packages/certbot/_internal/main.py", line 1630, in renew
    renewal.handle_renewal_request(config)
  File "/snap/certbot/2035/lib/python3.8/site-packages/certbot/_internal/renewal.py", line 510, in handle_renewal_request
    raise errors.Error(
certbot.errors.Error: 1 renew failure(s), 0 parse failure(s)
2022-05-31 17:39:22,215:ERROR:certbot._internal.log:1 renew failure(s), 0 parse failure(s)
2

Currently Let's Encrypt production servers will try from 4 locations (locations and number may change in future).

Since some requests got through one guess is your firewall has some sort of DDOS prevention or "smart blocking" which does not let repeated requests from different IPs.

3 Likes
3

I would expect to see three or four connections, not just two. The “secondary validation” servers are currently in AWS. Can you make sure your website is reachable from AWS eu-central-1 (Frankfurt)?

3 Likes
4

Your domain and IP address are not sensitive.

Perhaps you missed this bit of text in the template:

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

3 Likes
5

How?

my iptables firewall blocks by Geo IP some Asian countries and some extra IPs.
But I also run certbot with all firewall blocks removed (and got the same failure).

6

No, no, I did not miss this text.
But this data is sensitive for me.
Of course, I can send this data to Letsencrypt team.

7

How are you connected to the internet?

If you are self-hosted at home does your router have any kind of firewall?

If at a company or university, do they have another firewall?

2 Likes
8

If you got the same failure, then you have not run certbot with all firewall blocks removed.

3 Likes
9

Tried https://ping.psa.fun/

aws
aws1920Ă—1030 261 KB

10

Dedicated server at Hetzner (DE).
Hetzner has own firewall for each dedicated server but I do not use it all.
Internal Hetzner firewall is OFF.
I use iptables only.

All my iptables rules are in the one cmd file .
Sure I removed all the lines with blocks while trying to get certbot working.
Moreover certbot worked fine for several years with these rules ...

11

You might also be interested in Amazon Connect Tools, which was linked from some AWS documentation page. Looks like it runs from a browser to AWS, so you probably need to run it on your server somehow. Or use a VPN to your server for example.

3 Likes
12

I used VPN.
This service shows SUCCESS for all locations except "GovCloud us-gov-west-1" (failure)

    {
      "name": "GovCloud",
      "code": "us-gov-west-1",
      "success": false,
      "mediaItems": [
        {
          "endpoint": "stun:TurnNlb-d7c623c23f628042.elb.us-gov-west-1.amazonaws.com:3478",
          "candidate": null,
          "success": false,
          "timedOut": true,
          "error": "Timeout"
        }
      ]
    }

aws2
aws21723Ă—486 97.6 KB

13

At Hetzner? If so, have you asked them if they changed anything? That some requests get through is strongly indicative of a firewall block in "your" equipment (that is, not LE).

As example, we recently had a vendor of firewalls (Palo Alto Networks) change a setting that blocked acme http challenge URLs. This is not your symptom but perhaps Hetzner has started using some new equipment or software to protect their infrastructure.

Just curious, can you reproduce failure with letsdebug.net ?

3 Likes
14

already.

Let's Debug
Test result for mydomain.com using http-01
All OK!
OK

No issues were found with mydomain.com. If you are having problems with creating an SSL certificate, please visit the Let's Encrypt Community forums and post a question there.
15

Interesting. I would have to double-check but last time I looked the Let's Encrypt test system only used US based servers to test authentication. The difference is that production LE servers use a 4th in Germany. The Let's Debug also uses the LE test system.

Can you see 3 requests in your webserver logs and see if the 3 LE test requests were all US?

2 Likes
16

My logs show only 2 requests from US (see my first post).

17

Yes, on your failed try with production LE servers.

But, you should also see the results from the Let's Debug test. It uses the LE test system so you should see 3 successful challenge requests. (Update: well, not successful as in http 200 but successful as test to show http 404)

2 Likes
18

Let's Debug test gave only 2 lines (404 error) in the web server access log:

[US][mydomain.com] 172.104.24.29 - [31/May/2022:19:42:04 +0300] "GET /.well-known/acme-challenge/letsdebug-test HTTP/1.1" 404 6 "-" "Mozilla/5.0 (compatible; Let's Debug emulating Let's Encrypt validation server; +https://letsdebug.net)" "-" [Cedar Knolls, United States]
[US][mydomain.com] 66.133.109.36 - [31/May/2022:19:42:05 +0300] "GET /.well-known/acme-challenge/bM0Z8BsEO8B31jfsqq7X8Em0z_TtDcIBbsE94ZN_QCU HTTP/1.1" 404 6 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-" [-, United States]
19

Hmmm. I just tried and I get 3 total requests in my server logs. One with user-agent saying letsdebug.net for its test file (same IP 172.104.24.29) and 2 requests from LE server (user agent has letsencrypt.org)

In any event, I thought this might be helpful to compare to your production request but I guess not.

2 Likes
20

The logs only show the requests that have made it past whatever is blocking them.
So those logs aren't very useful [at showing what never made it to the log].
The place to look is within the path; Anything that could be doing such blocking.

1 Like