Verification fails but I see LE collecting the verification token from my server

Help
1

My domain is:
obfuscated

I ran this command:

certbot certonly --agree-tos --keep --text --email obfuscated --server https://acme-staging-v02.api.letsencrypt.org/directory -d obfuscated --http-01-port 1337 --standalone --preferred-challenges http-01 --debug

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for obfuscated
Waiting for verification...
Challenge failed for domain obfuscated
http-01 challenge for obfuscated
Cleaning up challenges
Exiting abnormally:
Traceback (most recent call last):
  File "/usr/local/bin/certbot", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/dist-packages/certbot/main.py", line 15, in main
    return internal_main.main(cli_args)
  File "/usr/local/lib/python3.7/dist-packages/certbot/_internal/main.py", line 1347, in main
    return config.func(config, plugins)
  File "/usr/local/lib/python3.7/dist-packages/certbot/_internal/main.py", line 1233, in certonly
    lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
  File "/usr/local/lib/python3.7/dist-packages/certbot/_internal/main.py", line 121, in _get_and_save_cert
    lineage = le_client.obtain_and_enroll_certificate(domains, certname)
  File "/usr/local/lib/python3.7/dist-packages/certbot/_internal/client.py", line 409, in obtain_and_enroll_certificate
    cert, chain, key, _ = self.obtain_certificate(domains)
  File "/usr/local/lib/python3.7/dist-packages/certbot/_internal/client.py", line 343, in obtain_certificate
    orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
  File "/usr/local/lib/python3.7/dist-packages/certbot/_internal/client.py", line 390, in _get_order_and_authorizations
    authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
  File "/usr/local/lib/python3.7/dist-packages/certbot/_internal/auth_handler.py", line 91, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, best_effort)
  File "/usr/local/lib/python3.7/dist-packages/certbot/_internal/auth_handler.py", line 180, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.
Please see the logfiles in /var/log/letsencrypt for more details.

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: obfuscated
   Type:   connection
   Detail: Fetching
   http://obfuscated/.well-known/acme-challenge/1Nq67fUeLXrxMoYQt_-U3fI1vKIgKx6ba4CcYjr4IdM:
   Timeout during connect (likely firewall problem)

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.

My web server is (include version):
nginx 1.19.0, combined with Certbot 1.5.0 in this docker image: https://github.com/staticfloat/docker-nginx-certbot

The operating system my web server runs on is (include version):
Ubuntu 20.04 LTS

My hosting provider, if applicable, is:
NA

I can login to a root shell on my machine (yes or no, or I don’t know):
yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):
No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot):
Certbot 1.5.0

I’m trying to get a certificate using the docker Image I linked earlier. The automatic process failing so I started debugging but have hit a dead end.
Initially I noticed that while Certbot was trying to verify my site, the Nginx log did show a request coming from the load balancers’s IP (forwarding the request of the LE verification server presumably) to collect the challenge response, and nginx returned a 200.

So next thing I did was set up a tcpdump monitoring session on the host server that runs this docker container, and there I could also verify the challenge response being returned to my LB’s IP:

09:22:43.641352 IP (tos 0x0, ttl 62, id 18659, offset 0, flags [DF], proto TCP (6), length 313)
    10.51.11.103.46734 > 172.18.0.14.80: Flags [P.], cksum 0xc57a (correct), seq 1:274, ack 1, win 229, length 273: HTTP, length: 273
        GET /.well-known/acme-challenge/1Nq67fUeLXrxMoYQt_-U3fI1vKIgKx6ba4CcYjr4IdM HTTP/1.1
        Host: obfuscated
        User-Agent: Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)
        Accept: */*
        Accept-Encoding: gzip
        Connection: close

09:22:43.641358 IP (tos 0x0, ttl 62, id 18659, offset 0, flags [DF], proto TCP (6), length 313)
    10.51.11.103.46734 > 172.18.0.14.80: Flags [P.], cksum 0xc57a (correct), seq 1:274, ack 1, win 229, length 273: HTTP, length: 273
        GET /.well-known/acme-challenge/1Nq67fUeLXrxMoYQt_-U3fI1vKIgKx6ba4CcYjr4IdM HTTP/1.1
        Host: obfuscated
        User-Agent: Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)
        Accept: */*
        Accept-Encoding: gzip
        Connection: close

09:22:43.641370 IP (tos 0x0, ttl 64, id 13051, offset 0, flags [DF], proto TCP (6), length 40)
    172.18.0.14.80 > 10.51.11.103.46734: Flags [.], cksum 0xc1d4 (incorrect -> 0xff49), ack 274, win 501, length 0
09:22:43.641370 IP (tos 0x0, ttl 64, id 13051, offset 0, flags [DF], proto TCP (6), length 40)
    172.18.0.14.80 > 10.51.11.103.46734: Flags [.], cksum 0xc1d4 (incorrect -> 0xff49), ack 274, win 501, length 0
09:22:43.642807 IP (tos 0x0, ttl 64, id 13052, offset 0, flags [DF], proto TCP (6), length 263)
    172.18.0.14.80 > 10.51.11.103.46734: Flags [P.], cksum 0xc2b3 (incorrect -> 0x5582), seq 1:224, ack 274, win 501, length 223: HTTP, length: 223
        HTTP/1.1 200 OK
        Server: nginx/1.19.0
        Date: Fri, 14 Aug 2020 07:22:43 GMT
        Transfer-Encoding: chunked
        Connection: close

        57
        1Nq67fUeLXrxMoYQt_-U3fI1vKIgKx6ba4CcYjr4IdM.akD02rC-U7IEe3TjDIy3F7sjnc8l_xu5J9SMt7c32lc
        0

09:22:43.642807 IP (tos 0x0, ttl 64, id 13052, offset 0, flags [DF], proto TCP (6), length 263)
    172.18.0.14.80 > 10.51.11.103.46734: Flags [P.], cksum 0xc2b3 (incorrect -> 0x5582), seq 1:224, ack 274, win 501, length 223: HTTP, length: 223
        HTTP/1.1 200 OK
        Server: nginx/1.19.0
        Date: Fri, 14 Aug 2020 07:22:43 GMT
        Transfer-Encoding: chunked
        Connection: close

        57
        1Nq67fUeLXrxMoYQt_-U3fI1vKIgKx6ba4CcYjr4IdM.akD02rC-U7IEe3TjDIy3F7sjnc8l_xu5J9SMt7c32lc
        0

To be 100% sure that my setup was working, I ran the certbot command again with the --debug-challenges flag, which allowed me to test the challenge response retrieval before initiating the LE verification. I was indeed able to retrieve the response using both my home internet connection and my smartphone’s 4G connection (2 different ISP’s, no VPN’s).

So what could this be?
My server allows inbound traffic on TCP ports 80 and 443 (through a load balancer) and allows outbound traffic to the internet on TCP ports 80 and 443 as well.

2

Hi @agKaspar

checking your url:

http://test.crea.nkcn-cia.be/.well-known/acme-challenge/1Nq67fUeLXrxMoYQt_-U3fI1vKIgKx6ba4CcYjr4IdM

I see a Bad Gateway - http status 502.

If Letsencrypt has a Timeout, there may be a regional firewall. So you may see some of the Letsencrypt checks, but not all -> other -> timeout.

3

That challenge response is not being hosted by certbot anymore, since it has failed, hence the 502.

I have now just run the following command:

certbot certonly --agree-tos --keep --text --email <obfuscated> --server https://acme-staging-v02.api.letsencrypt.org/directory -d obfuscated --http-01-port 1337 --standalone --preferred-challenges http-01 --debug --debug-challenges -v

And got the following output:

Root logging level set at 10
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Requested authenticator standalone and installer None
Single candidate plugin: * standalone
Description: Spin up a temporary webserver
Interfaces: IAuthenticator, IPlugin
Entry point: standalone = certbot._internal.plugins.standalone:Authenticator
Initialized: <certbot._internal.plugins.standalone.Authenticator object at 0x7fc649df4dd8>
Prep: True
Selected authenticator <certbot._internal.plugins.standalone.Authenticator object at 0x7fc649df4dd8> and installer None
Plugins selected: Authenticator standalone, Installer None
Picked account: <Account(RegistrationResource(body=Registration(key=None, contact=(), agreement=None, status=None, terms_of_service_agreed=None, only_return_existing=None, external_account_binding=None), uri='https://acme-staging-v02.api.letsencrypt.org/acme/acct/15154385', new_authzr_uri=None, terms_of_service=None), dc6c2a08d8e0821bae5e82492b5499f9, Meta(creation_dt=datetime.datetime(2020, 8, 14, 8, 56, 20, tzinfo=<UTC>), creation_host='77521dc9fbdf', register_to_eff=None))>
Sending GET request to https://acme-staging-v02.api.letsencrypt.org/directory.
Starting new HTTPS connection (1): acme-staging-v02.api.letsencrypt.org:443
https://acme-staging-v02.api.letsencrypt.org:443 "GET /directory HTTP/1.1" 200 724
Received response:
HTTP 200
Server: nginx
Date: Fri, 14 Aug 2020 08:57:30 GMT
Content-Type: application/json
Content-Length: 724
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "cMViqKVGgQE": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
  "keyChange": "https://acme-staging-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
    "website": "https://letsencrypt.org/docs/staging-environment/"
  },
  "newAccount": "https://acme-staging-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-staging-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-staging-v02.api.letsencrypt.org/acme/new-order",
  "revokeCert": "https://acme-staging-v02.api.letsencrypt.org/acme/revoke-cert"
}
Obtaining a new certificate
Generating key (2048 bits): /etc/letsencrypt/keys/0003_key-certbot.pem
Creating CSR: /etc/letsencrypt/csr/0003_csr-certbot.pem
Requesting fresh nonce
Sending HEAD request to https://acme-staging-v02.api.letsencrypt.org/acme/new-nonce.
https://acme-staging-v02.api.letsencrypt.org:443 "HEAD /acme/new-nonce HTTP/1.1" 200 0
Received response:
HTTP 200
Server: nginx
Date: Fri, 14 Aug 2020 08:57:30 GMT
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: 0002kVA4qNbjGuaiM8IUBLjJR-pQt_yrfNyk7F5cCFaAkuc
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800


Storing nonce: 0002kVA4qNbjGuaiM8IUBLjJR-pQt_yrfNyk7F5cCFaAkuc
JWS payload:
b'{\n  "identifiers": [\n    {\n      "type": "dns",\n      "value": "obfuscated"\n    }\n  ]\n}'
Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/new-order:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS1zdGFnaW5nLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYWNjdC8xNTE1NDM4NSIsICJub25jZSI6ICIwMDAya1ZBNHFOYmpHdWFpTThJVUJMakpSLXBRdF95cmZOeWs3RjVjQ0ZhQWt1YyIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXN0YWdpbmctdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9uZXctb3JkZXIifQ",
  "signature": "Z4fzqJn_RzAzBv896oEn7XMhv-oqDmTqgy97NhZ6iSCJ2HT7y3GgoYEsLBIn-qVmftZ7JDQnTE62GskHkudJ6TL3HORbN0ALW1GKjmYzTQ2aZNACn5hvcCQ16nYnkSjGK13r_qooccGTD4FJVRv7_7KvZbl2EwEelDhSaFx8QmpniHsOYbZnt3EXlCxBRj8lMQ9Fdqa1DZuXvZYN0jWlL4zRGwamvGr_zfs_RCA3Ox4ihvoD3vx6HedH7Dgd5eGgqXb2G-LjWgD_baB-fBnpPEURuNiwTHIY51o9c1VGzBhTkxLPiVdx2RMMPUyHo8gtL0z9VtJ1Cm-rEIN4goHSNw",
  "payload": "ewogICJpZGVudGlmaWVycyI6IFsKICAgIHsKICAgICAgInR5cGUiOiAiZG5zIiwKICAgICAgInZhbHVlIjogInRlc3QuY3JlYS5ua2NuLWNpYS5iZSIKICAgIH0KICBdCn0"
}
https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/new-order HTTP/1.1" 201 354
Received response:
HTTP 201
Server: nginx
Date: Fri, 14 Aug 2020 08:57:30 GMT
Content-Type: application/json
Content-Length: 354
Connection: keep-alive
Boulder-Requester: 15154385
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
Location: https://acme-staging-v02.api.letsencrypt.org/acme/order/15154385/131465925
Replay-Nonce: 00028vBL1Fh7M3dKnGlvyrsh0lvos58Fb17Bw-LodjA8Qak
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "status": "pending",
  "expires": "2020-08-21T08:56:36Z",
  "identifiers": [
    {
      "type": "dns",
      "value": "obfuscated"
    }
  ],
  "authorizations": [
    "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/94109956"
  ],
  "finalize": "https://acme-staging-v02.api.letsencrypt.org/acme/finalize/15154385/131465925"
}
Storing nonce: 00028vBL1Fh7M3dKnGlvyrsh0lvos58Fb17Bw-LodjA8Qak
JWS payload:
b''
Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/94109956:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS1zdGFnaW5nLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYWNjdC8xNTE1NDM4NSIsICJub25jZSI6ICIwMDAyOHZCTDFGaDdNM2RLbkdsdnlyc2gwbHZvczU4RmIxN0J3LUxvZGpBOFFhayIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXN0YWdpbmctdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9hdXRoei12My85NDEwOTk1NiJ9",
  "signature": "lWjH7J_EXQJHMKpAlMUMBUUwP8VTIo_Ywz5rj5_MUoH7-nsfyKecqwvTsudOzhICKsdVIJKndmbfXkHxC-o0Y3NrnJ9fnZNK3RH5t3f_IQ3OyvAtpfVdFTCnOvPw9pkAeqdpWcbB8lvFMfC0UA5ZU8e0C9CNRxEdi_RlOXpniUhuzZS7uVLR7_2cWh3BnkVcpiFKQqffKdvmLYMK7ZtDwZsm5FdBRuwaaknrk7gqbvbQHnk-0t5z5QVd6Uw9vhv1yLd2bwO9DuHRIlezU4quCWyRRP_fXIqB24X50Vj6NUv1K45UmCgVaNrFG7PEf2iw5j7PktBQwEdRUljDZYBB2w",
  "payload": ""
}
https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/94109956 HTTP/1.1" 200 817
Received response:
HTTP 200
Server: nginx
Date: Fri, 14 Aug 2020 08:57:30 GMT
Content-Type: application/json
Content-Length: 817
Connection: keep-alive
Boulder-Requester: 15154385
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: 0001D69fE34q9YtjKPrLZgk-bE_x5bKmvr2fhKpyV2-3Xjc
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "obfuscated"
  },
  "status": "pending",
  "expires": "2020-08-21T08:56:36Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "pending",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/94109956/3iAKlA",
      "token": "MRwXHLne3dQYNnPeT-9l6kqmSxwd-DItfEHQx8R2DCA"
    },
    {
      "type": "dns-01",
      "status": "pending",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/94109956/brWXJg",
      "token": "MRwXHLne3dQYNnPeT-9l6kqmSxwd-DItfEHQx8R2DCA"
    },
    {
      "type": "tls-alpn-01",
      "status": "pending",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/94109956/pqgUpA",
      "token": "MRwXHLne3dQYNnPeT-9l6kqmSxwd-DItfEHQx8R2DCA"
    }
  ]
}
Storing nonce: 0001D69fE34q9YtjKPrLZgk-bE_x5bKmvr2fhKpyV2-3Xjc
Performing the following challenges:
http-01 challenge for obfuscated
Successfully bound to :1337 using IPv6
Certbot wasn't able to bind to :1337 using IPv4, this is often expected due to the dual stack nature of IPv6 socket implementations.
Waiting for verification...

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Challenges loaded. Press continue to submit to CA. Pass "-v" for more info about
challenges.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Press Enter to Continue

So now certbot is waiting for me to press ENTER to continue the verification.
If I now go to http://obfuscated/.well-known/acme-challenge/MRwXHLne3dQYNnPeT-9l6kqmSxwd-DItfEHQx8R2DCA (using the new token from the log above)
It works for me.

I don’t know how long certbot will wait for me to press ENTER, because if it fails or continues, that link won’t work again

4

So that proofs your configuration is completely buggy.

--standalone starts an own webserver, so you can never see the static validation file.

And there

http://test.crea.nkcn-cia.be/.well-known/acme-challenge/MRwXHLne3dQYNnPeT-9l6kqmSxwd-DItfEHQx8R2DCA

runs now a webserver, so why do you use standalone?

Please start there:

https://certbot.eff.org/docs/using.html

5

Could you elaborate on what you consider buggy, please?

The way this docker image works, is that it sets up Certbot in standalone mode (with its werserver listening on localhost:1773), while configuring a reverse proxy route in nginx to forward any /.well-known/acme-challenge/* requests to the certbot server. That explains why the 502 happens when certbot isn’t listening. But still, the fact that the current challenge request works, should allow verification to succeed, no?

6

Let's Encrypt uses multiple vantage points from different regional datacenters to perform the validation.

1 Like
7

Just to make this a little more straightforward, here are the 4 IPs that I see when making a certificate order right now:

66.133.109.36 - - [14/Aug/2020:20:26:32 +1000] "GET /.well-known/acme-challenge/jNgdyVXZc1o13JR3ftmYY0VmSi9HmUUSBmirdyVDBOE HTTP/1.1" 404 118 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
34.209.232.166 - - [14/Aug/2020:20:26:32 +1000] "GET /.well-known/acme-challenge/jNgdyVXZc1o13JR3ftmYY0VmSi9HmUUSBmirdyVDBOE HTTP/1.1" 404 118 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
3.128.26.105 - - [14/Aug/2020:20:26:32 +1000] "GET /.well-known/acme-challenge/jNgdyVXZc1o13JR3ftmYY0VmSi9HmUUSBmirdyVDBOE HTTP/1.1" 404 118 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
52.28.236.88 - - [14/Aug/2020:20:26:32 +1000] "GET /.well-known/acme-challenge/jNgdyVXZc1o13JR3ftmYY0VmSi9HmUUSBmirdyVDBOE HTTP/1.1" 404 118 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"

@agKaspar based on your error message, I would say that the primary validation server (66.133.109.36) is failing to connect to your LB. It might be easier if you can record some pcaps on your LB rather than doing so inside your NAT.

(Future readers - these IPs regularly change, don’t rely on them).

2 Likes
8

This is useful, thanks!
Unfortunately I don’t have any acces to my LB, but I’ve asked the relevant people to take a look at the logs and see what’s being blocked. I’ll report back later

9

@Osiris @_az the LB is indeed protected by geofencing rules, which likely explains why not all requests from LE were succeeding.
Since it’s a requirement for this project to keep the geofence rules in place, I’ll have to get a certificate elsewhere.

Thanks for the help regardless

10

Or use the dns-01 challenge, unless your DNS also has such geo rules.

Also, perhaps your LB can have an exemption for the /.well-known/acme-challenge/ path.

11

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.