Let's Encrypt Timeouts

[anneal]

Around 11am CST today, our Let's Encrypt client (using acme-client) started receiving error responses from Let's Encrypt of these two formats:

Fetching https://ourcentralvalidationserver.com/acme-challenge/SZhujNBrHQdPUTEj3_DqMAJZ46nJxjUwRJjdkD7o4BI: Error getting validation data

Fetching https://ourcentralvalidationserver.com/acme-challenge/Q5Qy0neRk0e3EqGKLJvib7BeQJ4LrkyoLKogR1LuNSU: Timeout after connect (your server may be slow or overloaded)

We are using version 0.4.1 of acme-client. Docs here: GitHub - unixcharles/acme-client at 0.3.7
These errors show up in parsing the response to:
challenge = acme_client.challenge_from_hash(challenge_data) status = challenge.verify_status

Our logs continue to show that our central validation server returned a 200 responses to these requests within 100 ms. Can anyone tell us the timeout length so we can check if response times are too long? Any other ideas?

[stevenzhu]

Hi,

Hmm… That is not the real domain name, right?

If you aren’t showing us the right domain name, how do we help?

Thank you

[schoen]

I’m guessing that’s not the real domain name, but it would be helpful to know the real domain name so we could look for, for example, an IPv6 misconfiguration.

[anneal]

Yes, we did not show the actual domain of our central validation server. This is only occurring for some acme challenge responses, not all of them.

Our validation server does not have IPv6 configured.

[cpu]

@anneal

Can anyone tell us the timeout length so we can check if response times are too long?

The timeout is 10s. From this side it looks like the timeout was accurately reported.

[anneal]

Thanks! We’ll dig in to see if we can reconcile this with our logging.

[Osiris]

Isn’t it possible to add more information to the error messages reported by Boulder? For instance, I think it would be very helpful if some error messages contained the IP address used. If an user would see an IP address he/she doesn’t recognise in the error message, that could help a lot.

[cpu]

@Osiris

For instance, I think it would be very helpful if some error messages contained the IP address used. If an user would see an IP address he/she doesn’t recognise in the error message, that could help a lot.

This information is present in the validation records returned by the Let's Encrypt ACME Server for the challenge that failed validation. E.g.:

  "challenges": [
    {
      "type": "http-01",
      "status": "invalid",
      "error": {
        "type": "urn:acme:error:connection",
        "detail": "Fetching http://example.com/.well-known/acme-challenge/token: Timeout during connect (likely firewall problem)",
        "status": 400
      },
      "uri": "https://acme-v01.api.letsencrypt.org/acme/challenge/<foo>/<bar>",
      "token": "<token>",
      "keyAuthorization": "<keyauth>",
      "validationRecord": [
        {
          "url": "http://example.com/.well-known/acme-challenge/<token>",
          "hostname": "example.com",
          "port": "80",
          "addressesResolved": [
            "xx.xx.xx.xx",
            "yy.yy.yy.yy"
          ],
          "addressUsed": "xx.xx.xx.xx"
        }
      ]
    },

Its up to individual clients to decide how much of this information to present for cases like this. The IP address being incorrect is only one possible reason for this error case.

[Osiris]

@cpu

This information is present in the validation records returned by the Let’s Encrypt ACME Server for the challenge that failed validation. E.g.:

(…)

Its up to individual clients to decide how much of this information to present for cases like this. The IP address being incorrect is only one possible reason for this error case.

Ah yes, makes sense. Thus, it would be helpful if clients would pass that particular piece of info to the client

[JuergenAuer]

@Osiris

Thus, it would be helpful if clients would pass that particular piece of info to the client

The client program may save the order-url, something like

https://acme-v02.api.letsencrypt.org/acme/order/AccountId/OrderId

There are the authorization-urls (one per domain name). Open per browser, there is the challenge-list with the validationRecord - information.

So if the client-program saves the order-url, the information can be found with a browser.