Let's Encrypt http-01 ACME challenge fails with timeout

Peter Cooper Jr. via Let's Encrypt Community Support:

Wow, the actual Tor project. Welcome to the Let's Encrypt community! And thank you for all you do.

Thanks! Same goes for all the folks involved in the Let's Encrypt
efforts. <3

I don't recall seeing a "Network unreachable" error from the Let's Encrypt validation servers before. It's probably happened, but that's got to be a rare one. (I've seen it plenty from systems that were misconfigured outbound to not be able to reach the Let's Encrypt servers, but you seem to have covered that.) So this might be tricky, but we'll see what we can do to help. I've got some questions for you, though I'm not sure what the solution would be even once we have the answers.

Your post title mentions "timeout", is there a "timeout" in some other error message, or is "Network unreachable" (without anything saying "Secondary validation") the only error you get?

Yes. There is nothing about "Secondary validation". I had read Seth's
post
(Unexpected renewal failures since April 2024? Please read this!)
and was looking for any secondary validation related things but found
none. I think I am just not getting that far in the validation dance.

Do you get the same error when trying in the staging environment (adding --dry-run to the certbot command?)

Yes.

You say that it stopped in April, but looking at the crt.sh history you linked to I think you had some problems even before that. Certbot by default (I'm assuming you haven't changed this) will start trying to renew 30 days before expiration, running twice a day to check whether renewal needs to happen. So if your last renewal was Jan. 10, then certbot will have been trying since Mar. 10. But your certificate before that was issued Aug. 23, so certbot would have been trying to renew it starting on Oct. 22, but renewal didn't actually happen until that Jan. 10 certificate. And similarly there was a gap with no certificate between Aug. 2 and Aug. 23. Have you been needing to do manual interventions to get certificates these other times, or do you do something other than the default automatic certbot schedule?

Yeah, we needed to do stuff manually, correct. So far, we never looked
closer at that and debugged why the automatic renewal did not work...
(Doing the manual renewal every three months was okay-ish)

There was a small change in how Let's Encrypt's validators were set up in July 2023. I don't really think it's related, but I'm going to tag @jcjones just in case he wants to look into it.

Have you tried any other Certificate Authorities? I think you could use certbot renew --dry-run --server https://api.test4.buypass.no/acme/directory to try BuyPass Go's staging environment. (Or do you need to set up a new cert in certbot to switch CAs instead of just doing a renewal?) There are several free CAs that support ACME, though Buypass Go is the only one I know of that both has a staging server and doesn't require any separate account to be set up. And I don't know how amenable other CAs are to issuing for the Tor project.

No, renewal is fine and what we actually want. I did not try any because
I actually was not sure which to pick. So, I just did the dry-run
command you gave above:

Failed to renew certificate relay-02.torproject.net with error:
HTTPSConnectionPool(host='api.test4.buypass.no', port=443): Max retries
exceeded with url: /acme/directory (Caused by
ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at
0x7f4df4ada310>, 'Connection to api.test4.buypass.no timed out. (connect
timeout=45)'))

So, it seems the problem is more widespread than just Let's Encrypt
related. That's annoying but a "good" result, though. :slight_smile: Thanks for your
help.

3 Likes