Verification Timeouts

Help
1

Hey there,
getting lot’s of Verification Timeouts when Renewing Certs today.
Are there Problems on LE side? Takes really long to switch from “status: pending” to “status: valid” …
Can’t spot any Probs on our Network … ?
thx, bye from Austria
Andreas

ps) Is there a best practice on how often the Client should check Status before ending with Timeout? I can see that it’s currently 5 times with 1 sec delay in between - should this be higher? did something change that it takes longer than a few weeks before?
pps) now also tested with 5 times with 5 seconds delay - still timeouts :-/

1 Like
2

What verification type are you using?

Anyhow, it’s not supposed to be a problem… nor is supposed to be done interactively.

Wait, are you developing your own client?

1 Like
3

we are using our own Client (based on https://github.com/analogic/lescript) since years, without any problems … it’s deeply integrated in our SAAS CMS … so yes - own client! :wink:

Using http-01 Challenge

1 Like
4

Hi @futureweb

I use 5 seconds delay. 1 second is a little bit short.

The multi perspective validation requires some time. DNS + domain check.

1 Like
5

Hi Jürgen,
also tried 5 times with 5 seconds right now - still timeouts … :-/
Never experienced this since a few weeks ago … now also urn:ietf:params:acme:error:rateLimited Errors start to appear - I guess because of “forgotten” timed our requests?!? (just a guess?)

7

you should probably try exponential backoff. first you try after 1 second, then 5, then 25, then 2 minutes, then 10 minutes, one hour... and you should not stop before either success or a week passes.

It looks like this: Rate Limits - Let's Encrypt

You can have a maximum of 300 Pending Authorizations on your account. Hitting this rate limit is rare, and happens most often when developing ACME clients. It usually means that your client is creating authorizations and not fulfilling them. Please utilize our staging environment if you’re developing an ACME client.

For users of the ACME v2 API you can create a maximum of 300 New Orders per account per 3 hours. A new order is created each time you request a certificate from the Boulder CA, meaning that one new order is produced in each certificate request.

8

I think that's a wrong written client.

And what's a "verification timeout"?

This

isn't a timeout, that's only a slow validation.

If your client cancels the status check, you have a pending order -> max. 300 pending orders, next problem.

If connecting Letsencrypt has a timeout, wait. Then creating a new order is the wrong way.

1 Like
9

darn … seems like the Client we choose a few years ago got some issues with slow validation and/or timeouts to LE … will try to reach out to the coder of https://github.com/analogic/lescript/ + will dig myself in the code if this behaviour can be enhanced …

It seems like this Client tries 5 times with 1 second delay between (also tried to set it higher 5sec, 10sec, …) and then throws an Error …

1 Like
10

there are other php clients, if you need them:

1 Like
11

yea thx - try to avoid replacing the whole Client as it’s quite deeply integrated into our CMS … serving thousands of Domains with LE Certs … sigh
If possible I would prefer to fix the used Client … but as I’m not really deep into ACME Prot. I guess it means some reading for me … :grimacing:

1 Like
12

Do I understand correct that a signed Request to https://acme-v02.api.letsencrypt.org/acme/authz-v3/authorizationsID before the Script is throwing Error should clean out the Pending Auth regardless it’s Status? :thinking:
No special Payload within the Request needed?

13

Here is a helper tool: https://tools.letsdebug.net/clear-authz

1 Like
14

yea - thx - already stumbled over this one - unfortunately there is no Source Code of it … as I would need to Re-Code the Cancel Logic within the PHP Client I’m using … sigh

15

you don’t. you can use multiple clients with the same key.

1 Like
16

I don’t know how that client works. Never used ACME-v1.

But:

Order - select a challenge -> pending -> challenge created -> say “Please check that challenge” -> then wait. Throwing an exception after 5 seconds is wrong. So add a loop.

1 Like
17

using ACME-v2, not v1
There is a loop checking if verification is pending / valid - after 5 rounds it's throwing Error - so I guess that's where I need to clean the Pending Auth.
According to https://letsencrypt.org/docs/rate-limits/:

If you have a large number of pending authorization objects and are getting a rate limiting error, you can trigger a validation attempt for those authorization objects by submitting a JWS-signed POST to one of its challenges, as described in the ACME spec. The pending authorization objects are represented by URLs of the form https://acme-v02.api.letsencrypt.org/acme/authz/XYZ , and should show up in your client logs. Note that it doesn’t matter whether validation succeeds or fails. Either will take the authorization out of ‘pending’ state.

So I guess it should solve the Problem when doing this before throwing the Error? In addition to modifying the Loop so it uses exponential delay ... 1s, 5s, 25s, ...

Just want to be sure I don't do anything "wrong" here when modifying the Client ... :wink:

1 Like
18

Then you have already the loop. Play there - add more checks and more time.

And may be a better log.

PS: I would add something like

1, 5, 6, 7, 8, ...

or

1, 5, 10, 20, 30, 30, 30

2 Likes
19

yea - just modifying the loop to give the Verification Process more time …
But if it finally is taking too long, several minutes I want to clear out the Pending Auth - The Client will then try it again within the next few days. (When Validation is hopefully a bit faster … :wink: )

So sending an empty, signed Post to https://acme-v02.api.letsencrypt.org/acme/authz-v3/AuthID should do it’s work - am I correct here?

20

Info: With increasing SleepTime between the checks all Certs I’ve tested have been successfully verified by now …
But just in case if it will take even longer for some Certs I want to implement the “clean Pending Auth” too …

21

In no case they will take more than a week (before failing, eventually). Let them run, restarting them will not make them go faster.