How to clear pending authz requests, logs are gone

Help
1

Hello everyone,

I’m creating some infrastructure via terraform and for certs and dns using letsencrypt & cloudflare and that works really good.

Before I started using terraform with acme, I’ve used some ansible playbook and the vms which requested the cert are gone/deleted.

Is there a way to clear the auth in pending if I don’t have the logs anymore?
Is there some “timeout” / cooldown period for those in pending?

br

2 Likes
2

Welcome to the Let’s Encrypt Community :slightly_smiling_face:

You pose a great question.

You can have a maximum of 300 Pending Authorizations on your account. Hitting this rate limit is rare, and happens most often when developing ACME clients. It usually means that your client is creating authorizations and not fulfilling them. Please utilize our staging environment if you’re developing an ACME client. Exceeding the Pending Authorizations limit is reported with the error message too many currently pending authorizations .

I feel like there are three steps to the process you are requesting, given that you know your account url (or JWK), which you must if you’re asking the question that you are:

  1. Query for unfulfilled (not yet valid / no cert link) orders. (i.e. sift all orders for an account)
  2. Query for pending authorizations for each unfulfilled order. (i.e. sift authorizations for pending)
  3. Authorize a challenge for each pending authorization to get it to change state.

The first step I am searching for. The second step follows easily from the first. The third step is straightforwardly documented below. Accomplishing the first step might avoid the bold part below.

If you happen to know the combinations of domain names you were trying to certify, but never completed, you could submit “new” orders for them, which would result in “hooking into” your pending orders. You could then let them fail (or succeed), which would clear the pending authorizations.

Clearing Pending Authorizations

If you have a large number of pending authorization objects and are getting a Pending Authorizations rate limiting error, you can trigger a validation attempt for those authorization objects by submitting a JWS-signed POST to one of its challenges, as described in the ACME spec. The pending authorization objects are represented by URLs of the form https://acme-v02.api.letsencrypt.org/acme/authz/XYZ, and should show up in your client logs. Note that it doesn’t matter whether validation succeeds or fails. Either will take the authorization out of ‘pending’ state. If you do not have logs containing the relevant authorization URLs, you need to wait for the rate limit to expire. As described above, there is a sliding window, so this may take less than a week depending on your pattern of issuance.

Note that having a large number of pending authorizations is generally the result of a buggy client. If you’re hitting this rate limit frequently you should double-check your client code.

1 Like
3

The orders field on account objects is not (yet) implemented in the Let’s Encrypt ACME server (boulder):


That is needed for the point number 1. on your list.
2 Likes
4

Ah. :slightly_smiling_face:

Thanks for the assist. In implementing my Open ACME client I did notice that submitting a “new order” based on the same certificate signing request (same domains) resulted in returning the same order url if the order was not yet fulfilled. This would be one way of (blindly) accessing existing orders to trigger their challenges and thus clear the pending authorizations. I believe.

1 Like
5

Hello @griffin

Thank you for your detailed reply.

I’m hitting actually the “Duplicate Certificates” limit.

From what I see the next certificate can be issued at the following date -> “The next time this certificate can be issued is 11 Sep 2020 16:56:43 UTC”

Does this mean 1 new certificate will be available at that date or that limit itself will be lifted ?

Thanks!

br

1 Like
6

Welcome back br :slightly_smiling_face:

It’s a rolling limit, meaning that you can only get 5 certificates with the exact same set of (sub)domains in a any given 7-day window. Might I ask why you would even need 5 duplicate certificates? The only functional difference between them usually is their private keys. Normally the only good reasons for creating a duplicate certificate are:

  • Loss of the certificate due to software or hardware change
  • Compromise of the private key for the original certificate

The typical reason that most people hit this limit though is in debugging their configurations. To remedy this, we recommend the staging environment:

PS: Once you are sure your configuration is alright (hopefully using the staging environment), I can explain how to get another certificate if you need to. You should be able to use the last one you acquired though. I have already given you a big hint in this post for getting your next certificate, but hopefully you should not need to do so until it’s time to renew. :wink:

7

The ratelimit is 300 pending authorizations per week (rolling 7 day window) per account.

In your situation, the best option is to do the following:

  1. Change your system to use the staging environment. You should have been using this since the start.
  2. Create a new account, which will be unaffected by your existing account's throttle.

That ratelimit is applied across accounts per week (rolling 7 day window). The typical way to get around that is to add a new subdomain to the certificate.

I am noting these workarounds as quick emergency fixes. Do not rely on them, seek to avoid using them.

The documentation is a bit dated and opinionated; I should file a ticket over it. At one point, these issues were mostly happening because of people developing new clients. Most recently, ratelimiting like yours is being encountered by people who are setting up automated systems via tools like Docker, AWS, Ansible, and potentially Terraform now. Many people don't read the fine print, or don't understand the statelessness of their systems. Every time they spin up a node/service/whatever, their systems try to get a new certificate. Don't do that.

Work on getting your system up and running against the staging API's endpoints and figure out a way to ensure your obtained certificates - and logs - can persist and operate within the limits.

I hope this helps!

2 Likes
8

Thank you again @griffin for the fast and detailed reply.

This limit was hit either because of the often “terraform apply” & “terraform destroy” I’ve done at various points in time in order to test my infrastructure deployments or because of the of an ansible playbook that used certbot to request those certificates and failed, thus resulting into requests stuck into pending state.

Those VMs got destroyed by “terraform destroy” and I’ve never managed to clear those pending requests.

I will look at the resources you sent and switch to staging until I get the rate limit cleared.

@jvanasco thank you for the reply.
I agree that not reading the fine print caused this more or less and also the way terraform/ansible/automation is thinking about the spin up of the whole infrastructure.

The automation is more or less not matching very will with this limit and the way all these tools are thinking about resources and the way letsencrypt is seeing them.

I’ll update/change my strategy regarding certificate requesting/generation and it should be OK.

Thank you all for the support!

br

1 Like
9

You could create a new ACME account private key and register another account. :slightly_smiling_face:

10

Most popular models for scalable systems do not have the nodes communicate with ACME directly. Designs such as:

  • Certs are provisioned from an internal API/Server, which returns a certificate from persistent storage or failsover to generating a new one
  • A single node obtains the certs, and either:
    • Certs are loaded off a shared storage block
    • The server rsyncs/etc the certs into other nodes

These are just some of the options. Many more are discussed in this community and other online forums.

I personally prefer the centralized server/api model.

1 Like
11

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.