Tool for Clearing Pending Authz - Looks at Certbot Log Files and Uses ACME Library

[ahaw021]

Hi All

There have been several post about pending authorizations and having to wait 7 days.

Part of the challenge is that this is done without user being aware as to the root cause and currently there are no ways of fixing the problems

Could this have been a github issue - yes, but I think there are lots of learnings which might be beneficial

I am hoping to present the problem space in a way that can be beneficial to all clients (not just certbot)

A) Research
B) Inducing Fail Conditions
C) Learnings from Fail Conditions
D) Building a Challenge List
E) Deactivating Challenges
F) Integrating with Certbot

Andrei

[ahaw021]

Research

So lets try to figure this problem out

https://letsencrypt.org/docs/rate-limits/

The specification is quite obvious however one needs to figure out but how do all these things tie together

A) Authz Limits are tied to an account
B) A really good attempt: Tool for clearing / invalidating pending authz
C) Some documenation https://letsencrypt.org/docs/rate-limits/

Challenges for me:

A) Where are these authz been generated
B) Who keeps the score

So in order to figure these things out. Let’s induce some fail conditions aka throw some spanners in the works (and i have to confess this is the best part of the job )

Andrei

[ahaw021]

Inducing Fail Conditions

I wanted to investigate a couple of things

A) If we give certbot a couple of commands with the same domain is it smart enough to not ask for new authz
B) Where in the chain do authz come from and how do we break this

Enough theory lets do it

certbot certonly --manual --staging -d authz1.firecube.xyz -d authz2.firecube.xyz -d authz3.firecube.xyz

Lets run it twice but not let certbot submit the challenges

A) The first set of challenges are ignored. Certbot submits the authz requests however after we break (crl c) after the submissions the challenges are in pending

C) on the second run certbot does not pick up the fact we asked for the same domains previously and creates new challenges

We now have two log files with 6 challenges in totals

Next Steps:

5 domains with the command below

certbot certonly --manual --staging -d authz-set30-1.firecube.xyz -d authz-set30-2.firecube.xyz -d authz-set30-3.firecube.xyz -d authz-set30-4.firecube.xyz -d authz-set30-5.firecube.xyz

first run we say no to IPs being logged

Authz are still generated despite us saying no

second run we accept logging of IP but run through the 5 challenges without submitting a challenge

Overall:

We have 16 pending challenges.
We have 4 log files

@jsha @cpu - am I correct in understaning the currently boulder has no way of ansering a Authz Query - for example - given an account with ID X please give me all pending autz

@bmw @schoen - is there anything in the works for certbot to solve this (don't want to spend too much time if you are working on solution)

Andrei

[ahaw021]

Learning from Fail Conditions

The main thing that fail conditions teach me is this

A) Not everything goes to plan
B) When it doesn’t most client ask for new authz instead of reusing exisitng ones
C) Clients don’t have a way of keeping registered authz
D) my preference would be on fail conditions for client to submit a challenge anyway so the fialure clears the pending authz

All sorts of other factors make me think this is a problem so lets go on and solve it

Andrei

[ahaw021]

Building a List of Challenges and Authz

This is currently done via a python script. Most of the libraries are standard libraries and this has been tested with Python 2.7 and 3.6

ahaw021/LE_FIND_PENDING_AUTHZ/blob/master/LE_FIND_PENDING_AUTHZ.py

The script should be run in the logs directory (for now) and output should be something like the below

@ericfg @Jeromnimo - do you mind testing?

you should run python LE_FIND_PENDING_AUTZ.py and it should produce a similar output

The next phase is how to submit the challenges to clear the authentications

Andrei

[ahaw021]

Deactivating Challenges

In order to deactivate challenges you will need to specify the path to the private key (usually stored in JSON format)

The code will then use the inbuilt ACME library to deactivate the pending authorisations

I have tried to make the code not invalidate things that have already failed.

The HTTP challenge is used to invalidate the authz (it was the first one)

You can then run it again and see that as all pending authz are now invalidated the tool is smart enough not to try invalidating again

Note: the tool takes a 1 second sleep between each invalidation (so as not to throw lots of requests at a production server)

At 300 pending requests (which is what the current limit is) this will take about 5 minutes. So start it and go get a coffe

Andrei

[cpu]

@ahaw021

@jsha @cpu - am I correct in understaning the currently boulder has no way of ansering a Authz Query - for example - given an account with ID X please give me all pending autz

That's correct. There's no ACME support for this.