Registration Rate Limit in Staging Using Python ACME Client


#1

I’m trying to use the Python ACME client in the staging environment for testing (using https://acme-staging.api.letsencrypt.org/directory). However, I’m getting this error:

acme.messages.Error: urn:acme:error:rateLimited :: There were too many requests of a given type :: Error creating new registration :: too many registrations for this IP: see https://letsencrypt.org/docs/rate-limits/

It’s happening when I call acme.client.Client(…).register(). Do I have to do something different with this for staging? Should I be saving the registration from a previous run? If so, how is that that done? I didn’t see any documentation of how to do that, and the example (https://github.com/certbot/certbot/blob/master/acme/examples/example_client.py) does not save the registration.


#2

Yes.

By saving the key you generate and the regr returned from register, and not calling register more than once.

You can see how Certbot constructs the client here:

and earlier in the call stack:


#3

When performing a request, I need the return value of register() (i.e. the regr variable in the code shown below).

client = acme.client.Client(DIRECTORY_URL, key)
regr = client.register()

It looks like the code you referenced has regr stored in self.account.regr, which in turn comes from the account_ constructor argument. It sounds like what I need to do is store regr in a file for use on future runs. Is that correct? How do I serialize regr to a file?


#4

Yes, as well as the private key, which is separate. I believe the regr includes the public component of the key, but this isn’t sufficient to be able to re-use on further runs.

I believe you can serialize it to a string with

regr.json_dumps()

and then persist it however you want.

Edit: and the reverse

regr = messages.RegistrationResource.json_loads(...)

Edit 2: to_json -> json_dumps, sorry.


#5

I tried the approach shown below and it looks okay. The JSON saved looks similar to that produced by Certbot. I’m missing one item. How do I deserialize json_pkey in the later run?

Initial run:

… create jwk_pkey …
client = acme.client.Client(DIRECTORY_URL, jwk_pkey)
regr = client.register()
json_pkey = jwk_pkey.json_dumps()
json_regr = regr.json_dumps()
… save json_pkey and json_regr somewhere …
client.agree_to_tos(regr)

Later run:

… retrieve json_pkey and json_regr from save location …
regr = acme.messages.RegistrationResource.json_loads(json_regr)
jwk_pkey = ??? how to deserialize json_pkey ???
client = acme.client.Client(DIRECTORY_URL, jwk_pkey)
client.agree_to_tos(regr)


#6

I also think you’ll need to initialize the client on subsequent runs as per acme_from_config_key from Certbot, otherwise the regr won’t be bound to the ACME client …


#7

So in other words, don’t use client = acme.client.Client(DIRECTORY_URL, jwk_pkey) on subsequent runs, but instead use:

net = acme.client.ClientNetwork(jwk_pkey,regr)
client = acme.client.BackwardsCompatibleClientV2(net,jwk_pkey,server)

What does the server argument represent? Is that just DIRECTORY_URL? I tried using that, and got an attribute error for agree_to_tos. I tried moving client.agree_to_tos(regr) to before saving the JSON data. I then got another attribute error (shown below).

authzr = client.request_challenges( identifier = acme.messages.Identifier(typ=acme.messages.IDENTIFIER_FQDN, value=domain) )
File “/usr/lib/python2.7/site-packages/acme/client.py”, line 731, in getattr
raise AttributeError()
AttributeError

The attribute name that’s causing the error is request_challenges.


#8

Here’s a little client that works standalone, maybe it’ll help through these interface problems.

from acme import client
from acme import messages

import os.path

import josepy as jose
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives.asymmetric import rsa

key = None
regr = None
acme_client = None

def make_client(key, account=None, directory="https://acme-staging-v02.api.letsencrypt.org/directory"):
  return client.BackwardsCompatibleClientV2(client.ClientNetwork(key, account=account), key, directory)

# Need to generate a private key
if not os.path.isfile("private_key.json"):
  print "Will generate private key ..."
  key = jose.JWKRSA(key=rsa.generate_private_key(public_exponent=65537,key_size=2048,backend=default_backend()))
  with open("private_key.json", "w") as f:
    f.write(key.json_dumps())
# Otherwise already have a private key
else:
  with open("private_key.json") as f:
    key = jose.JWKRSA.json_loads(f.read())

# Need to register with ACME server
if not os.path.isfile("regr.json"):
  print "Will register account ..."
  acme_net = client.ClientNetwork(key)
  acme_client = make_client(key)
  # You can attach emails and whatever to the NewRegistration
  regr = acme_client.new_account_and_tos(messages.NewRegistration())
  with open("regr.json", "w") as f:
    f.write(regr.json_dumps())
else:
  with open("regr.json") as f:
    regr = messages.RegistrationResource.json_loads(f.read())

if not acme_client:
  acme_client = make_client(key, regr)

# Do whatever
with open("/etc/letsencrypt/csr/0000_csr-certbot.pem") as f:
  order = acme_client.new_order(f.read())
  print order

#9

Thanks. That’s helpful. The registration part appears to be working now.

Your last part (new_order()) is different from the example (https://github.com/certbot/certbot/blob/master/acme/examples/example_client.py), which uses:

authzr = client.request_challenges( identifier = acme.messages.Identifier(typ=acme.messages.IDENTIFIER_FQDN, value=domain) )
authzr, authzr_response = client.poll(authzr)
client.request_issuance(josepy.util.ComparableX509(csr), (authzr,))

The first line (request_challenges()) is where I was getting the attribute error.

What’s the correct way to request a certificate?


#10

Also, using request_challenges() at the bottom of your example client gave me the same attribute error.


#11

I’m quite sure that example script is an abandoned baby that nobody has updated for the current API.

At this point I’m reading how Certbot uses the library as much as you are, so I’m not sure I can be of much further help :laughing:.

There doesn’t seem to be a separate request_challenges in v2.

# Make a cert
with open("/etc/letsencrypt/csr/0000_csr-certbot.pem") as f:
  order = acme_client.new_order(f.read())
  print order.uri

  for authz in order.authorizations:
    print "{} {}".format(authz.body.identifier.value, authz.uri)

    # Choose one of the challenges, e.g. HTTP-01 or DNS-01
    for chal in authz.body.challenges:
      print chal.chall

    # Set your challenges with the calculated key-authz ...
    # .....

  # and then finalize the order (which will make the certificate available)
  print acme_client.finalize_order(order, datetime.datetime.now() + datetime.timedelta(seconds=30))

#12

If the example is abandoned and not current, then I’m not surprised it doesn’t work.

Thanks for the code suggestion. I worked with it for a while, but realized it’s getting too much into the protocol details than is necessary for situation. I really just want something I can call to get a certificate directly from a certificate signing request. Perhaps something like:

pem_certificate = get_certificate(pem_certificate_signing_request, webroot_path)

So far I haven’t been able to find a higher level library to accomplish this. At the moment I’m running certbot from python, and interacting with it by saving, reading, and deleting files from the file system. It’s not particularly elegant, but has the advantage of skipping the protocol details (which are still evolving). It would be nice if in the future, in addition to the command line interface, certbot also had a python interface with function arguments as inputs and return values as outputs.


#13

Hi @doc987,

I’ll file an issue to note that we should update the examples in the acme library.

Also, have you taken a look at the ACME client implementation list? There are several implementations in Python

which are generally considerably more minimalist than Certbot and might be easier for you to modify and/or use as working examples.


#14

Thanks. I had actually checked out all the Python on PHP options in the “ACME v2 Compatible Clients” list on that page (https://letsencrypt.org/docs/client-options/#acme-v2-compatible-clients), but didn’t find a better option. Below is a brief description of what I did find.

  1. LEClient PHP library: Can’t get certificate as a return value. Output always goes to the filesystem (getCertificate() in LEClient/src/LEOrder.php).
  2. le-acme2-php library: Not enough activity.
  3. Sewer: Nice that it has a command line and library interface, but only has DNS (no HTTP) validation.
  4. stonemax/acme2 PHP client: Can’t get certificate as a return value. Output always goes to the filesystem (getCertificateFile() in src/services/OrderService.php).
  5. ACME Tiny: Meant for command line use, and can’t do wild card certificates.
  6. itr-acme-client PHP library: Not enough activity.

#15

Have you looked at Pebble?

It’s an acme-v2 fake server. You can use it as the endpoint instead of staging for client development.

if i were making a v2 client, I would use pebble for development and unit/integrated testing.

i made a v1 client and had to piece together the necessary endpoints of a fake v1 server from scratch, using reverse engineering and the acme spec. it’s not fun. pebble gives you everything for v2.


#16

lol give me a break. Why should a library that is based on an IETF draft need “activity”? You implement the draft and go.

How about giving that library an honest go. It seems to tick all of your boxes. Nothing is going to do 100% of your job, especially if you want to insist on the lie that that ACME is a simple, stateless protocol and can be described with get_certificate(csr_pem, document_root).


#17

jvanasco: Thanks for the suggestion. However, I’m really trying to avoid making a client. My application generates a certificate signing request on it’s own. What I was looking for was a library to handle the ACME portion. In other words, accept a certificate signing request, and return a certificate. There a several command line programs that look like they handle this pretty well, and have a lot of features, but they seem to be designed for command line use only (no library interface). I can run the command line programs from my program (which I’m currently doing), it’s just not ideal (*sigh*).

_az: Thanks for all your help. However, in my case, activity is an EXTREMELY important item. I can’t base a program on software that has a good chance of being abandoned. If that does happen, I’m forced to either (A) change my program to use a different tool, or (B) take over maintenance of the software. In case B, I end up doing the very thing I was trying to avoid in the first place, i.e. handling the ACME protocol myself. I have no idea whether the particular project you mentioned will continue to be maintained or not, but when very small groups are involved, I believe abandonment is much more likely. Even though Certbot doesn’t provide the best mechanical fit, I think it’s much more likely to be maintained. This is one of the top considerations.

I did not state that the protocol was simple. In fact, I’m not prepared to delve into all its complexities. If I was, then I wouldn’t need to be looking for a library that already handles those complexities. The fictitious get_certificate() function I described was for illustration. It was not meant to be a one line description of ACME.


#18

February 2018, I had the same problem. There are a lot of libraries, but they handle the local part (generating + storing keys, request, certificates, change the binding of the webserver, errorhandling usw.) different. And it’s not clear how they handle these things.

So I used the samples in the documentation ( https://tools.ietf.org/html/draft-ietf-acme-acme-12 ) and wrote my own library. Most is standard, the only thing, which is a little bit tricky: The signature. But if this is done, it works.

If you can create a new account, then most is done.

If there is an error and a user doesn’t see the error, the cronjob replies the command the next day -> the same error, after a few days this hits the Letsencrypt-Limit.

Own code: If there is an error (sample: Creating the certificate request or sending it), then this step can be checked -> no new order. The code checks if there is an open order, so no new order for the same domain name can be created.

Reading this forum: A lot of errors are local. Then https://crt.sh/ shows: There are a lot of new certificates, but the binding doesn’t work. It isn’t an opinion to repeat errors.


#19

[quote=“doc987, post:17, topic:62479”]
Thanks for the suggestion. However, I’m really trying to avoid making a client. My application generates a certificate signing request on it’s own. What I was looking for was a library to handle the ACME portion. In other words, accept a certificate signing request, and return a certificate. There a several command line programs that look like they handle this pretty well, and have a lot of features, but they seem to be designed for command line use only (no library interface). I can run the command line programs from my program (which I’m currently doing), it’s just not ideal (sigh).[/quote]

Those are all things you can test locally, against Pebble, instead of staging.


#20

Oh, I see what you’re saying. Test against Pebble (instead of Let’s Encrypt) while building an application. That could be handy. No limits on test requests.