Clarification on renewal and rate limits wrt volatile work/config dirs

[cstrahan]

I've read through Rate Limits - Let's Encrypt, which suggests that I should be able safely renew any existing certs. However, I suspected that the docs (as they often seem to) assumes that you have a persistent/non-volatile directory set via --work-dir/--config-dir. However, I wish to automate Let's Encrypt from ephemeral environments (such as AWS Lambda), where --work-dir/--config-dir will point at a temporary, short-lived directory, and I won't be persisting those directories in any way. The question now is, how should the docs on rate limits (and by extension, accounts) be interpreted?

(The core of my uncertainty is emboldened below, though the other questions should help provide some context and insight into my thought process here.)

Starting with this section:

You can create a maximum of 10 Accounts per IP Address per 3 hours. You can create a maximum of 500 Accounts per IP Range within an IPv6 /48 per 3 hours.

So first order of business is to determine if a non-persistent config dir leads to duplicate account creation. Being fairly new to Let's Encrypt, I first had to read up on what constitutes an account (Finding Account IDs - Let's Encrypt):

If you’re using Certbot, you can find your account ID by looking at the “uri” field in /etc/letsencrypt/accounts/acme-v01.api.letsencrypt.org/directory/*/regr.json.

A simple test would (seemingly) be to run certbot, then rm -rf the config dir, and then run certbot again and check if I get different uris in the **/regr.json file. Sure enough, I arrive at two different uris. Am I correct in my understanding that this will quickly lead to me hitting this particular rate limit (I'd test it myself, but I'd rather avoid putting unnecessary stress on the Let's Encrypt servers)? If yes, is there any way to avoid this without copying the entire work dir around (perhaps some docs on persisting just this bit, if necessary)?

The next concern I had was about renewals: if the config dir isn't persisted, and I use certbot --keep-until-expiring [...], will the following hold:

1. Will certbot be able to infer (say, from inspecting the target domain's current TLS certificate -- as reported by the server, as opposed to checking on-disk -- and perhaps any info stored in Let's Encrypt's servers) that my certificate is (or is not) about to expire, and (perhaps) additionally check that the cert was issued by Let's Encrypt, and thus avoid issuing another certificate? Empirically, this does not seem to be the case (I am issued a second cert).
2. If another certificate is issued, does that then count as a duplicate, or a renewal? I suspect the former, but I'm not entirely sure.

I can understand #1, as it would be tricky to define the "correct" semantics for such inference such that everyone would be pleased, e.g. if the server is configured with a cert from a different CA and that cert is far from expiration, is the "right" thing to do:

1. Wait for the current cert to expire, or
2. Authorize and install a new cert from Let's Encrypt immediately

I'm sure many people would want to migrate immediately, while others might want to hold off (for whatever reason), so inspecting some non-ephemeral config directory makes a bunch of sense as a sane way to handle renewal. However, I'd recommend clarifying the language in the docs so that they describe how things work in an ephemeral environment (admittedly, that's more of a certbot thing than a general Let's Encrypt thing; regardless, that info should be easily discoverable somewhere). For my own purposes, this can be easily worked around by doing my own check on the expiry of the cert before invoking certbot.

I should probably note that I've currently tested with the certbot-s3front plugin; given my reasoning, I doubt that should impact the observed behavior in such a way that it deviates from the core semantics, but I could be wrong: if any of my empirical observations sound off, please let me know.

In conclusion, I'd like to set up a recurring AWS Lambda function to keep my certs up to date, but I'm not convinced that I can do that in such a way that I won't likely run up against rate limits that are uniquely imposed upon such ephemeral environments.

[Patches]

@cstrahan

So first order of business is to determine if a non-persistent config dir leads to duplicate account creation.

Yes. But are you really running your lambda function more than 10 times every three hours? (Except for testing, and there's the staging server [certbot --dry-run] for that.) What makes you think you would hit this rate limit?

@cstrahan

Will certbot be able to infer (say, from inspecting the target domain’s current TLS certificate – as reported by the server, as opposed to checking on-disk – and perhaps any info stored in Let’s Encrypt’s servers) that my certificate is (or is not) about to expire, and (perhaps) additionally check that the cert was issued by Let’s Encrypt, and thus avoid issuing another certificate?

No. Certbot examines the certificate files on disk to determine renewal. You won't even be able to use the renewal function since your domain names and validation configuration won't be persisted. (How is certbot to know whether to use dns-01, a webroot, or the Apache plugin, or even which domain names to renew, without persisting that somehow?)

Every time you run certbot it will be like issuing a new certificate as far as it is concerned. So yes, you are correct that you will have to track whether renewal is required on your own.

@cstrahan

If another certificate is issued, does that then count as a duplicate, or a renewal?

Both. It would be counted as a duplicate certificate towards the five certificates per identical domain set per week rate limit, and as a exempted renewal towards the 20 certificates per registered domain per week rate limit.

@cstrahan

However, I’d recommend clarifying the language in the docs so that they describe how things work in an ephemeral environment

The docs go into great detail about what certbot persists where (certificates in /etc/letsencrypt/{live,archive}, validation configuration in /etc/letsencrypt/renewal/your domain.conf, etc.) If you don't persist any of that then you can assume that every time you run certbot it's like you just installed it? I'm not sure how the certbot documentation can make this more clear. Do you have any concrete suggestions?

The only thing I can think of is maybe explaining exactly how certbot figures out what and when to renew in the renewing certificates section, as this has been a point of confusion before.

[Patches]

Stepping back from all your other questions...

@cstrahan

I’d like to set up a recurring AWS Lambda function to keep my certs up to date

I don't think you would really want to use certbot if you aren't interested in using its persistence mechanisms, as they enable most of its biggest benefits.

One option to consider would be using one of the client libraries for a language Lambda supports. This would be much easier than calling out to certbot for all that you would use certbot for, with the sole exception of DNS verification.

Because you aren't running it on the server where you can easily serve a file to complete validation, I assume you are planning to use DNS verification to validate ownership of the domains listed on your certificates. This is the only thing where a client like certbot would come in handy, so you don't have to deal with adding a TXT record to Route 53, etc. by yourself. (Though that isn't so terribly difficult.)

But in this case, a more lightweight client like acme.sh might be easier to implement and understand than a more heavyweight one like certbot. (And it currently has better support for DNS verification anyway.)

But no client that I'm aware of implements renewal checking by making HTTPS connection to the server. So if you really want to do it that way, you would have to go about that yourself. Or, a less brittle way to check certificate expiry without persisting anything on your side would be to query crt.sh, (which unfortunately isn't well-documented,
but possible).

@cstrahan

I’m not convinced that I can do that in such a way that I won’t likely run up against rate limits that are uniquely imposed upon such ephemeral environments.

If you implement renewal correctly you shouldn't have to worry about the domain rate limits any more than you would in a classical configuration.

The only way I can think of the account rate limit being a problem for you is if you have to get more than 10 certificates, and you issue them all in separate lambda functions, and the functions really end up making requests from the same IP address all 10 times. I'm not that familiar with how Lambda works, but I believe you would actually rotate through different backing EC2 instances, rendering this issue moot.

But if you're really concerned about that possibility you could batch all your certificate requests in one lambda function that uses the same account key for the duration. Or else just persist the account key, which might be easier for you when using a client library from Lambda vs shelling out to certbot.

[cstrahan]

Thanks for the reply!

Yes. But are you really running your lambda function more than 10 times every three hours? (Except for testing, and there’s the staging server [certbot --dry-run] for that.) What makes you think you would hit this rate limit?

I have a bunch of projects, so I'll definitely want to renew 10 (or more) on some set schedule. Lambda can reuse the same instance, which would mean that I could hit that limit (and there's no telling who else has created an account from that IP before me). It may be "unlikely", but given that I can't quantify that probability (is it 1 in 1000? 1 in 1M? who knows), and given that I don't like to leave things to chance when I can help it, I'd much rather put that last little bit of effort into making things almost fool proof (excepting cataclysmic events).

It looks like other people have thought of this before me, though, so I can probably just follow their example. For example, there a couple projects (1, 2) that store the key in S3 upon first account creation, and then reuse that key thereafter; I'll probably just do something similar.

Emphasis mine:

No. Certbot examines the certificate files on disk to determine renewal. You won’t even be able to use the renewal function since your domain names and validation configuration won’t be persisted. (How is certbot to know whether to use dns-01, a webroot, or the Apache plugin, or even which domain names to renew, without persisting that somehow?)

That objection makes perfect sense, once one knows that "renew" in certbot's vernacular means "automatically iterating over all previously obtained certs, performing the same process/settings (domain, challenge type, etc) that were originally used to obtain each -- going so far as to install a cron job to do all of the above on a regular basis". I had the more general notion of "renewal" in mind, which basically amounts to "explicitly re-invoke the same command I used to procure a cert the first time around" (so to answer your question directly: it wouldn't know -- I'd explicitly tell it what to do).

My concern was due to an inference on my part: "if certbot has a 'renew' subcommand and various flags regarding renewal (e.g. --keep-until-expiring), then surely that implies that there's some difference in what happens protocol wise (with potential implications regarding state in the config/work dir), so I probably need to ensure I do whatever magic tells Boulder that I'm specifically renewing (despite that being weird / seemingly unecessary, as that info should already be available to it, because surely it's persisting a record of which domains it has issued certs for), right? Otherwise, why else would there be a separate renew subcommand and extra flags, if what's going over the wire is the same?"

Now I see that was probably a bad inference; I dove into certbot without a clear understanding of what it is: a very high level, user friendly, automated way of managing certificates obtained via ACME. My mistake was thinking it was much like other low level tools I use daily, where it's generally safe to assume that each flag and subcommand probably maps 1-to-1 with the core mechanisms/protocols involved (there's little abstraction over the kernel interface in, say, mknod). Now, assuming I understand correctly, I see that there isn't any significant difference in what happens protocol wise when 'renew' is used -- the only difference is in the automation.

Ironically, I think the combination of my general distrust in software doing the intuitive/sane thing and the fact that certbot has succeeded in being so user friendly resulted in me being more confused than warranted .

The docs go into great detail about what certbot persists where (certificates in /etc/letsencrypt/{live,archive}, validation configuration in /etc/letsencrypt/renewal/your domain.conf, etc.) If you don’t persist any of that then you can assume that every time you run certbot it’s like you just installed it? I’m not sure how the certbot documentation can make this more clear. Do you have any concrete suggestions?

I think we were talking past each other because of my misconception of what all "renew" entailed, as described above. At the moment, I don't think there's anything that really needs changing.

If another certificate is issued, does that then count as a duplicate, or a renewal?

Both. It would be counted as a duplicate certificate towards the five certificates per identical domain set per week rate limit, and as a exempted renewal towards the 20 certificates per registered domain per week rate limit.

Gotcha, makes sense.

I don’t think you would really want to use certbot if you aren’t interested in using its persistence mechanisms, as they enable most of its biggest benefits.

Yep, I think that's what I'm now realizing.

One option to consider would be using one of the client libraries for a language Lambda supports. This would be much easier than calling out to certbot for all that you would use certbot for, with the sole exception of DNS verification.

Right -- I might use an existing project, or I might roll my own. I wanted to avoid using less battle-tested software, and I also wanted to avoid writing my own software (not my idea of a fun way to spend my weekend), but either way, I think certbot is the wrong tool for this application.

Because you aren’t running it on the server where you can easily serve a file to complete validation, I assume you are planning to use DNS verification to validate ownership of the domains listed on your certificates. This is the only thing where a client like certbot would come in handy, so you don’t have to deal with adding a TXT record to Route 53, etc. by yourself. (Though that isn’t so terribly difficult.)

In my initial use case of Let's Encrypt -- getting TLS set up on my blog -- I'm using S3 to host all of my content, so it's actually pretty straightforward to plunk the challenge file in the right spot and use http-01 (that's what certbot-s3front does for you, which has worked well enough to get my initial cert issued).

But in this case, a more lightweight client like acme.sh might be easier to implement and understand than a more heavyweight one like certbot. (And it currently has better support for DNS verification anyway.)

That might come in handy -- thanks!

My own confusion aside, Let's Encrypt has been great; thanks to you and everyone else who has contributed to Let's Encrypt, certbot, et al.

Sorry for the confusion. I probably could have avoided the need for inquiry if I had read the ACME spec (which would have assuaged by concerns regarding renewal), but at that point yesterday, I had already burned 12 continuous hours going from nothing, to Caddy in front of my wiki, packaging certbot-s3front for my OS (NixOS), using certbot-s3front to obtain the cert for my blog (hosted on S3), configuring CloudFront for the first time, and tweaking policies/configuration across S3 and CloudFront to make redirection from cstrahan.com to www.cstrahan.com and HTTP to HTTPS all work selectively (excepting requests to /.well-known/acme-challenge/*, so that the http-01 challenge would work). At that point I really, really didn't want to be knee deep in RFCs if I could help it.

[Patches]

@cstrahan

Now, assuming I understand correctly, I see that there isn’t any significant difference in what happens protocol wise when ‘renew’ is used – the only difference is in the automation.

Yes, exactly.

@cstrahan

surely that implies that there’s some difference in what happens protocol wise (with potential implications regarding state in the config/work dir), so I probably need to ensure I do whatever magic tells Boulder that I’m specifically renewing (despite that being weird / seemingly unecessary, as that info should already be available to it, because surely it’s persisting a record of which domains it has issued certs for), right?

This is actually a security decision. A number of misissuance incidents experienced by other CAs are due to some sort of magic in the renewal process causing some important detail to be forgotten. By making them exactly the same operation, Let's Encrypt and other ACME CA implementations can avoid an entire class of bugs!

If there were an ACME renewal operation that used information persisted with the CA, it would almost certainly require a message signed by the previous account key, or else anyone could just get a new certificate for your domain!

So I think the reason I thought you mostly understood that renewal wasn't it's own operation and didn't go into that more is that with your earlier questions you seemed to already understand that the account key wasn't super important to keep around.

@cstrahan

excepting requests to /.well-known/acme-challenge/*, so that the http-01 challenge would work

This isn't strictly required as long as the redirect lands at the right validation file. The validation server will follow any redirect to HTTP or HTTPS and accept any certificate, even self-signed or expired. You could even use redirects to concentrate validation into one S3 bucket if you wanted to.