Certbot on AWS Lambda

Help
1

Hi,

I am running certbot on AWS lambda. We have to generate/renew approximate 5000 certificates per month for all our clients.

As per Lets-encrypt limits we can create only 10 account per IP address and 500 accounts per IP range in 3 hrs.

When we use AWS lambda, AWS will allocate one IP address to it. When one lambda function is invoked it issues one certificate and next certificate will get a fresh lambda function.

Suppose we want to issue 1000 certificates in 1hr on AWS lambda. Then we trigger 1000 lambda functions (Assume sequentially for simplicity). Suppose AWS allocates the same IP for all the 1000 functions. Since each invocation is fresh environment, certbot will create new account and it easily hit the account limit. Even if AWS allocate different IP but same IP range then also we will hit the limit easily.

How to solve this issue? Is it possible to backup the account details and restore the account details on each function invocation? if we backup and restore the account details then what id AWS allocates different IP address for each invocation? Is there any link between IP address and account?

If there is a link between IP address and account, then how lets encrypt knows the IP address? does it use publicly visible IP or the machine IP(Which can be a private network IP)

Thanks

2

Any help?

3

Please be more patient. It only has been 4 hours since you've opened your thread and your question is rather specific. I for one have no clue what AWS Lambda even is, so I cannot help you. In fact, there is a chance noone here has experience with AWS Lambda, although I hope someone can help you.

4 Likes
4

I might take a crack at it later but I was hoping @petercooperjr would. I only know Lambda from reading - not actual use. And, large volumes are a design issue. A bit complicated

Peter has experience so here's hoping ...

@sridhard If you have not yet, you should read at least this topic

4 Likes
5

Yes, you can store the account credentials somewhere (eg, AWS Parameter Store, Secrets Manager, etc). Then you will have just one account. We can grant your account a higher rate limit for your use-case. You must use a single account if you want rate limit exceptions, as those apply to the account.

If you are writing custom Lambda code, you may want to consider using a library for ACME in whatever language you're writing instead of relying on certbot. It may give you more control over how you're executing.

You can test against the Staging API which has much higher rate limits while you contact us to get a production rate limit override. See this page for information on how to do that: Rate Limits - Let's Encrypt

5 Likes
6

Agree with that and your other comments. And it is often nice to externalize such things.

I assume they would use a custom Lambda image. If so, they could just place a "starter kit" for acme in that image which could include the account id. Or, use other more generic persistent stores like EFS of course.

3 Likes
7

Well, gee, I guess I've been summoned. Yes, I do have experience using Lambda to get Let's Encrypt certificates, but I've only done it for my hobby stuff and not actually as a real production use case. But I'll see if I can help give some pointers.

Yes, the core idea (for Lambda-type function-as-a-service things in general) is that you need to be mindful of all the "state" that you need. For Let's Encrypt, you want your account key and certificate keys to be stored somewhere else, so that your Lambda can load the account key, maybe get the certificate key from somewhere else if the Lambda isn't generating it itself, request the certificate, and then store the certificate (and key if it generated it) somewhere for your application to use.

AWS has plenty of places to store things in its ecosystem, of course. Probably the most "correct" would be to have all your keys in Secrets Manager, but I have my account key in Parameter Store and my certificate keys in S3, since using those is much cheaper, which works fine for me but like I said it's just hobby-type stuff I'm using.

If I were trying to build a Lambda out of certbot, I'd probably explore using Elastic File System to store the persistent parts. But I'm not sure certbot is really the best client to be using for this. Certbot is really designed around just running on a traditional VM-type server and handling it for you. At a scale of 5000 certs per month you're probably outgrowing the kinds of simple management that certbot gives you. I know there are some larger-scale ACME clients out there, though I don't know of anything specifically designed to hook into the AWS/Lambda ecosystem.

Your account access is entirely based on your account private key, and not on the IP used to connect. Let's Encrypt does log the IP, of course, and if you look at the account object in the ACME protocol you can see the IP first used to create the account. And the IP used to create the account is how the account-creation rate limits are tracked. But no, as long as you're using the same account key then you don't need to worry about that each invocation of the API will be using a different IP.

I hope that gives you some direction. If you want to see the code I hacked together for my own use case, I did write up a blog post about it, but it's really more of some ideas to use as a starting point rather than an out-of-the-box solution. Especially at your scale, you'd want to do a lot more tracking of which certificates need renewal and track failures rather than just setting a scheduled task to run every two months and hope they work like I do. :slight_smile:

5 Likes
8

Thanks for the detailed explanation. I will try as you suggested

1 Like
9

ok..thanks. I will try as per your suggestion

1 Like
10

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.