Terraform aws acme certificate time limit exceeds

[petercooperjr]

@FlorianGerdes

With AWS ACM the issue is, that amazon fully manages the certificates for you. So there is way to get your hands on the private certificate yourself to have an EC2 access it. That was my use-case: I wanted an EC2 auto-scaling group to access the certificate directly.

If what you're running within the EC2 is nginx (or perhaps some other software that supports PKCS11 though I don't think much does), then you may want to take a look at AWS Certificate Manager for Nitro Enclaves, which doesn't let you access the private key directly but has an interface for a program running on the EC2 instance to utilize it.

[FlorianGerdes]

I was just now retrying the terraform call from within a machine in my AWS account. I tried it 3 times and turns out the error messages i received were different to the ones I got locally:

edit: Which indicates some connection problem. Maybe that was the problem all along? I'll make another test tomorrow with a different network setup.

@MikeMcQ

@FlorianGerdes

my (current) workaround is to get a loadbalancer in between

If you were going to acquire a new cert for each new spin-up that would be problematic for those purposes and may incur LE rate limits. To avoid that you would need to acquire certs in a different dedicated process and store them durably for quick load on spin-up. And, occasionally refresh certs in long-running instances. Maybe this was your plan all along but the LB does that for you.

Exactly. I wasn't gonna regenerate the certificate every time, but store it somewhere safely (e.g. AWS Secret Manager) and just have the EC2 use it.

[petercooperjr]

@FlorianGerdes

I wasn't gonna regenerate the certificate every time, but store it somewhere safely (e.g. AWS Secret Manager) and just have the EC2 use it.

Another option, or at least what I've done, is to run a Lambda to do the renewal (via DNS-01 challenges) and store it securely (I just use S3, though Secrets Manager would probably be wiser), entirely separately from the systems using it. Then the systems using it just need to get the certificate from the secure storage each time. I've got the system I hacked together posted at https://cooperjr.name/2020/08/13/acme-lambda-renewal/, though be aware that it's more of a proof-of-concept you could use as a starting point for your own development than an off-the-shelf client designed for the public.

[MikeMcQ]

@FlorianGerdes

some connection problem. Maybe that was the problem all along?

I don't know. My thought was Terraform was just reporting the error message it got from the Let's Encrypt server. If that was true the connection is between the LE server and Route53 so not affected by your requester location.

But, if @petercooperjr's post #8 is correct then Terraform could be doing its own validation of the path to Route53 and failing for the various reasons shown. If this is true the requester network setup matters. This would also explain why we can see your TXT record from other locations and no one else has reproduced the error message.

Mind you, I am just theorizing. Is it possible to ask Terraform if they do this?

I still think a load balancer is viable. You are in part reinventing a wheel. It might be a better solution - not just a work-around.

[rg305]

@FlorianGerdes

Because i was struggling with the certificate in the first place. I would setup such a record once I have the rest of the pieces in place. Is that a problem?

No, but it does limit your authorization choices to just DNS challenges.

[FlorianGerdes]

@MikeMcQ

But, if @petercooperjr's post #8 is correct then Terraform could be doing its own validation of the path to Route53 and failing for the various reasons shown.

@petercooperjr

What should happen is that the ACME client should be checking Route 53's API to see if the DNS servers are in sync (that is, if the change set is done) before proceeding.

Isn't that why the connectivity is the problem? The ACME client runs locally. So there might be some weird vm dns settings that do not allow the connection.

Anyways, I tried the entire process from a machine within the same AWS account, allowing all traffic in the security groups (for now). Turns out it works.

Happy that we solved the issue, even though as you suggested @MikeMcQ I'll sick to the loadbalancer setup. Thank you very much for your support everyone. Nice community.

Any last thoughts?