I have a question regarding a large TLS migration.
We are migrating approximately 1,200 applications from an old Kubernetes cluster to a new platform running Caddy. Currently, all certificates are managed by cert-manager and issued by Let's Encrypt using HTTP-01 validation.
To avoid hitting Let's Encrypt issuance rate limits during the migration, we are considering exporting the existing certificates and private keys from the Kubernetes TLS secrets and importing them into Caddy instead of reissuing all certificates immediately.
My questions are:
Is it a good practice to migrate the existing certificates from cert-manager to Caddy in this way?
Are there any risks or drawbacks to serving certificates originally issued by cert-manager from Caddy?
If Caddy is later configured to manage the same domains with Let's Encrypt, will it be able to renew or replace those certificates without issues?
Has anyone performed a similar migration at this scale, and are there any lessons learned or best practices to share?
The goal is to avoid a large certificate issuance burst during the migration while allowing Caddy to take over certificate management over time.
My personal reading of the Let's Encrypt rate limits is that a certificate renewal is NOT tied to a particular account (my checking of the Boulder source code indicates to me that ARI renewals have to be done using the same account, but non-ARI renewals do not). Obviously you'd want to test that.
Since in theory these would count as renewals you COULD tell Caddy these are new certificates that it manages and you shouldn't hit any rate limits, but you'd probably want to space the migration out and not do all 1200 at once.
The Caddy questions are probably better directed to the Caddy Community: https://caddy.community/ You may get some further comments about that here but keep that in mind in case you don't.
That is a legitimate concern but fellow volunteers have already described the rate limit issues. And, as @kenh1 pointed out you could migrate in batches rather than all at once. That may be preferred anyway so you don't have them all renewing at the same time as well. Not so much for rate limits but to avoid having all certs affected by the same short outage (regardless of cause)
You can probably achieve this but obviously you need to do a test in a dev/staging environment so you have a completed tested migration plan. Nobody (including yourself) will be able to fully predict what could go wrong and you especially need to exercise the point at which Caddy starts to take over it's own renewals.
There are a couple of (outdated) caddy projects to use a secrets store, the benefit of that approach would be you could migrate your certs to the secrets store, then caddy can maintain them:
This is definitely a caddy community question, @mholt might have an idea on the best strategy.
One small point I’d add: if you import the existing certs, keep the private keys out of the container image/deployment repo and put them straight into whatever storage Caddy will actually use in production. Then test one or two names through a full Caddy-managed renewal before migrating the batch. That catches storage/permission/account issues without creating a 1,200-cert renewal wave later.
Yeah there's quite a few ways to go about this. It depends what storage backend you'll be using, how many Caddy instances there are, and some of your own preferences.
If the certificates and keys are moved into Caddy's storage backend with their metadata files, it can take over managing them without any additional issuances, only as certs are getting close to expiration.
I'd be happy to help you with this migration with a sponsorship. I can build you a program that moves your certs over so that Caddy will smoothly start using them all at once. Feel free to email me using the email on our support page and we'll get you set up with a sponsorship, and/or we can book a call at https://matt.chat if you prefer that.