Issuing/Distributing Wildcard Cert?

Apologies up front for not filling out the form, but I think a straightforward explanation will be best. Say I plan to have many subdomains under foobar.com. I’ve already successfully generated a wildcard cert for this domain name using acme.sh. It worked great and all is good.

Now, in the future, I’ll be spinning up various VMs in an automated fashion where these VMs will each have an Nginx web server running and reachable by some unique subdomain like one.foobar.com or two.foobar.com, etc.

I’m not sure what the best way to install the wildcard cert on these spun up VMs in an automated fashion is. I’m concerned about hitting Let’s Encrypts renew/issue limit if I just use acme.sh --issue each time. I’m thinking maybe I have the existing wildcard cert files automatically copied to the new VM and then use the acme.sh --install-cert or --deploy option to get the cert installed and working on the VM. However, I’m finding very little info on either of these options.

Anyone have any advice? Thanks in advance.

Can’t you somehow “mount” or “map” an external directory containing the certificate/private key in those VMs?

Hi Osiris - Thanks for the reply. Our requirements are that these VMs will actually run on physical machines that are located in separate, fairly locked down, physical locations. So giving these machines visibility to a shared drive somewhere in the cloud unfortunately isn’t an option.

You should not be generating new certificates, but deploying existing ones.

In your situation, since the servers are spun-up and not already online, I would centralize all the certificates onto a single machine and do something like:

  1. a nightly crontab on each machine will rsync against the centralized certificate repo, then restart nginx

  2. a post-issuance hook would enroll the Certificate+Key into an encrypted source control system/webserver (that also has authorization systems) and have your servers access and decrypt the latest certificate on startup. a crontab would check nightly too.

An example of this would be using Blackbox (https://github.com/StackExchange/blackbox) which utilizes a GPG keyring to encrypt files at-rest within a repository. The machine that generates certs would use a post-issuance certbot hook to copy the file to the source tree, enroll it for encryption then commit/push. The other machines would have a decryption key, and a crontab every day to check for new certs from a central repository and copy/decrypt/restart your server. it’s more advanced than rsync, but the client machines would just need a “reader” gpg key and access to the repository - which limits the potential security issues.

1 Like

Hi jvanasco - Thanks for the thorough and informative reply. One question:

Why not? Will generating a new certificate invalidate previous generated certificates? My understanding is that it will not invalidate the old certificate: Using older certificate after auto renew issued a replacement

The old certificates are still valid until expiry or being revoked.

My concern is there is an API limit of 5 duplicate certificates per-week - so you can only generate 5 *.example.com certificates each week. If the machines are always on, that’s not an issue - but in an automated environment, that number is uncomfortably low. If you have automated scaling systems in place, you could blow through this limit in a few minutes (or even seconds).

You could get around this limit slightly by having each machine request *.example.com + {machine_uuid}.example.com (and having dns handle that machine_uuid). In this setup, you have 5 certs per week per machine - but will now be concerned with the 50 new certificates per week API limit. To set this up though, you’re basically looking at a system would not make more sense to support dedicated certificates instead of wildcards. Renewals don’t count against the “new certs” ratelimit, but they do count against the “duplicate certs” limit, so automated systems can still impact this negatively.

There are definitely elements about your setup that I don’t know and could absolutely change my opinion, but from what you have shared… it sounds like you will either have to spend a lot of work to ensure you don’t hit the rate limits, or a smaller amount of work to distribute centralized certificates.

1 Like

Thank you, @jvanasco. Yes, I wasn’t 100% sure I was reading the renew limit information correctly that indeed the limit is 5 renewals of my wildcard certificate per week. Okay, now that I’m sure this is indeed the limit, it makes issuing new certificates less attractive than distribution a single certificate.

Personally, every one of those “work arounds” around the rate limits are not very social towards Let’s Encrypt. The rate limits are there for a reason. There are technically possible solutions to @terenced issue. Not very kind to Let’s Encrypt to choose the “lazy” way and impose more load on their systems.

1 Like

Thanks, @Osiris. I hear what you’re saying and don’t want to be the lazy guy making Let’s Encrypt do more work just b/c I’m lazy :slight_smile:

One additional question: Does a certificate renewal issue a new certificate or just extend the expiration date of an existing certificate in Let’s Encrypt’s database? If the answer is the latter, it seems I could simply issue the certificate once, use this certificate on all of my machines, and it will be good infinitely as long as I have it on a single machine that did the renewal within every 90 days.

However, on the flip side, if a renewal does issue new certificate files, then I’m confused what the difference between an issue and renew is, since clearly Let’s Encrypt has different rate limits on issue (50 per week) verse renewal (5 per week).

The “expiration date” is embedded within the certificate itself. It’s not something in a Let’s Encrypt database, that’s not how the PKI system works. To “renew” a certificate, the date within the certificate needs to be changed, but then the signature would become invalid. Therefore, the certificate needs to be signed again by the Let’s Encrypt private key. In essence, you’re getting a whole new certificate. The term “renewal” is just a made up description of a certificate with hostnames within it for which a certificate has been issued previously. You can then use this new term to differentiate specific rate limits for example.

1 Like

Okay, thank you @Osiris for the clarification, I figured the certificate was modified in some way but wasn’t aware how exactly.

As an aside you could also look at certificate/secret ‘vault’ storage. For instance with Certify The Web (my app) you can acquire your certificates and store them in Hashicorp Vault (local or cloud) or Azure Key Vault, then the consumer services/servers just regularly fetch which ‘secret’ they are entitled to from the vault (and they can assume it’s the latest version). That way the servers just have to see able to see your vault with the necessary credentials just to fetch their own cert and they don’t need to be performing cert renewals etc directly.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.