I have many servers running nginx that all are serving files on the same domain. I thought I could run certbot certonly on each of those servers to generate fullchain.pem and privkey.pem files for nginx to use.
But I hit a rate limit on the second server because the first one ran certbot command a few minutes earlier. I got an error saying There were too many requests of a given type :: Error finalizing order :: too many certificates (5) already issued for this exact set of domains in the last 168 hours.
What's the right way to handle generating certificates when you need them on multiple servers? Should you run certbot certonly in one central place (instead of all the nginx servers individually) and push the generated files to all the nginx servers? Or is there any centralized way to do a request through certbot to request the same certificates that were generated earlier from letsencrypt?
That means on server 1 I run certbot certonly (using DNS plugin) and it generates the certificate. Then on server 2 could I do certbot fetch_already_generated to get the same certificate that was generated by server 1, without hitting the rate limit?
Do we need to put fullchain.pem and privkey.pem on each nginx server ourselves or does letsencrypt have a way to get the certificates it generated earlier?
As Osiris already stated, this is not how a CA like Let's Encrypt operates. Let's Encrypt neither generates nor receives your private key. Your private key is private and yours only. Let's Encrypt therefore cannot give you your private key (privkey.pem). Your public part - the signed certificate - (fullchain.pem) can in theory be downloaded from some server (e.g CT logs or whatever), but it is of no use without the private key.
The right way to handle this is to generate the certificate on some more or less central server, and then distribute the required files via a secure channel to the other servers. Ideally this is also an automated process. CDNs often use this method (or similar techniques) for their certificates.
They use example.com and *.example.com as their domains and are accessed at different subdomains like domain1.example.com. So I generate this same certificate and put it on all servers.
Is there any downside to do it this way than to register different certificates like domain1.example.com and domain2.example.com, to generate these on each individual server instead of a central place?
If you generate the certificate on each server, then you've already found one:
If you only generate the certificate once and then copy it around to all the other servers, that would be OK (but doesn't explain the error you received).
If you generate only the individual name(s) needed by each server on each server, then you may overcome that limit error (but will lose the added security, by obscurity, that a wildcard name provides).
but will lose the added security, by obscurity, that a wildcard name provides
Can you explain what does this mean? How does a non wildcard certificate remove security by obscurity. Doesn't someone still need to know the exact domain to know it exists?
If I'm a random bad guy, I can see in CT logs that you've got a certificate for vulnerable.example.com and conclude that you are running vulnerable, a vulnerable service I know how to attack, on that name. Or I can see you have a certificate for chicago-office.example.com and try my attack on that anyway, hoping the Chicago Office has the vulnerable service.
There are some big problems with this strategy, but they aren't necessarily insurmountable. One: If anybody, ever, asks about these DNS names on public Internet systems there's an excellent chance they appear in "passive DNS" data. Big DNS servers don't sell your DNS lookups, but they do sell the set of answers seen to queries, as passive DNS. So you can't ask "What queries did saveee9 run?" or "Who queried pornhub.com?" but you can ask "What DNS answers end in example.com?" and get back a big list from several providers who have bulk data sources. Even if your servers aren't reachable from the public Internet, it only takes one misconfigured VPN, or mistaken click of a bookmark to cause the DNS query.
Two: You probably don't pick unpredictable names. I don't need passive DNS or CT logs to guess that www.company.example is the company web server and that it's worth seeing if mail.company.example, smtp.company.example, jira.company.example and so on exist. This won't find xp493mv2ig.company.example but let's be serious here, your company doesn't use such names.
Zero Trust is the right model. Wildcards are probably not a good idea from a security POV, although they are convenient and sometimes outright necessary.