Thanks for chiming-in, Erica.
As @erica and others said, it's not a good idea to make assumptions about this because it can change in a future Certbot release. This structure was brainstormed and proposed at a Certbot developer retreat back in 2015 (?) and was a compromise among many different goals and use cases, some of which have not aged that well. Notably, we imagined a lot of people using something like what's now called
certonly and then manually inspecting and configuring their certificate paths. As Certbot's integration with web servers has evolved, the idea of doing this is typically unnecessarily cumbersome and not many users have gotten a benefit from it.
In fact, there are current discussions in the Certbot issue tracker about the idea that Certbot should potentially store less historical information about old certificates, keys, and chains. Currently, it tries to archive all of them that it's ever obtained, and never replace any. One developer has also proposed moving away from symlinks because they don't work as well on Windows (which wasn't a target platform for the original version of Certbot).
Most of the functionality you're asking about is in
and the functions directly called from there. Again, the
_internal in the path is a reminder that the Certbot developers don't expect users or other code to make assumptions about this.
The basic idea is that
renewal/example.com.conf (the "renewal configuration file") contains metadata that defines a particular certificate in the various renewed versions that Certbot has obtained and tracked. (In the past, this was referred to as a "certificate lineage" because it typically includes a sequence of related certificates over time that replace each other in a chronological order. But the term "lineage" was phased out in documentation several years ago because of the possibility that many users wouldn't understand the sequential-replacement-with-history-tracking idea it was meant to evoke.) The user-visible paths that are intended to be used in any reference to a tracked file are the symlinks in
live/example.com, and they are normally relative symlinks to
../../archive/example.com. The symlink names in
live don't have versions because they connote "the currently active version". The "archived" files in
archive, which are the targets of these symlinks, do have versions, normally starting with
1 and counting up sequentially with every successful renewal. The version with the highest number is the current version, because it derives from the most recent successful renewal, and the corresponding symlink in
live should point to it.
For example, for a certificate that has been successfully issued or renewed 54 times, the most recent number suffix in
archive is expected to be
archive/example.com/fullchain54.pem, in which case
live/example.com/fullchain.pem should be a symlink pointing to
../../archive/fullchain54.pem. The corresponding relationship also applies to the other tracked files (
If you successfully renew that certificate again, there will then be an
archive/example.com/fullchain54.pem (and the other files) and the symlinks in
live will get updated to point at those.
A few more likely questions and answers about
Q. Why do we keep all these old versions?
A. Originally, to provide a way for system administrators to roll back to an earlier version of their certificates if they were unhappy with the new ones for some reason!
Q. Does that ever happen?
A. Basically not, maybe a handful of times a year if someone has exceed the certificates per registered domain rate limit by getting certificates with reduced coverage.
Q. Where is the list of domains that Certbot requests for a particular certificate stored?
A. When it's not specified via the command line, Certbot will take this list by directly parsing the most recent existing relevant certificate in
/etc/letsencrypt/live/certname/cert.pem. The list of names requested in a renewal certificate will be identical to the list of names in the most recently issued certificate under the same cert name (subdirectory of /etc/letsencrypt/live).
Q. What about the -0000 and -0001 certificates?
A. These can be created if you request a new certificate that partially overlaps an existing certificate, especially if you request a new certificate that contains a strict subset of the names covered by an existing certificate. Certbot is willing to believe that you intentionally specified a reduced/partially overlapping list because you explicitly wanted two different, overlapping certificates. This turns out not to be true very often in practice, but we thought it could be at one point.
Q. How do I avoid getting the -0000 and -0001 certificates?
A. If you want to do something to an existing certificate, including deliberately changing its domain coverage, use
--cert-name to indicate explicitly which certificate you want Certbot to act on. Certbot takes this as an explicit request to do something to the existing certificate and will not create a new one.
Q. Why shouldn't I move or rename stuff in
A. (1) The Certbot code makes various assumptions about where the files are located, what they're called, and where the symlinks point. (The structure of
/etc/letsencrypt was used in lieu of a database, and it essentially has implicit referential integrity constraints.) If you violate these assumptions, Certbot can get confused and be unable to renew a certificate. This is also part of why you need to preserve symlinks if you back up
/etc/letsencrypt or copy it to another system.
(2) As mentioned above, these behaviors could change in a future release at any point.
Q. Should I create headaches or random backwards-compatibility challenges for @erica and the other developers by writing other code that makes assumptions about the internal structure of
Q. How can I automate stuff related to Certbot without making those assumptions?
A. Use the hooks, like
--deploy-hook, which provide a documented interface to let scripts do things related to certificate renewal and deployment. If there's no hook that does the integration thing you need, you can ask the Certbot developers to add one or to add additional environment variables to the interface of an existing hook to provide the information that your script needs.
Q. Can I store certificates in my own preferred way outside of
A. Yes, for example you could make a separate copy elsewhere with a
--deploy-hook and do whatever you want with that copy. You can also get non-automatically-renewable certificates outside
/etc/letsencrypt by requesting a certificate with
certbot certonly --csr and specifying a CSR file, although people sometimes find this annoying to deal with. It's certainly received much less love and attention than the auto-renewing
/etc/letsencrypt path, and you might have a better experience scripting around a different client if you don't already have a static CSR file representing a request for each certificate you think you might want.
Q. What about
A. This is a separate archive of every private key ever generated by your copy of Certbot in association with a new certificate request (without any metadata indicating which particular certificate it's associated with). The
csr directory similarly archives the CSRs that it generated under-the-hood in the same process. It was probably not helpful to save these and, indeed, the old private keys could be a privacy risk if some TLS sessions with that server were negotiated using an (obsolete) non-forward-secret ciphersuite¹, because in that case the old private keys could be used to decrypt an archived copy of those encrypted sessions. Fortunately, most TLS sessions today use forward-secret ciphersuites, whose privacy isn't compromised by later disclosing the associated certificate's private key.
The benefits that we've seen people get from
keys in practice, as recently pointed out by @bmw, are if people have accidentally deleted other parts of
/etc/letsencrypt (in which case you can get a copy of your own certificate from a public Certificate Transparency log service, and then find the matching key in
/etc/letsencrypt), or potentially if Certbot crashes at certain points in the certificate issuance process. Of course, both of these cases are also pretty rare.
¹ Suppose Alice and Bob each connected to your HTTPS server, which was hosting some kind of chat service, and used it to have a chat with each other, whose contents your server wasn't programmed to log anywhere. Meanwhile, GCHQ recorded both encrypted HTTPS connections as they passed over fiber-optic connections, and saved them in a data center somewhere in case they came in handy later. If Bob's browser used a forward-secret ciphersuite, your web server's hard drive won't contain anything that would help GCHQ decrypt its copy of his HTTPS connection. (Instead, it would probably have to break one of the cryptographic primitives used in the ciphersuite, which is probably very hard, at least without a big quantum computer.) If his browser used a non-forward-secret ciphersuite, GCHQ could use any of Certbot's copies of the certificate's private key on your server's hard drive, if it could get ahold of those copies, to decrypt the connection at any point in the future.
Funny you mention, as SlickStack has always used the
certonly command for exactly those reasons, as we wanted to be super-organized in the server file structure. Plus, our LEMP stack configuration only supports a single TLD domain, meaning we can optimize things quite a bit server-wide.
...And that's precisely why I've always wondered some of these things about Certbot and LE, e.g. do we really need all these folders and historical files? It seems like so many of the decisions that Certbot have made over the years are based on LE rate-limiting rules. "Better not delete anything, just scan old files and see if that combination of domains on a cert exists already...." etc. I mean theoretically if Lets Encrypt servers issued you a new certificate bundle on every single request and old files were either deleted or overwritten every single time, it would be much cleaner overall. I guess that is why some admins prefer other scripts like
acme.sh or otherwise, but I like how Certbot is very Ubuntu-friendly. Perhaps I need to study more about where Certbot ends and Lets Encrypt begins... ultimately, I think the "automation" that some of these tools try to introduce can result in more questions (or surprises) for developers who prefer knowing exactly what is going on and scheduling their own SSL renewals, etc.
Like @danb35 hits upon above, with no automatic expiration of files, you are caught between not wanting to mess with anything (breaking your certs), and simultaneously wanting to cleanup old files for domains you no longer control or that might have expired. For someone like myself who is aiming for a very clean server environment running Nginx, without any cPanel/Apache and only a single TLD domain, Certbot's collection of files and folders kinda stand out like a sore thumb...
Anyway thanks for your detailed info @schoen it will probably help a lot of Googlers!
Thanks for the correction!
Certbot == an ACME client (one of many), (currently) developed by the EFF.
Let's Encryt == ACME server and the CA of ISRG.
It's not that hard
But, next to the rate limits, that would be very, very anti-social with regard to Let's Encrypt! Every issued certificate uses resources and while the cert might be free for you, it costs Let's Encrypt money to issue those certs. Not per certificate per se, but LE does have operational costs et cetera and with higher loads on the LE systems (i.e., when more certs are issued), the higher the costs would get (for example, if new hardware needs to be bought).
So thinking "Meh, I can get a new, free cert anyway, I don't care if I got the same one yesterday and the day before yesterday.." is in my opinion anti-social behaviour.
Hear! Hear! Several of the contributors to this very thread, myself included, have assisted many people suffering from this very dilemma (or some facet of it). I can personally think of more than a few recent instances where complications and misunderstandings of "certificate management" have been the very source of a certbot user's woes.
Perfect case in point:
That's 59 back-and-forth posts to untangle based on a misconception of how to create backups!
Hence why I have advocated for
restore functions (among other things):
Several of you in this thread will not be able to see the topic I just cited because it's in the #lounge, which is restricted to Regulars, Leaders, Moderators, and Staff. I might drag it out into the public for further discussion if there's interest. I am especially interested in the consideration of @certbot-devs there. I'm willing to draft issues against certbot (and possibly PRs as I can) should they arise.
Most of those issues are due to PEBKAC though and not the fault of certbot.
I'm not blaming certbot, per se. I do feel that software in many aspects is only as useful as the understanding of it by its user-base. Hence why I believe that every human-software API should be firmly grounded in KISS. I can all-but-guarantee that if a version of certbot operated exactly as I have described therein (and enough people were able to use it), the headaches of multitudes of certbot users would be alleviated instantly.
As an ACME client developer myself (who is currently in heavy redevelopment of his client), I've certainly taxed my brain with what I am positive are many of the same concerns that the certbot developers have faced for years (possibly since certbot's inception). Given the difference of target audience (nearly mutually-exclusive), some of the design decisions are quite different. For instance, I've opted for NO backups of certificates or private keys for simplicity of management and (a bit of) security. I figured what's the use of having older, duplicate certificates lying around. Not like they're going to be used. If there's a compromise, just get another certificate, rather than collecting "backups".
Yes, I don't think Certbot is very well-optimized for that. For example, there's no option to request a certificate to be saved in the current working directory, except with
--csr, so you can't do that if you don't have or don't want to use a CSR file.
Certbot is most optimized around doing its own automated renewals with its own web server integrations or with mostly-Certbot-provided authenticators and plugins that handle certain other cases well.
But why? Sure, domains you no longer control can be taken care of with
certbot delete. But so what if it keeps old files around? All of the files for a cert put together (cert, chain, fullchain, privkey) consume less than 10 KB of space. They renew, by default, 6x/yr. That means that, for every cert you have, it will grow by about 60 KB/yr. I don't really see value in keeping the old files 99% of the time, but I also don't see that there's a strong reason not to.
Really, it seems like the confusion goes away if you treat the
/etc/letsencrypt directory as a black box. If you want to act on the certificate files directly, certbot (or many other clients) is perfectly capable of saving a copy of them somewhere else.
I'm 99% sure that's exactly what @jessuppi wants to avoid.
I think you're right, but that just leads to the question of "why use certbot, then"? Because that's the way it's designed to work. It's really looking like:
and it'd be much better to use a different client for this application. It's not like there aren't a lot to choose from.
Personally, I would find it flattering if such loyalty were shown to my software that system administrators would openly request to better understand it in order to fully integrate it into their systems. Granted, certbot is open source, but having absolutely lucid documentation of functionality, especially of such a significant endeavor, only serves to strengthen both the development and deployment efforts of certbot. Having utilized and written such documentation while working in R&D on commercial electronics and embedded test devices for telecom, I fully understand the intensity of resources and monotony involved. Multiple-thousand page functional specifications stemming into various manuals and wikis can certainly be daunting.
...and I'm probably on the opposite end of things, in that I'm not much of a fan of certbot at all--it's far too big, with far too many dependencies, for what it needs to do (if
acme.sh had the
acme-dns integration script that certbot does, I'd probably never use certbot). So I may be much more ready to suggest an alternate client than some would be.
I hear ya there for sure. To avoid the dependency issue for my own client (written in PHP), one of my fundamental design requirements was to only utilize the functionality of the most basic set of proven libraries included with the default configuration of GoDaddy shared hosting. We're talking things like OpenSSL and cURL here. I haven't dug into the guts of acme.sh, but I would much imagine a similar set of basics.
Out of curiosity, what is lacking from acme.sh's existing acme-dns script? https://github.com/acmesh-official/acme.sh/wiki/dnsapi#45-use-acme-dns-api
- Behind-the-scenes management of all the acme-dns credentials
- Management of multiple sets of credentials (thereby allowing me to get multiple certs using acme-dns).
- The acme-dns credentials are stored in
.acme.sh/account.conf, which makes me believe it can only handle one set of credentials. If that's correct, it means that a given acme.sh installation can only issue certs for a single FQDN using acme-dns--no multi-SAN certs, and really only a single cert. If that's not correct, the docs are badly in need of an update--which they are in any event.
- The acme-dns credentials are stored in
acme-dns, I must obtain a set of credentials (
curl -s -X POST https://auth.acme-dns.io/register | python -m json.tool), create the appropriate CNAME record, export the credentials into the appropriate environment variables, and then call
acme.sh. Lots of manual work, and as noted above, it can only handle one FQDN per acme.sh installation.
By contrast, with certbot and @joohoi's hook script, I call certbot and tell it to use that script. The script checks the requested domain(s) against the credentials issued previously; if it/they already exist, it's assumed the CNAME records are present, loads the relevant credentials, and goes to town--all without user interaction. If it doesn't have a set of credentials matching one or more domains, it obtains them, stores them, and prompts you to create the appropriate CNAME record(s). Creating the CNAME is the only user interaction needed with acme-dns in this case. Multiple certs, multi-SAN certs, wildcards, are all handled easily.
Thanks for the detailed explanation. I didn't realize the acme.sh implementation was so limited. Not that it necessarily helps your situation, but Posh-ACME's acme-dns implementation works very much like the certbot hook script. Everything is automated except the prompt to create the initial CNAMEs.
I agree, acme.sh should manage multiple credentials for the same dns plugin, I hope someone can add it to acme.sh but just in case, sometimes I've used this to be able to use 2 or more credentials for the same plugin... I know this is dirty as hell but...
cd /root/.acme.sh/ cp dnsapi/dns_acmedns.sh dns_acmedns01.sh sed -i 's/dns_acmedns/dns_acmedns01/g' dns_acmedns01.sh sed -i 's/ACMEDNS_/ACMEDNS01_/g' dns_acmedns01.sh
Now issue your cert:
export ACMEDNS01_UPDATE_URL="https://auth.acme-dns.io/update" export ACMEDNS01_USERNAME="<username>" export ACMEDNS01_PASSWORD="<password>" export ACMEDNS01_SUBDOMAIN="<subdomain>" acme.sh --issue --dns dns_acmedns01 -d example.com -d www.example.com
I know, it is not the best approach but could save your ass in some situations
Edit: I forgot to say that you can also specify a different
--config-home parameter, this way you don't need to modify the dns plugin but you must use always the
--config-home in all your commands and create a new cron entry.
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.