After a few years of learning about (and using) Certbot using custom certbot commands, I've realized that I still don't have a firm grasp of the chronological steps that the API performs.
For example, on Ubuntu, I believe that apt install certbot creates the /etc/letsencrypt/ directory but none of the child directories.
Then after you run certbot commands, child directories are created:
/etc/letsencrypt/accounts/
/etc/letsencrypt/archive/
/etc/letsencrypt/csr/
/etc/letsencrypt/keys/
/etc/letsencrypt/live/
/etc/letsencrypt/renewal/
/etc/letsencrypt/renewal-hooks/
That's about all I know. Obviously, the Certbot API stores SSL certs in the /archive/ folder, and somehow decides which ones to symlink to from within the /live/ folder... but it does not seem documented anywhere as far as the order of the steps that are performed during this process.
I realize things change over time, but I think it would be helpful to sysadmins to understand a bit better the steps that the Certbot API goes through if possible.
For example:
First the certbot CLI tests the connection to Let's Encrypt servers (API)
The API then records server IP address and verifies /etc/letsencrypt/ folder exists
Next the API compares the domains that you are requesting an SSL cert for with the existing cert files in your local /archive/ folder (along with history of requests for those FQDNs)
I hope I'm not asking for too much, but instead of spending hundreds of hours trying to analyze the API, I was hoping that a Certbot developer might be willing to share briefly... thanks!
Edit: before more posts muddy up this thread, I realize that Certbot is a client that connects to the Lets Encrypt servers (API) and is not itself an API... no need to clarify this further here
I'm not sure what you mean by "certbot API"? Most users just use it as a command line interface UI ACME client. While I've heard certbot indeed also sports an API, I've never heard of this API actually being used.
Also, I'm curious, what's your ultimate goal with the requested knowledge? How do you benefit from learning the ultimate and intricate inner workings of certbot?
Certbot doesn't have an API, aside from the commandline flags. LetsEncrypt implements the ACME API - https://tools.ietf.org/html/rfc8555 - which is well documented in the RFC and even has flowcharts.
Certbot is client for the LetsEncrypt API. The only thing that makes it similar to other ACME clients, is that it is built to communicate with the ACME API reference.
In terms of what Certbot does under the hood and it's internal logic flow, that is essentially all "black box" (i.e. private methods that are [purposefully] not documented), as they are likely to change across releases. IIRC, the only things that project aims to keep consistent for backwards compatibility, are the commandline flags and directory structure.
There is a Contributors guide on github, which gives some information and points to a developer chat - https://certbot.eff.org/docs/contributing.html ; that might be your best way to find out exactly how Certbot is accomplishing a particular task right now.
The Certbot system is fairly large and complex though. There are routines designed to determine the certificate hierarchies and renewal dates, others for going through the ACME-api and storing certs on disks, others for installing the certs into webservers, even some for running webservers themselves.. and I've missed a lot.
Those are the internal API docs generated by the autodoc, intended for contributors and developers. Nothing in there is exposed to the public in any way, or would be an API in the context of the original question. autodoc basically pulls the docstrings from the Python files and publishes them onto the templates, mostly about what each class or method does and what the arguments are.
I don't know if the OP is looking for info on the ACME API or the internal certbot logic, but the RFC and contributors-guide/chat-channel are the best resources for each.
Thanks for your helpful response. I assumed some sensitive processes can't be publicly exposed, however even a very basic flowchart of Certbot/LE file management seems non-existent.
For example, even understanding these would help:
How and when does LE determine which /archive/ files to link from /live/ symlinks?
If the /archive/ files and/or /live/ files are deleted/corrupt then what happens?
What conditions prompt LE to delete any child folders?
etc, etc.
I realize it might sound strange, but after years of using Certbot in my FOSS project SlickStack (automated LEMP server provisioning using bash scripts), my understanding of LE hasn't evolved much. Like many sysadmins, I prefer manual configuration and intricate organization of the server, but when it comes to LE it is still largely just "see what happens" or "wait for Certbot to return a message"...
It also makes me question the purpose of having commands like certbot delete or even the point of trying to clean up LE certs (which I've tried various ways) when that might just return a "rate limited" rejection message and leave your server cert-less for hours/days -- on the other hand, if you never delete any files it seems like Certbot intelligently scans /archive/ and symlinks the correct cert if it hasn't been revoked... but back to the topic at hand, the lack of info is frustrating.
In the end, everything is open source. Not only certbot, but also Boulder, the CA software designed, developed and used by Let's Encrypt.
If you really want to, you can "just" look at the source code and find out that way.
Note that certbot an ACME client, one of many. Originally developed by Let's Encrypt, years ago the development went to the EFF. If you want to learn how LE works, you probably want to learn more about the ACME protocol in RFC 8555 or Boulder (referenced above).
Why? Why are you so curious about how certbot, for example, makes the symlinks? I'm pretty sure documenting such questions isn't worth the effort at all. 99,9999999 % of users wouldn't care at all, for the other 0,0000001 % the source code is enough.
Everything is opensource and publicly exposed. The items you listed are largely undocumented because they are constantly changing. If you search the source on github - or even issues and the commit log - you'll see the logic behind each of these actions can change quite a bit over the course of a single year. The source (and dev channel) can give you an idea of how things work today, but that's not guaranteed to work tomorrow.
Where certbot places and reads the files hasn't changed at all, but all the logic/decision-trees on how it gets there has been a moving target. The only thing we can really rely on as Certbot consumers, is that symlinks (or files) will be placed in /live/.
Perhaps in your use-case, using something like acme-sh might be better suited?
Thanks, just hoping an overview might be available in lieu of months of research.
I figured it changed too often to be worth documenting in detail, but even some basic understanding seems like it would help server admins be aware of what's going on, at least.
Thanks for the suggestion, if anyone else has a comment on this topic please let me know...
By observation: when a new certificate under a given name is issued, its files are stored in /etc/letsencrypt/archive/$cert_name/{cert|chain|fullchain|privkey}n.pem, with n representing a sequential number (the first one is 1, the second is 2, etc.). The symlinks in live/$cert_name/ should always point to the newest set of files (i.e., the highest number in /archive/).
Things break. Don't mess directly with the contents of /etc/letsencrypt.
AFAIK, only running certbot delete would result in deleting any files/directories under /etc/letsencrypt. There's no automatic expiration of files; old ones appear to be retained indefinitely.
One pretty obvious case would be for a cert for a domain you no longer control. Keeping it around would result in nothing but repeated errors (probably not enough to hit any rate limit, but useless and annoying nonetheless).
If you've already issued five duplicate certs within the past week, and you then delete all your copies of that cert, I guess this would be possible--it seems a little far-fetched, though.
And while I can understand curiousity, like @Osiris I'm a little puzzled why this information is apparently important to you.
Certbot dev here. We do in fact have an api, see here for the api and here for info on backward compatibility. Note that only a narrow subset of how certbot functions is included in the API; for example, I see /archive/ listed here; it's really meant for certbot's internal storage, and we make no guarantees about its continued behavior there.
As @erica and others said, it's not a good idea to make assumptions about this because it can change in a future Certbot release. This structure was brainstormed and proposed at a Certbot developer retreat back in 2015 (?) and was a compromise among many different goals and use cases, some of which have not aged that well. Notably, we imagined a lot of people using something like what's now called certonly and then manually inspecting and configuring their certificate paths. As Certbot's integration with web servers has evolved, the idea of doing this is typically unnecessarily cumbersome and not many users have gotten a benefit from it.
In fact, there are current discussions in the Certbot issue tracker about the idea that Certbot should potentially store less historical information about old certificates, keys, and chains. Currently, it tries to archive all of them that it's ever obtained, and never replace any. One developer has also proposed moving away from symlinks because they don't work as well on Windows (which wasn't a target platform for the original version of Certbot).
Most of the functionality you're asking about is in
and the functions directly called from there. Again, the _internal in the path is a reminder that the Certbot developers don't expect users or other code to make assumptions about this.
The basic idea is that renewal/example.com.conf (the "renewal configuration file") contains metadata that defines a particular certificate in the various renewed versions that Certbot has obtained and tracked. (In the past, this was referred to as a "certificate lineage" because it typically includes a sequence of related certificates over time that replace each other in a chronological order. But the term "lineage" was phased out in documentation several years ago because of the possibility that many users wouldn't understand the sequential-replacement-with-history-tracking idea it was meant to evoke.) The user-visible paths that are intended to be used in any reference to a tracked file are the symlinks in live/example.com, and they are normally relative symlinks to ../../archive/example.com. The symlink names in live don't have versions because they connote "the currently active version". The "archived" files in archive, which are the targets of these symlinks, do have versions, normally starting with 1 and counting up sequentially with every successful renewal. The version with the highest number is the current version, because it derives from the most recent successful renewal, and the corresponding symlink in live should point to it.
For example, for a certificate that has been successfully issued or renewed 54 times, the most recent number suffix in archive is expected to be archive/example.com/fullchain54.pem, in which case live/example.com/fullchain.pem should be a symlink pointing to ../../archive/fullchain54.pem. The corresponding relationship also applies to the other tracked files (chain54.pem, privkey54.pem, and cert54.pem).
If you successfully renew that certificate again, there will then be an archive/example.com/fullchain54.pem (and the other files) and the symlinks in live will get updated to point at those.
A few more likely questions and answers about /etc/letsencrypt:
Q. Why do we keep all these old versions?
A. Originally, to provide a way for system administrators to roll back to an earlier version of their certificates if they were unhappy with the new ones for some reason!
Q. Does that ever happen?
A. Basically not, maybe a handful of times a year if someone has exceed the certificates per registered domain rate limit by getting certificates with reduced coverage.
Q. Where is the list of domains that Certbot requests for a particular certificate stored?
A. When it's not specified via the command line, Certbot will take this list by directly parsing the most recent existing relevant certificate in /etc/letsencrypt/live/certname/cert.pem. The list of names requested in a renewal certificate will be identical to the list of names in the most recently issued certificate under the same cert name (subdirectory of /etc/letsencrypt/live).
Q. What about the -0000 and -0001 certificates?
A. These can be created if you request a new certificate that partially overlaps an existing certificate, especially if you request a new certificate that contains a strict subset of the names covered by an existing certificate. Certbot is willing to believe that you intentionally specified a reduced/partially overlapping list because you explicitly wanted two different, overlapping certificates. This turns out not to be true very often in practice, but we thought it could be at one point.
Q. How do I avoid getting the -0000 and -0001 certificates?
A. If you want to do something to an existing certificate, including deliberately changing its domain coverage, use --cert-name to indicate explicitly which certificate you want Certbot to act on. Certbot takes this as an explicit request to do something to the existing certificate and will not create a new one.
Q. Why shouldn't I move or rename stuff in /etc/letsencrypt?
A. (1) The Certbot code makes various assumptions about where the files are located, what they're called, and where the symlinks point. (The structure of /etc/letsencrypt was used in lieu of a database, and it essentially has implicit referential integrity constraints.) If you violate these assumptions, Certbot can get confused and be unable to renew a certificate. This is also part of why you need to preserve symlinks if you back up /etc/letsencrypt or copy it to another system.
(2) As mentioned above, these behaviors could change in a future release at any point.
Q. Should I create headaches or random backwards-compatibility challenges for @erica and the other developers by writing other code that makes assumptions about the internal structure of /etc/letsencrypt?
A. Nope!
Q. How can I automate stuff related to Certbot without making those assumptions?
A. Use the hooks, like --deploy-hook, which provide a documented interface to let scripts do things related to certificate renewal and deployment. If there's no hook that does the integration thing you need, you can ask the Certbot developers to add one or to add additional environment variables to the interface of an existing hook to provide the information that your script needs.
Q. Can I store certificates in my own preferred way outside of /etc/letsencrypt?
A. Yes, for example you could make a separate copy elsewhere with a --deploy-hook and do whatever you want with that copy. You can also get non-automatically-renewable certificates outside /etc/letsencrypt by requesting a certificate with certbot certonly --csr and specifying a CSR file, although people sometimes find this annoying to deal with. It's certainly received much less love and attention than the auto-renewing /etc/letsencrypt path, and you might have a better experience scripting around a different client if you don't already have a static CSR file representing a request for each certificate you think you might want.
Q. What about keys?
A. This is a separate archive of every private key ever generated by your copy of Certbot in association with a new certificate request (without any metadata indicating which particular certificate it's associated with). The csr directory similarly archives the CSRs that it generated under-the-hood in the same process. It was probably not helpful to save these and, indeed, the old private keys could be a privacy risk if some TLS sessions with that server were negotiated using an (obsolete) non-forward-secret ciphersuite¹, because in that case the old private keys could be used to decrypt an archived copy of those encrypted sessions. Fortunately, most TLS sessions today use forward-secret ciphersuites, whose privacy isn't compromised by later disclosing the associated certificate's private key.
The benefits that we've seen people get from keys in practice, as recently pointed out by @bmw, are if people have accidentally deleted other parts of /etc/letsencrypt (in which case you can get a copy of your own certificate from a public Certificate Transparency log service, and then find the matching key in /etc/letsencrypt), or potentially if Certbot crashes at certain points in the certificate issuance process. Of course, both of these cases are also pretty rare.
¹ Suppose Alice and Bob each connected to your HTTPS server, which was hosting some kind of chat service, and used it to have a chat with each other, whose contents your server wasn't programmed to log anywhere. Meanwhile, GCHQ recorded both encrypted HTTPS connections as they passed over fiber-optic connections, and saved them in a data center somewhere in case they came in handy later. If Bob's browser used a forward-secret ciphersuite, your web server's hard drive won't contain anything that would help GCHQ decrypt its copy of his HTTPS connection. (Instead, it would probably have to break one of the cryptographic primitives used in the ciphersuite, which is probably very hard, at least without a big quantum computer.) If his browser used a non-forward-secret ciphersuite, GCHQ could use any of Certbot's copies of the certificate's private key on your server's hard drive, if it could get ahold of those copies, to decrypt the connection at any point in the future.
Funny you mention, as SlickStack has always used the certonly command for exactly those reasons, as we wanted to be super-organized in the server file structure. Plus, our LEMP stack configuration only supports a single TLD domain, meaning we can optimize things quite a bit server-wide.
...And that's precisely why I've always wondered some of these things about Certbot and LE, e.g. do we really need all these folders and historical files? It seems like so many of the decisions that Certbot have made over the years are based on LE rate-limiting rules. "Better not delete anything, just scan old files and see if that combination of domains on a cert exists already...." etc. I mean theoretically if Lets Encrypt servers issued you a new certificate bundle on every single request and old files were either deleted or overwritten every single time, it would be much cleaner overall. I guess that is why some admins prefer other scripts like acme.sh or otherwise, but I like how Certbot is very Ubuntu-friendly. Perhaps I need to study more about where Certbot ends and Lets Encrypt begins... ultimately, I think the "automation" that some of these tools try to introduce can result in more questions (or surprises) for developers who prefer knowing exactly what is going on and scheduling their own SSL renewals, etc.
Like @danb35 hits upon above, with no automatic expiration of files, you are caught between not wanting to mess with anything (breaking your certs), and simultaneously wanting to cleanup old files for domains you no longer control or that might have expired. For someone like myself who is aiming for a very clean server environment running Nginx, without any cPanel/Apache and only a single TLD domain, Certbot's collection of files and folders kinda stand out like a sore thumb...
Anyway thanks for your detailed info @schoen it will probably help a lot of Googlers!
Certbot == an ACME client (one of many), (currently) developed by the EFF.
Let's Encryt == ACME server and the CA of ISRG.
It's not that hard
But, next to the rate limits, that would be very, very anti-social with regard to Let's Encrypt! Every issued certificate uses resources and while the cert might be free for you, it costs Let's Encrypt money to issue those certs. Not per certificate per se, but LE does have operational costs et cetera and with higher loads on the LE systems (i.e., when more certs are issued), the higher the costs would get (for example, if new hardware needs to be bought).
So thinking "Meh, I can get a new, free cert anyway, I don't care if I got the same one yesterday and the day before yesterday.." is in my opinion anti-social behaviour.
Hear! Hear! Several of the contributors to this very thread, myself included, have assisted many people suffering from this very dilemma (or some facet of it). I can personally think of more than a few recent instances where complications and misunderstandings of "certificate management" have been the very source of a certbot user's woes.
Perfect case in point:
That's 59 back-and-forth posts to untangle based on a misconception of how to create backups!
Hence why I have advocated for backup and restore functions (among other things):
Several of you in this thread will not be able to see the topic I just cited because it's in the Lounge, which is restricted to Regulars, Leaders, Moderators, and Staff. I might drag it out into the public for further discussion if there's interest. I am especially interested in the consideration of @certbot-devs there. I'm willing to draft issues against certbot (and possibly PRs as I can) should they arise.
I'm not blaming certbot, per se. I do feel that software in many aspects is only as useful as the understanding of it by its user-base. Hence why I believe that every human-software API should be firmly grounded in KISS. I can all-but-guarantee that if a version of certbot operated exactly as I have described therein (and enough people were able to use it), the headaches of multitudes of certbot users would be alleviated instantly.
As an ACME client developer myself (who is currently in heavy redevelopment of his client), I've certainly taxed my brain with what I am positive are many of the same concerns that the certbot developers have faced for years (possibly since certbot's inception). Given the difference of target audience (nearly mutually-exclusive), some of the design decisions are quite different. For instance, I've opted for NO backups of certificates or private keys for simplicity of management and (a bit of) security. I figured what's the use of having older, duplicate certificates lying around. Not like they're going to be used. If there's a compromise, just get another certificate, rather than collecting "backups".
