Why am I getting expiration notices?

Help
1

I use LetsEncrypt so that I don't have to worry about renewing certificates. I continue to receive expiration notice emails. So far as I know, my connections to LetsEncrypt are working fine and my certificates are being automatically renewed as expected.

Are these notification emails spurious, or do I need to pay attention to them?

Here is the text of the email I just now received regarding 'covid.zeetix.com':

Hello,

Your certificate (or certificates) for the names listed below will expire in 9 days (on 21 Nov 22 15:51 +0000). Please make sure to renew your certificate before then, or visitors to your web site will encounter errors.

We recommend renewing certificates automatically when they have a third of their total lifetime left. For Let's Encrypt's current 90-day certificates, that means renewing 30 days before expiration. See https://letsencrypt.org/docs/integration-guide/ for details.

covid.tms.micallef.zeetix.com
micallef.zeetix.com
tms.micallef.zeetix.com

For details about when we send these emails, please visit: https://letsencrypt.org/docs/expiration-emails/ In particular, note that this reminder email is still sent if you've obtained a slightly different certificate by adding or removing names. If you've replaced this certificate with a newer one that covers more or fewer names than the list above, you may be able to ignore this message.

For any questions or support, please visit: https://community.letsencrypt.org/ Unfortunately, we can't provide support by email.

If you are receiving this email in error, unsubscribe at:
  http://delivery.letsencrypt.org/track/unsub.php?u=30850198&id=9d5fee7fd2f3471ead1eb422d8a8f3f3.Mw%2FLJbhvRJIpbPt23sKkfecD2Ws%3D&r=https%3A%2F%2Fmandrillapp.com%2Funsub%3Fmd_email%3Dt%252A%252A%252A%252A%2540z%252A%252A%252A%252A.%252A%252A%252A
Please note that this would also unsubscribe you from other Let's Encrypt service notices, including expiration reminders for any other certificates.

Regards,
The Let's Encrypt Team

I invite the guidance of this community and appreciate your attention. Details of the configuration of one of my two sites follow.

Here is the content of '/etc/letsencrypt/renewal/covid.zeetix.com.conf':

# renew_before_expiry = 30 days
version = 1.31.0
archive_dir = /etc/letsencrypt/archive/covid.zeetix.com
cert = /etc/letsencrypt/live/covid.zeetix.com/cert.pem
privkey = /etc/letsencrypt/live/covid.zeetix.com/privkey.pem
chain = /etc/letsencrypt/live/covid.zeetix.com/chain.pem
fullchain = /etc/letsencrypt/live/covid.zeetix.com/fullchain.pem

# Options used in the renewal process
[renewalparams]
account = 9a4a1625821cf5d9346139f02aec1144
authenticator = apache
installer = apache
server = https://acme-v02.api.letsencrypt.org/directory
key_type = rsa

Here is the content of today's log ('/var/log/letsencrypt/letsencrypt.log'):

2022-11-11 04:50:02,945:DEBUG:urllib3.connectionpool:http://localhost:None "GET /v2/connections?snap=certbot&interface=content HTTP/1.1" 200 97
2022-11-11 04:50:03,424:DEBUG:certbot._internal.main:certbot version: 1.32.0
2022-11-11 04:50:03,424:DEBUG:certbot._internal.main:Location of certbot entry point: /snap/certbot/2511/bin/certbot
2022-11-11 04:50:03,424:DEBUG:certbot._internal.main:Arguments: ['-q', '--preconfigured-renewal']
2022-11-11 04:50:03,424:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#nginx,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2022-11-11 04:50:03,453:DEBUG:certbot._internal.log:Root logging level set at 40
2022-11-11 04:50:03,454:DEBUG:certbot._internal.display.obj:Notifying user: Processing /etc/letsencrypt/renewal/covid.zeetix.com.conf
2022-11-11 04:50:03,482:DEBUG:certbot._internal.plugins.selection:Requested authenticator <certbot._internal.cli.cli_utils._Default object at 0x7ffae8645940> and installer <certbot._internal.cli.cli_utils._Default object at 0x7ffae8645940>
2022-11-11 04:50:03,511:DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): r3.o.lencr.org:80
2022-11-11 04:50:03,573:DEBUG:urllib3.connectionpool:http://r3.o.lencr.org:80 "POST / HTTP/1.1" 200 503
2022-11-11 04:50:03,574:DEBUG:certbot.ocsp:OCSP response for certificate /etc/letsencrypt/archive/covid.zeetix.com/cert2.pem is signed by the certificate's issuer.
2022-11-11 04:50:03,575:DEBUG:certbot.ocsp:OCSP certificate status for /etc/letsencrypt/archive/covid.zeetix.com/cert2.pem is: OCSPCertStatus.GOOD
2022-11-11 04:50:03,578:DEBUG:certbot._internal.display.obj:Notifying user: Certificate not yet due for renewal
2022-11-11 04:50:03,579:DEBUG:certbot._internal.plugins.selection:Requested authenticator apache and installer apache
2022-11-11 04:50:03,583:DEBUG:certbot._internal.plugins.selection:Selecting plugin: * apache
Description: Apache Web Server plugin
Interfaces: Installer, Authenticator, Plugin
Entry point: apache = certbot_apache._internal.entrypoint:ENTRYPOINT
Initialized: <certbot_apache._internal.override_fedora.FedoraConfigurator object at 0x7ffae863f880>
2022-11-11 04:50:03,584:DEBUG:certbot.plugins.storage:Plugin storage file /etc/letsencrypt/.pluginstorage.json was empty, no values loaded
2022-11-11 04:50:03,584:DEBUG:certbot._internal.display.obj:Notifying user: 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2022-11-11 04:50:03,584:DEBUG:certbot._internal.display.obj:Notifying user: The following certificates are not due for renewal yet:
2022-11-11 04:50:03,584:DEBUG:certbot._internal.display.obj:Notifying user:   /etc/letsencrypt/live/covid.zeetix.com/fullchain.pem expires on 2023-01-30 (skipped)
2022-11-11 04:50:03,584:DEBUG:certbot._internal.display.obj:Notifying user: No renewals were attempted.
2022-11-11 04:50:03,584:DEBUG:certbot._internal.display.obj:Notifying user: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2022-11-11 04:50:03,584:DEBUG:certbot._internal.renewal:no renewal failures

I see no indication of any auto-renewal issues here.

Standard form answers follow

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: 'covid.zeetix.com' and 'byron.zeetix.com'

I ran this command: (standard letsencrypt install)

It produced this output: (standard letsencrypt output)

My web server is (include version): Apache/2.4.37 (rocky)

The operating system my web server runs on is (include version): Rocky Linux 8.6 (Green Obsidian)
kernel: Linux 4.18.0-372.26.1.el8_6.x86_64

My hosting provider, if applicable, is: AWS EC2

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 1.32.0

2

No, it's not. The email is mentioning different hostnames quite clearly:

As you can see from the CT log history for micallef.zeetix.com (which includes subdomains) at crt.sh | micallef.zeetix.com I see two certs expiring soon. You'll probably receive another expiry email in the next few days for the covid.covid.micallef.zeetix.com cert.

7 Likes
3

Indeed, I noticed that as well after posting the update.

I see something similar with the other domain I'm using -- 'byron.zeetix.com'.

The subject of the notification email is: 'Let's Encrypt certificate expiration notice for domain "byron.zeetix.com" (and 4 more)'. Here is the content:

Hello,

Your certificate (or certificates) for the names listed below will expire in 9 days (on 21 Nov 22 15:36 +0000). Please make sure to renew your certificate before then, or visitors to your web site will encounter errors.

We recommend renewing certificates automatically when they have a third of their total lifetime left. For Let's Encrypt's current 90-day certificates, that means renewing 30 days before expiration. See https://letsencrypt.org/docs/integration-guide/ for details.

byron.zeetix.com
covid.tms.byron.zeetix.com
tms.byron.zeetix.com
wiki.byron.zeetix.com
zeewiki.byron.zeetix.com

For details about when we send these emails, please visit: https://letsencrypt.org/docs/expiration-emails/ In particular, note that this reminder email is still sent if you've obtained a slightly different certificate by adding or removing names. If you've replaced this certificate with a newer one that covers more or fewer names than the list above, you may be able to ignore this message.

For any questions or support, please visit: https://community.letsencrypt.org/ Unfortunately, we can't provide support by email.

If you are receiving this email in error, unsubscribe at:
  http://delivery.letsencrypt.org/track/unsub.php?u=30850198&id=da00c39058704a239600449c554432b9.Mw%2FLJbhvRJIpbPt23sKkfecD2Ws%3D&r=https%3A%2F%2Fmandrillapp.com%2Funsub%3Fmd_email%3Dt%252A%252A%252A%252A%2540z%252A%252A%252A%252A.%252A%252A%252A
Please note that this would also unsubscribe you from other Let's Encrypt service notices, including expiration reminders for any other certificates.

Regards,
The Let's Encrypt Team

Here is the result of running 'certbot certificates' on 'byron.zeetix.com':

# certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Found the following certs:
  Certificate Name: byron.zeetix.com
    Serial Number: 40bf0e1388f3808f32699dc87365e5aeb3f
    Key Type: RSA
    Domains: byron.zeetix.com covid.tms.byron.zeetix.com fullstack.tms.byron.zeetix.com tms.byron.zeetix.com
    Expiry Date: 2022-12-15 18:16:38+00:00 (VALID: 33 days)
    Certificate Path: /etc/letsencrypt/live/byron.zeetix.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/byron.zeetix.com/privkey.pem
  Certificate Name: insecure-test.byron.zeetix.com-0001
    Serial Number: 3a0a8470524eda7a6cb8b29290ca91012a0
    Key Type: RSA
    Domains: insecure-test.byron.zeetix.com byron.zeetix.com
    Expiry Date: 2023-01-25 11:16:38+00:00 (VALID: 74 days)
    Certificate Path: /etc/letsencrypt/live/insecure-test.byron.zeetix.com-0001/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/insecure-test.byron.zeetix.com-0001/privkey.pem
  Certificate Name: insecure-test.byron.zeetix.com
    Serial Number: 3d84bdf3390cd01215a92173968b0598ff2
    Key Type: RSA
    Domains: insecure-test.byron.zeetix.com secure-test.byron.zeetix.com
    Expiry Date: 2023-01-25 11:16:46+00:00 (VALID: 74 days)
    Certificate Path: /etc/letsencrypt/live/insecure-test.byron.zeetix.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/insecure-test.byron.zeetix.com/privkey.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

The last two items in the above response are irrelevant to this thread.

Here are the domains, excerpted from the response:

byron.zeetix.com,
covid.tms.byron.zeetix.com,
fullstack.tms.byron.zeetix.com,
tms.byron.zeetix.com

It is true that I removed the following domains from this cert ()

wiki.byron.zeetix.com
zeewiki.byron.zeetix.com

The notification email includes this: "If you've replaced this certificate with a newer one that covers more or fewer names than the list above, you may be able to ignore this message."

Presumably that's what's going on here -- the current certificate for 'byron.zeetix.com' no longer covers 'wiki.byron.zeetix.com' and 'zeewiki.byron.zeetix.com'.

It does, however still cover the four domains answered by 'certbot certificates'.

I think I can ignore both of these. It appears to me that, in the fullness of time, the domains enumerated in a notification email might be narrowed slightly. I expect the two domains that I removed to expire. I expect the domains enumerated by 'certbot certificates' to continue to be renewed automatically.

Perhaps the text of the notification email might be changed to more accurately reflect what appears to be happening -- the certificate used by the enumerated domain names has been replaced by a new certificate that does not include those domain names.

The confusing part, for me, is the implication of the notification email that the certificate for "byron.zeetix.com" is expiring. The reality appears to be that the certificate for 'byron.zeetix.com' with the expiring domain names has been replaced by a newer certificate for 'byron.zeetix.com' that does not include some of the domain names.

The bottom line is that LetsEncrypt appears to be doing the right thing -- it appears that the notification emails confused me.

4

The Let's Encrypt servers don't know that. You, as a person know that, perhaps Certbot, which is a local ACME client only, could be made that smart, but the Let's Encrypt ACME server doesn't and cannot know which cert was not renewed on purpose or which cert should be renewed.

5 Likes
5

Let's Encrypt ACME server doesn't and cannot know which cert was not renewed on purpose or which cert should be renewed.

Perhaps I'm being unclear. I understand that the ACME server can't know that -- that's what I contemplate communicating in an improved notification email message.

It appears that there are actually two certificates at play here -- let me call them "expiring certificate" and "newer certificate" -- each pertaining to the topmost domain name ("byron.zeetix.com" in this case). I think the ACME server knows of each of the expiring and newer certificates.

Here is some notification email text that I would find less confusing:

A certificate for byron.zeetix.com will expire in 9 days (on 21 Nov 22 15:36 +0000). A newer certificate for byron.zeetix.com (expiring on 15 Dec 22 18:16 +0000) has different domain names from the expiring certificate.

Visitors to your web site who use domain names not on the newer certificate may encounter errors.

Here are the domain names on the expiring certificate:

byron.zeetix.com
covid.tms.byron.zeetix.com
tms.byron.zeetix.com
wiki.byron.zeetix.com
zeewiki.byron.zeetix.com

Here are the domain names on the newer certificate:

byron.zeetix.com
covid.tms.byron.zeetix.com
fullstack.tms.byron.zeetix.com
tms.byron.zeetix.com

The remainder of the notification email is clear as-is.

6

Yes, but the hostnames are different. The ACME server only considers certs with the same set of hostnames as a renewal. The rest it will send an expiry email for, as it doesn't know the lack of renewal was intentional or not. Without that knowledge I'm not sure how the email could be improved.

Hm, I understand your point better after reading your post again. It's probably a possibility to make the emailer smarter, but I'm not sure (and note that I'm not LE staff) the LE developers have the dev time to do something like that. Especially as some other, IMO more important, features have been shelfed before due to lack of time.

4 Likes
7

I understand about priorities.

Regarding byron.zeetix.com, I think that there is just one hostname (byron.zeetix.com) and I think there is an older and newer certificate for that hostname. I think the difference between the older and newer certificate is the "domains" (in the 'domains' field answered by 'certbot certificates' for the certificate with hostname 'byron.zeetix.com').

If nothing else, perhaps our exchanges on this topic will help those who come after sort this out.

8

I don't see how the email can be any clearer.
A certificate renewal is only considered as such when the new cert has the exact same set of names - nor more and no less.
Anything else is just another certificate - which may or may not be used by a system as a replacement certificate for any number of previously issued certificates.

It's all about perspective: From where you are sitting, it may seem quite obvious that one "replaced" the other. But, from anywhere else on the planet, that isn't the case.

4 Likes
9

@rg305: This response is not helpful.

To wit:

A certificate renewal is only considered as such when the new cert has the exact same set of names - nor more and no less.

Please slow down. The '/etc/letsencrypt' directory on my system has the following three subdirectories of interest:

renewal/
archive/
live/

Each of these three subdirectories of interest has a subdirectory that contains certificates. I will call the name of this subdirectory the "hostname" of the certificate it pertains to.

In the case of 'byron.zeetix.com', the three directories of interest are therefore:

/etc/letsencrypt/renewals/byron.zeetix.com/
/etc/letsencrypt/archive/byron.zeetix.com/
/etc/letsencrypt/live/byron.zeetix.com/

The hostname of interest in this part of this exchange is "byron.zeetix.com"

Presumably the ACME server knows about this hostname -- regardless of how many domains are associated with the hostname -- separately from whatever domains are specified by the certificate. The "expiration" date is a property of the certificate with a given hostname.

As I understand it, each certificate with a given hostname is renewed automatically by certbot. The renewed certificate has the same hostname and a new expiration date. My understanding of this is strengthened by the structure of '/etc/letsencrypt/live/byron.zeetix.com/', which contains symbolic links to specific versions (in '/etc/letsencrypt/archive/byron.zeetix.com/') of the following four items:

cert.pem
chain.pem
fullchain.pem
privkey.pem

At the moment, these four items are linked to the following specific counterparts in '/etc/letsencrypt/archive/byron.zeetix.com/':

/etc/letsencrypt/archive/byron.zeetix.com/cert10.pem
/etc/letsencrypt/archive/byron.zeetix.com/chain10.pem
/etc/letsencrypt/archive/byron.zeetix.com/fullchain10.pem
/etc/letsencrypt/archive/byron.zeetix.com/privkey10.pem

It's all about perspective: From where you are sitting, it may seem quite obvious that one "replaced" the other. But, from anywhere else on the planet, that isn't the case.

The perspective that I care about is what is and is not in the subdirectory structure managed by certbot and LetsEncrypt. So far as I know, the only agent that changes that subdirectory structure is certbot itself. I do not make changes from any shell and there are no other agents on this system that modify the directory structure under '/etc/letsencrypt/'.

Is there some other agent besides certbot that changes the directory structure we're discussing? Is there anywhere else on the planet where certbot does something different than it is doing on the system I've described here?

Telling me I'm stupid isn't helpful. Sending me notification emails that are false alarms -- especially when the server knows or should know that they are false alarms -- is at best counterproductive.

I'm doing my best to offer constructive feedback about how LetsEncrypt wasted several hours of my time today. I don't appreciate the tone of this specific response to that constructive feedback.

10

Hi :slightly_smiling_face:

Firstly, Let's Encrypt and certbot are managed by two completely independent companies. Sure, they have a close, shared history, but don't believe for an instant that Let's Encrypt has any awareness whatsoever as to what's in your certbot data directories.

Secondly, please read the post linked below. It explains absolutely everything that matters here.

7 Likes
11

Please note that Certbot, the ACME client developed by EFF, and Let's Encrypt, the CA from ISRG with an ACME server API, are two different things entirely.

Which is much appreciated.

That was probably quite unnecessary. I'm sure you've seen and read the documentation at Expiration Emails - Let's Encrypt as it was linked in the email. That should have explained everything which should have lead you to not waste several hours of your time today. If the documentation page was not clear enough, please provide any tips to improve it.

6 Likes
12

Nope. Pointing out your erroneous thinking is (helpful).

In what way are they false alarms? How is anyone without knowledge of your specific certificate usage, not acquisition history, usage, supposed to know whether or not you have chosen to use one certificate over another thereby obsoleting the latter? You could have load balancers pointing all over the planet with servers terminating TLS for any combination of domain names. Absolutely nothing can be correctly inferred from the certs you have sitting on any server anywhere. Even if you deleted a certificate from your certbot data directories, it could still be in use in memory on that very server (or anywhere else for that matter) and it certainly still exists in history via CTL.

Your lack of knowledge and incorrect assumptions wasted your time today. Happens to us all from time to time.

I didn't quite get the offense of the tone of said response, but then again I have known its author to be of impeccable fairness and compassion with a sometimes misinterpreted sense of humor. Perhaps not taking things personally is highly advisable here (as it is most everywhere).

7 Likes
13

I'm trying to do just that, and I'm getting defensive responses.

What I know is that I got a notification email from "Let's Encrypt Expiry Bot" warning me about the following domains that I want to be renewed automatically:

byron.zeetix.com
tms.byron.zeetix.com
covid.tms.byron.zeetix.com

The specifics are of how the "Let's Encrypt Expiry Bot" determined these are broken. They are broken because these domains are not expiring. They are not expiring because I can see from the directory structure managed by Let's Encrypt that they do not expire on 21-Nov-2022 as asserted in the email.

I expect a notification email like this to contain specific instructions about how to proceed. A link to a documentation URL that in turn says "If you’ve issued a new certificate that adds or removes a name relative to your old certificate, you will get expiration email about your old certificate" is not helpful because it offers no insight about whether or not further action is needed on my part.

I've tried to offer better language here, and I frankly regret the attempt. These responses leave me with the impression that defending the status quo is preferable to contemplating a change.

I feel as though I've jumped into an episode of "IT Crowd". :slight_smile:

14

...and here I think we have the nub of the misunderstanding. The expiration email system doesn't care about domains; it notifies you about certificates that are about to expire. If you had a certificate for a.foo.bar, b.foo.bar, and c.foo.bar, and replaced it with one for a.foo.bar, b.foo.bar, c.foo.bar, and d.foo.bar, then the former certificate will expire. And, if you've given Let's Encrypt your email address, you'll get notifications of that fact--three of them. And each one of those will tell you, as you read and quoted earlier in this topic:

If you've replaced this certificate with a newer one that covers more or fewer names than the list above, you may be able to ignore this message.

...and that should be the end of it--there's no reason you should have taken more than a few minutes to conclude that you've done exactly that and therefore could ignore this message.

You're right that the Let's Encrypt servers could see that FQDNs on the expiring cert were covered by another cert, if they were programmed to do so. But I'd disagree that they should then assume that the new cert was intended to be a replacement for the old cert, because there could well be a situation where a user intended to keep both certs.

9 Likes
16

There is no "big brother" watching over you or your use of your certs.
There is no way for anyone (without access to your systems) to know what you do or no longer do with any certs.

Your local instance of certbot might know what you think it should know about your cert usage.
But that knowledge is never shared with anyone ever.

Only you could know for sure and that is why the email says:
If you've replaced this certificate with a newer one that covers more or fewer names than the list above, you may be able to ignore this message.

A example case in point:
You could be running multiple independent servers behind the same IP address.

  • SMTP server
  • Web Server
  • Media Streaming

They each could share the exact same FQDN(s).
Each could have its' own ACME client.
Each could have its' own certificate.

If you added a web server to the list, the web server might decide to replace that current/active cert with a newer one that now has that one extra FQDN.
But the other two servers don't need to replace their certs, so now there are different cert "paths" taken.; As they nothing about the new web site.

If one was to assume that the new cert replaces all possible uses of any similar certs, then you would never get a notification when the ACME client on the other systems ever failed. Those certs would expire, and you would never be notified about it - because somewhere on the Internet exists a valid cert with all the names (plus one more name) than the cert that expired.

certificates are NOT like license plates for your car.
Where each plate is accounted for, and its' tag must be renewed individually.
With LE certs, you can get (almost) as many as you want and in as many combinations as you can imagine.
But, with that power comes a bit of added responsibility.

5 Likes
17

Did I miss that post?

5 Likes
19

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.