SSL renewal mysterious issue


#1

Today my server went off due to SSL issue (ERR_CERT_DATE_INVALID).

I use a crontab task (once a day) to automatically renew it so I was quite surprised.

I logged into sever and typed certbot certificates to check if my certificates were expired.

Surprisingly, there were not expired : Expiry Date: 2019-02-11 11:17:05+00:00 (VALID: 59 days).

After manual nginx reload it worked back again.

So at that point you might think, “oh, he forgot to have his nginx reload after the crontab renewal”… Well, it is not the case :
30 2 * * * /usr/bin/certbot renew --allow-subset-of-names --noninteractive --renew-hook "/bin/systemctl reload nginx" >> /var/log/le-renew.log

(I took this crontab task from a digitalocean tutorial by the way)

Well, maybe the renew worked but the reload nginx didn’t… however if the renew worked, why is my certificate expiring in 59 days and not in 90 days ?

I have no idea how I can have this situation of suddenly having this SSL error (nobody has access to the server except me and I didn’t access to it for a while).

Do you have any idea on how to figure out what caused this problem so that I can make sure it won’t happen again ?


#2

Hi,

What’s your domain name?

What’s your web server?

Did you configture the certificate automatically or manually?

Could you please try to run nginx -T and share us the result?

Thank you


#4

Hi,

I actually mean the uppercase T, which would also print out the virtual hosts configuration and would allow us to help you debug.

By the way, this is moved to #server-config.

Thank you


#5

Hello,

Well, uppercase T gives:

~# nginx -T
~# nginx: invalid option: "T"

But as I said everything worked great untill today, and nobody accessed the server for more than several weeks.


#6

Hi,

Do you mind to dig into virtual host files and see if the https virtual host (your domain that have questions) referred to a wrong SSL certificate path?

Thank you


#7

Yes, I’ve just done that. Within nginx config I have :

include snippets/ssl-mydomain.com.conf;
include snippets/ssl-params.conf; 

And within : ssl-mydomain.com.conf:

ssl_certificate /etc/letsencrypt/live/mydomain.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/mydomain.com/privkey.pem;

(I replaced domain name)

And the output of certbot certificates is:

-------------------------------------------------------------------------------
Found the following certs:
    (...)
    Certificate Path: /etc/letsencrypt/live/mydomain.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/mydomain.com/privkey.pem
-------------------------------------------------------------------------------

Actually the certfile linked behind is cert9.pem so it’s not the first time that I renew and I has been working so far.


#8

The current expiration date being 59 days ahead, it looks to me that actually no renewal was done so far.
But if the certbot renew failed, then why did my server go down, and up again after reload nginx ?
Also why does certbot renew --dry-run gives no error ?


#9

Hi,

If your browser claimed that the certificate for your API domain is expired, your browser probably have a old cache… (Or i’m connecting to the wrong server)

When i entered your website, your certificate is active and valid.
That’s the same status in OpenSSL.

C:\Users\stevenzhu>openssl s_client -connect api.mydomain.com:443 -status -ct -CAfile "C:\Program Files\Common Files\SSL\cacert.pem" | openssl x509 -noout -text
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN = api.mydomain.com
verify return:1
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            03:f6:87:ae:f1:63:7a:cc:c9:9c:73:2a:48:a0:9d:43:ef:69
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
        Validity
            Not Before: Nov 13 11:17:05 2018 GMT
            Not After : Feb 11 11:17:05 2019 GMT
        Subject: CN = api.mydomain.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:aa:e6:e0:eb:9f:e5:08:25:55:c2:00:bd:71:12:
                    b0:76:80:f3:ac:de:b5:6f:7b:0b:13:d9:20:18:4f:
                    b5:e7:b0:c3:6d:a6:6d:8d:cf:c9:3f:34:5e:03:6a:
                    5b:49:ee:8e:13:e2:22:66:d7:9c:ac:ec:34:c3:4b:
                    34:fe:18:ee:af:6b:70:7f:95:74:5b:1f:22:b7:2f:
                    77:f3:f2:86:fc:f3:99:f3:02:9c:16:11:8b:43:f1:
                    ab:0c:8b:29:00:7f:a3:cc:7a:fc:98:5d:7d:bb:bf:
                    4d:35:54:da:6a:6d:5b:de:b0:17:97:8b:38:ba:8e:
                    4e:e5:df:40:0b:a2:9f:7f:64:c7:72:bd:20:3e:26:
                    2b:71:be:ce:22:c1:62:a8:a4:b7:37:bd:01:08:ec:
                    08:38:23:35:72:82:e0:4f:37:39:a5:b3:58:f3:ba:
                    f6:73:c7:6c:ed:da:5b:e7:35:bd:7d:d4:90:12:c8:
                    22:89:be:5c:13:3f:a6:eb:c0:5a:56:27:77:87:90:
                    0e:e6:87:cc:ff:86:3e:17:a0:d1:7c:63:1a:36:b1:
                    2e:77:2c:cf:a0:72:2a:ee:86:2c:7a:59:14:b5:1a:
                    26:05:94:97:b4:90:98:bd:de:ef:61:8b:1f:0a:db:
                    5d:d5:31:5a:62:40:27:66:87:69:9f:ff:2b:18:d6:
                    bc:f5
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier:
                B8:1A:EC:B4:7D:BD:0D:8E:18:78:9C:4F:93:16:13:2C:5A:40:40:9C
            X509v3 Authority Key Identifier:
                keyid:A8:4A:6A:63:04:7D:DD:BA:E6:D1:39:B7:A6:45:65:EF:F3:A8:EC:A1

            Authority Information Access:
                OCSP - URI:http://ocsp.int-x3.letsencrypt.org
                CA Issuers - URI:http://cert.int-x3.letsencrypt.org/

            X509v3 Subject Alternative Name:
                DNS:api.mydomain.com, DNS:mydomain.com
            X509v3 Certificate Policies:
                Policy: 2.23.140.1.2.1
                Policy: 1.3.6.1.4.1.44947.1.1.1
                  CPS: http://cps.letsencrypt.org

            CT Precertificate SCTs:
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : E2:69:4B:AE:26:E8:E9:40:09:E8:86:1B:B6:3B:83:D4:
                                3E:E7:FE:74:88:FB:A4:8F:28:93:01:9D:DD:F1:DB:FE
                    Timestamp : Nov 13 12:17:05.037 2018 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:46:02:21:00:E1:E5:1F:9F:05:56:07:F4:06:10:C5:
                                9F:64:E9:4B:FB:2F:84:D7:61:89:97:C5:14:94:02:C7:
                                A9:E9:2D:09:F7:02:21:00:B7:3B:26:D7:04:4B:59:59:
                                2A:80:10:2D:CC:49:DA:B7:60:01:6C:0A:28:AB:1F:46:
                                41:F8:12:F7:B6:7F:F1:AE
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : 29:3C:51:96:54:C8:39:65:BA:AA:50:FC:58:07:D4:B7:
                                6F:BF:58:7A:29:72:DC:A4:C3:0C:F4:E5:45:47:F4:78
                    Timestamp : Nov 13 12:17:05.139 2018 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:46:02:21:00:DA:E5:EB:78:31:F8:2F:42:34:7D:97:
                                35:4E:D2:7D:CE:8F:A9:FB:5B:F8:50:C6:BD:EC:79:AB:
                                94:83:49:94:5A:02:21:00:FC:B4:52:A7:59:58:CC:A0:
                                A6:62:DC:B4:5A:3E:68:55:7F:C1:29:94:8D:1E:0F:EB:
                                E4:17:68:B0:83:31:79:27
    Signature Algorithm: sha256WithRSAEncryption
         80:58:a3:86:a0:f0:e5:7c:0f:05:ba:06:26:a1:2a:65:f4:52:
         01:92:ee:ef:20:ea:ec:e7:c2:4f:2b:1b:b8:ec:67:7d:e0:06:
         6c:05:da:7f:97:78:74:ad:8f:f2:a0:eb:55:cc:b5:80:92:5e:
         98:3f:59:e8:7a:52:99:fe:8b:84:14:a4:cc:2b:61:28:17:0d:
         df:e7:1d:5b:44:67:b7:a1:5e:17:45:b0:74:59:84:09:5e:ed:
         e0:ab:ea:20:ef:f0:f3:66:06:a6:79:6d:e5:55:88:ed:24:b5:
         20:f0:49:6d:e7:50:b9:ad:a5:8a:82:3c:3b:17:01:88:57:9f:
         e3:ce:66:70:ec:52:46:48:6d:f7:69:48:9d:93:a9:5f:3c:88:
         4b:0f:5c:cb:5c:a0:ac:1c:b4:26:19:4f:85:ff:27:20:2a:1c:
         dd:a2:f6:0d:a0:cd:da:26:c5:a6:7c:d3:f2:1d:83:a5:3b:42:
         18:f1:62:46:3c:b9:ff:6f:75:a9:8c:6a:5e:af:1f:8b:66:f2:
         2d:d0:f5:df:a4:3f:06:c2:6e:2a:1b:7c:31:90:f5:7f:a9:a8:
         3b:88:5d:4f:4a:55:27:da:56:53:be:58:67:0e:92:95:79:ed:
         79:9e:3a:d1:74:2a:e7:85:ab:97:92:2e:0b:64:84:6a:20:07:
         8e:08:db:ec

Thank you


#10

As stated in the first post, the certificate issue was solved after I systemctl reload nginx.

So I am aware that it is currently valid.

Also it’s not my computer’s cache only because I had two other different people on the globe reporting the same error at the same time.

Let me explain again:
If certbot renew succeeded (and problem was nginx didn’t reload), then why do I still have 59 days left instead of 90 ?
If certbot renew failed or didn’t happen, why do --dry-run gives no error, and why did my my webserver suddenly had ssl error ?


#11

Sorry, I missed this…

I’m sure that the renew does not happen, because normally certbot would renew 30 days in advance…

The renew does not happen because it’s “not the time”, not because some error happened.

Do you happen to have a screenshot / time of that error message?

Thank you


#12

Oh, I see, even after 30 days, it’s still too early to renew ! Well that’s even more mysterious then…
To summarize the facts :

  • Certificates were not close to expiration, everything were fine, nobody accessed the server
  • Despite that, suddenly I had ERR_CERT_DATE_INVALID on chrome, apps using it didn’t work either, I even remember Charles error stating something like “handshake failed” or something.
  • I restart nginx, works again

What did happen ?

(by the way can you replace my domain name from your above console output message please?)


#13

Maybe you have a different automated task that’s also running certbot renew, but without the hook? Like a systemd timer or another scheduled task created by your OS Certbot package?


#14

Right, I think we are getting closer. After thinking through it a lot here’s what I think happened:

  1. As letsencrypt accepts renewal 30 days before expiration, last renewal occured 30 days ago (now 60 days left).
  2. But for some reason the --renew-hook didn’t fire, so nginx didn’t reload
  3. nginx was using the old certificate for 30 days and today it expired

So problem is narrowed down to "renew hook did not happen after successful renewal".

When I look the renewal logs stated in my crontab above, it is full of The following certs are not due for renewal yet messages, but no successful messages ever, despite it successfully renewed many times.

So question is why is the --renew-hook not firing when renew is successful ?

Server : Debian 8.3
Certbot version : 0.10.2

EDIT
I see that systemctl list-timers outputs certbot.timer… So as you said, somehow I already had a periodic task running through systemd timer … ! How do I fix that ?


#15

Is it just me or is there no --renew-hook option? My certbot client (0.29.1 as of this message) has a --deploy-hook option (which I use, and it works) but no --renew-hook.


#16

The --renew-hook option was also deprecated but it’s definitely present in Certbot 0.10.2. For newer versions of Certbot, the recommended option is --deploy-hook.


#17

You could add the hook to your /etc/letsencrypt/renewal/mydomain.com.conf file in the [renewalparams] section at the bottom. Then it doesn’t have to be specified as a command line option to certbot renew because Certbot should always detect it automatically.

Otherwise, there should be a file under /etc/systemd/system somewhere that you could delete if you don’t want that timer to run at all. This has been a tricky issue about OS packaging because different OS packagers have different intentions about how to make Certbot perform automated renewal.


#18

How do you keep time sync on your system?


#19

I don’t know, I never thought about that. Is there anything particular to do / know about server time ?


#20

Time sync requires programs that may be specific to your O/S.
But in general, I always sync with NTP.ORG
They keep updated lists of time servers throughout the globe.


#21

Most Linux distributions now use systemd to do time sync for you. You can check its status using:

timedatectl

Example output:

      Local time: Thu 2018-12-13 23:30:20 CET
  Universal time: Thu 2018-12-13 22:30:20 UTC
        RTC time: Thu 2018-12-13 22:30:20
      Time zone: Europe/Paris (CET, +0100)
Network time on: yes
NTP synchronized: yes
RTC in local TZ: no