Today my server went off due to SSL issue (ERR_CERT_DATE_INVALID).
I use a crontab task (once a day) to automatically renew it so I was quite surprised.
I logged into sever and typed certbot certificates to check if my certificates were expired.
Surprisingly, there were not expired : Expiry Date: 2019-02-11 11:17:05+00:00 (VALID: 59 days).
After manual nginx reload it worked back again.
So at that point you might think, “oh, he forgot to have his nginx reload after the crontab renewal”… Well, it is not the case : 30 2 * * * /usr/bin/certbot renew --allow-subset-of-names --noninteractive --renew-hook "/bin/systemctl reload nginx" >> /var/log/le-renew.log
(I took this crontab task from a digitalocean tutorial by the way)
Well, maybe the renew worked but the reload nginx didn’t… however if the renew worked, why is my certificate expiring in 59 days and not in 90 days ?
I have no idea how I can have this situation of suddenly having this SSL error (nobody has access to the server except me and I didn’t access to it for a while).
Do you have any idea on how to figure out what caused this problem so that I can make sure it won’t happen again ?
Do you mind to dig into virtual host files and see if the https virtual host (your domain that have questions) referred to a wrong SSL certificate path?
-------------------------------------------------------------------------------
Found the following certs:
(...)
Certificate Path: /etc/letsencrypt/live/mydomain.com/fullchain.pem
Private Key Path: /etc/letsencrypt/live/mydomain.com/privkey.pem
-------------------------------------------------------------------------------
Actually the certfile linked behind is cert9.pem so it’s not the first time that I renew and I has been working so far.
The current expiration date being 59 days ahead, it looks to me that actually no renewal was done so far.
But if the certbot renew failed, then why did my server go down, and up again after reload nginx ?
Also why does certbot renew --dry-run gives no error ?
If your browser claimed that the certificate for your API domain is expired, your browser probably have a old cache… (Or i’m connecting to the wrong server)
When i entered your website, your certificate is active and valid.
That’s the same status in OpenSSL.
C:\Users\stevenzhu>openssl s_client -connect api.mydomain.com:443 -status -ct -CAfile "C:\Program Files\Common Files\SSL\cacert.pem" | openssl x509 -noout -text
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN = api.mydomain.com
verify return:1
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
03:f6:87:ae:f1:63:7a:cc:c9:9c:73:2a:48:a0:9d:43:ef:69
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
Validity
Not Before: Nov 13 11:17:05 2018 GMT
Not After : Feb 11 11:17:05 2019 GMT
Subject: CN = api.mydomain.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
Modulus:
00:aa:e6:e0:eb:9f:e5:08:25:55:c2:00:bd:71:12:
b0:76:80:f3:ac:de:b5:6f:7b:0b:13:d9:20:18:4f:
b5:e7:b0:c3:6d:a6:6d:8d:cf:c9:3f:34:5e:03:6a:
5b:49:ee:8e:13:e2:22:66:d7:9c:ac:ec:34:c3:4b:
34:fe:18:ee:af:6b:70:7f:95:74:5b:1f:22:b7:2f:
77:f3:f2:86:fc:f3:99:f3:02:9c:16:11:8b:43:f1:
ab:0c:8b:29:00:7f:a3:cc:7a:fc:98:5d:7d:bb:bf:
4d:35:54:da:6a:6d:5b:de:b0:17:97:8b:38:ba:8e:
4e:e5:df:40:0b:a2:9f:7f:64:c7:72:bd:20:3e:26:
2b:71:be:ce:22:c1:62:a8:a4:b7:37:bd:01:08:ec:
08:38:23:35:72:82:e0:4f:37:39:a5:b3:58:f3:ba:
f6:73:c7:6c:ed:da:5b:e7:35:bd:7d:d4:90:12:c8:
22:89:be:5c:13:3f:a6:eb:c0:5a:56:27:77:87:90:
0e:e6:87:cc:ff:86:3e:17:a0:d1:7c:63:1a:36:b1:
2e:77:2c:cf:a0:72:2a:ee:86:2c:7a:59:14:b5:1a:
26:05:94:97:b4:90:98:bd:de:ef:61:8b:1f:0a:db:
5d:d5:31:5a:62:40:27:66:87:69:9f:ff:2b:18:d6:
bc:f5
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Subject Key Identifier:
B8:1A:EC:B4:7D:BD:0D:8E:18:78:9C:4F:93:16:13:2C:5A:40:40:9C
X509v3 Authority Key Identifier:
keyid:A8:4A:6A:63:04:7D:DD:BA:E6:D1:39:B7:A6:45:65:EF:F3:A8:EC:A1
Authority Information Access:
OCSP - URI:http://ocsp.int-x3.letsencrypt.org
CA Issuers - URI:http://cert.int-x3.letsencrypt.org/
X509v3 Subject Alternative Name:
DNS:api.mydomain.com, DNS:mydomain.com
X509v3 Certificate Policies:
Policy: 2.23.140.1.2.1
Policy: 1.3.6.1.4.1.44947.1.1.1
CPS: http://cps.letsencrypt.org
CT Precertificate SCTs:
Signed Certificate Timestamp:
Version : v1 (0x0)
Log ID : E2:69:4B:AE:26:E8:E9:40:09:E8:86:1B:B6:3B:83:D4:
3E:E7:FE:74:88:FB:A4:8F:28:93:01:9D:DD:F1:DB:FE
Timestamp : Nov 13 12:17:05.037 2018 GMT
Extensions: none
Signature : ecdsa-with-SHA256
30:46:02:21:00:E1:E5:1F:9F:05:56:07:F4:06:10:C5:
9F:64:E9:4B:FB:2F:84:D7:61:89:97:C5:14:94:02:C7:
A9:E9:2D:09:F7:02:21:00:B7:3B:26:D7:04:4B:59:59:
2A:80:10:2D:CC:49:DA:B7:60:01:6C:0A:28:AB:1F:46:
41:F8:12:F7:B6:7F:F1:AE
Signed Certificate Timestamp:
Version : v1 (0x0)
Log ID : 29:3C:51:96:54:C8:39:65:BA:AA:50:FC:58:07:D4:B7:
6F:BF:58:7A:29:72:DC:A4:C3:0C:F4:E5:45:47:F4:78
Timestamp : Nov 13 12:17:05.139 2018 GMT
Extensions: none
Signature : ecdsa-with-SHA256
30:46:02:21:00:DA:E5:EB:78:31:F8:2F:42:34:7D:97:
35:4E:D2:7D:CE:8F:A9:FB:5B:F8:50:C6:BD:EC:79:AB:
94:83:49:94:5A:02:21:00:FC:B4:52:A7:59:58:CC:A0:
A6:62:DC:B4:5A:3E:68:55:7F:C1:29:94:8D:1E:0F:EB:
E4:17:68:B0:83:31:79:27
Signature Algorithm: sha256WithRSAEncryption
80:58:a3:86:a0:f0:e5:7c:0f:05:ba:06:26:a1:2a:65:f4:52:
01:92:ee:ef:20:ea:ec:e7:c2:4f:2b:1b:b8:ec:67:7d:e0:06:
6c:05:da:7f:97:78:74:ad:8f:f2:a0:eb:55:cc:b5:80:92:5e:
98:3f:59:e8:7a:52:99:fe:8b:84:14:a4:cc:2b:61:28:17:0d:
df:e7:1d:5b:44:67:b7:a1:5e:17:45:b0:74:59:84:09:5e:ed:
e0:ab:ea:20:ef:f0:f3:66:06:a6:79:6d:e5:55:88:ed:24:b5:
20:f0:49:6d:e7:50:b9:ad:a5:8a:82:3c:3b:17:01:88:57:9f:
e3:ce:66:70:ec:52:46:48:6d:f7:69:48:9d:93:a9:5f:3c:88:
4b:0f:5c:cb:5c:a0:ac:1c:b4:26:19:4f:85:ff:27:20:2a:1c:
dd:a2:f6:0d:a0:cd:da:26:c5:a6:7c:d3:f2:1d:83:a5:3b:42:
18:f1:62:46:3c:b9:ff:6f:75:a9:8c:6a:5e:af:1f:8b:66:f2:
2d:d0:f5:df:a4:3f:06:c2:6e:2a:1b:7c:31:90:f5:7f:a9:a8:
3b:88:5d:4f:4a:55:27:da:56:53:be:58:67:0e:92:95:79:ed:
79:9e:3a:d1:74:2a:e7:85:ab:97:92:2e:0b:64:84:6a:20:07:
8e:08:db:ec
As stated in the first post, the certificate issue was solved after I systemctl reload nginx.
So I am aware that it is currently valid.
Also it’s not my computer’s cache only because I had two other different people on the globe reporting the same error at the same time.
Let me explain again:
If certbot renew succeeded (and problem was nginx didn’t reload), then why do I still have 59 days left instead of 90 ?
If certbot renew failed or didn’t happen, why do --dry-run gives no error, and why did my my webserver suddenly had ssl error ?
Oh, I see, even after 30 days, it’s still too early to renew ! Well that’s even more mysterious then…
To summarize the facts :
Certificates were not close to expiration, everything were fine, nobody accessed the server
Despite that, suddenly I had ERR_CERT_DATE_INVALID on chrome, apps using it didn’t work either, I even remember Charles error stating something like “handshake failed” or something.
I restart nginx, works again
What did happen ?
(by the way can you replace my domain name from your above console output message please?)
Maybe you have a different automated task that’s also running certbot renew, but without the hook? Like a systemd timer or another scheduled task created by your OS Certbot package?
Right, I think we are getting closer. After thinking through it a lot here’s what I think happened:
As letsencrypt accepts renewal 30 days before expiration, last renewal occured 30 days ago (now 60 days left).
But for some reason the --renew-hook didn’t fire, so nginx didn’t reload
nginx was using the old certificate for 30 days and today it expired
So problem is narrowed down to "renew hook did not happen after successful renewal".
When I look the renewal logs stated in my crontab above, it is full of The following certs are not due for renewal yet messages, but no successful messages ever, despite it successfully renewed many times.
So question is why is the --renew-hook not firing when renew is successful ?
Server : Debian 8.3
Certbot version : 0.10.2
EDIT
I see that systemctl list-timers outputs certbot.timer… So as you said, somehow I already had a periodic task running through systemd timer … ! How do I fix that ?
Is it just me or is there no --renew-hook option? My certbot client (0.29.1 as of this message) has a --deploy-hook option (which I use, and it works) but no --renew-hook.
The --renew-hook option was also deprecated but it’s definitely present in Certbot 0.10.2. For newer versions of Certbot, the recommended option is --deploy-hook.
You could add the hook to your /etc/letsencrypt/renewal/mydomain.com.conf file in the [renewalparams] section at the bottom. Then it doesn't have to be specified as a command line option to certbot renew because Certbot should always detect it automatically.
Otherwise, there should be a file under /etc/systemd/system somewhere that you could delete if you don't want that timer to run at all. This has been a tricky issue about OS packaging because different OS packagers have different intentions about how to make Certbot perform automated renewal.
Time sync requires programs that may be specific to your O/S.
But in general, I always sync with NTP.ORG
They keep updated lists of time servers throughout the globe.
Most Linux distributions now use systemd to do time sync for you. You can check its status using:
timedatectl
Example output:
Local time: Thu 2018-12-13 23:30:20 CET
Universal time: Thu 2018-12-13 22:30:20 UTC
RTC time: Thu 2018-12-13 22:30:20
Time zone: Europe/Paris (CET, +0100)
Network time on: yes
NTP synchronized: yes
RTC in local TZ: no