Certbot renew failure: ssl handshake certificate_verify_failed

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:
erp.dekalbmetal.com

I ran this command: (on powershell)
certbot renew --dry-run

It produced this output:

PS C:\WINDOWS\system32> certbot renew --dry-run
Saving debug log to C:\Certbot\log\letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing C:\Certbot\renewal\erp.dekalbmetal.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Failed to renew certificate erp.dekalbmetal.com with error: HTTPSConnectionPool(host='acme-staging-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)')))

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
All simulated renewals failed. The following certificates could not be renewed:
  C:\Certbot\live\erp.dekalbmetal.com\fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile C:\Certbot\log\letsencrypt.log or re-run Certbot with -v for more details.

My web server is (include version):
Apache 2.4

The operating system my web server runs on is (include version):
Windows Server 2019 Standard

My hosting provider, if applicable, is:
self hosted

I can login to a root shell on my machine (yes or no, or I don't know):
yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
1.24.0

Certbot renewal has been working on this server for a couple years now but seem to be failing this time around. It's a little trickier to google this problem for a windows configuration with powershell (cmd prompt also available of course). Some results suggested this might be related to the root cert expiration last year, though I can verify that ISRG Root X1 is on the server's trusted root certificates and also previous renews since the expiration have succeeded. I also tried updating certbot but had the same errors before and after.

Any help is appreciated, thank you.

1 Like
2

Hi @RedUp, and welcome to the LE community forum :slight_smile:

Are the outbound HTTPS requests being blocked?
curl -I https://acme-staging-v02.api.letsencrypt.org/directory
wget https://acme-staging-v02.api.letsencrypt.org/directory

7 Likes
3

Does WIndows even have these applications by default?

Anyway, I think it might be something Windows certificate lazy loading related. If you surf to acme-staging-v02.api.letsencrypt.org with a browser (non-Firefox? I dunno if that is required for lazy loading to work..), does that work?

7 Likes
4

The outbound connections seem to be fine. curl output:

PS C:\WINDOWS\system32> curl

cmdlet Invoke-WebRequest at command pipeline position 1
Supply values for the following parameters:
Uri: https://acme-staging-v02.api.letsencrypt.org/directory


StatusCode        : 200
StatusDescription : OK
Content           : {
                      "j7m2IdHuwKE":
                    "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
                      "keyChange": "https://acme-staging-v02.api.letsencrypt.org/acme/key-change",
                      "meta": {
                        "c...
RawContent        : HTTP/1.1 200 OK
                    Connection: keep-alive
                    X-Frame-Options: DENY
                    Strict-Transport-Security: max-age=604800
                    Content-Length: 822
                    Cache-Control: public, max-age=0, no-cache
                    Content-Type: application/js...
Forms             : {}
Headers           : {[Connection, keep-alive], [X-Frame-Options, DENY], [Strict-Transport-Security, max-age=604800],
                    [Content-Length, 822]...}
Images            : {}
InputFields       : {}
Links             : {}
ParsedHtml        : System.__ComObject
RawContentLength  : 822
5

Browsing the url directly in Firefox and Edge both work with no https errors and showing valid cert info. Retrying the renew dry run after that had the same error as before.

6

That's an unusual error "self signed certificate ...". It's great that you are trying --dry-run but for this odd case could you leave that off and try to renew? Let us know what happens.

As an aside, there are 2 curl's on Windows. In powershell curl maps to invoke-webrequest which is the output @RedUp showed above. And, is fine. There is also a curl.exe which can be used at powershell prompt explicitly naming curl.exe so that below works. In native 'cmd' prompt just 'curl' uses the exe.

curl.exe -I 'https://acme-staging-v02.api.letsencrypt.org/directory'
HTTP/1.1 200 OK
Server: nginx
Date: Thu, 07 Jul 2022 12:32:34 GMT
Content-Type: application/json
Content-Length: 822
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
Replay-Nonce: 00024kHtztRk4LvfpJhCUZQGcL3ZZNunHQJdyJFSdGdKjiQ
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
9 Likes
7

Sure thing. I did a non dry run to start and the scheduled task also isn't dry running of course, but I figured while I'm troubleshooting I'd stop hitting prodution. Looks like the same error:

PS C:\WINDOWS\system32> certbot renew
Saving debug log to C:\Certbot\log\letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing C:\Certbot\renewal\erp.dekalbmetal.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Failed to renew certificate erp.dekalbmetal.com with error: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)')))

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
All renewals failed. The following certificates could not be renewed:
  C:\Certbot\live\erp.dekalbmetal.com\fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile C:\Certbot\log\letsencrypt.log or re-run Certbot with -v for more details.

The full log file if it might be any more help: Certbot Renew Log - Pastebin.com

1 Like
8

Hmmm. It is almost like you have a firewall that is rejecting outbound requests to those URLs and replying with its own self-signed cert. Yet, for some reason allowed your test curl to succeed. I know that sounds like bad explanation :slight_smile:

Can you try this command? In powershell prompt enter a q at end of display and press enter

openssl s_client -connect acme-v02.api.letsencrypt.org:443

Edit: Oh, I just realized I may have installed openssl so may not be part of Windows.

10 Likes
9

I found an openssl install for windows at Index of /SSL and ran the command there with the following output. I'm not sure if the error is related to it looking for config in a *nix directory or if it actually suggests what the problem is.

WARNING: can't open config file: /usr/local/ssl/openssl.cnf
OpenSSL> s_client -connect acme-v02.api.letsencrypt.org:443
CONNECTED(000001EC)
depth=1 C = US, O = Let's Encrypt, CN = R3
verify error:num=20:unable to get local issuer certificate
---
Certificate chain
 0 s:/CN=acme-v02.api.letsencrypt.org
   i:/C=US/O=Let's Encrypt/CN=R3
 1 s:/C=US/O=Let's Encrypt/CN=R3
   i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIF/DCCBOSgAwIBAgISA1TYq2j47O+YJF8yazNGcalDMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMjA2MjYwOTM5MzBaFw0yMjA5MjQwOTM5MjlaMCcxJTAjBgNVBAMT
HGFjbWUtdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcwggEiMA0GCSqGSIb3DQEBAQUA
A4IBDwAwggEKAoIBAQDcOttA6byYv12SjTFaieXRQ3NGKh9V0onVrPezvmjFEU5t
tH5sPd7+iRLosqiADS3gKzlpWqL5QQW84RTpiUCReLKmRFvn62+8dup1iX6bA0H1
poNFj89kt9tjMjBgBZwbNFVi4XCLiivJ3dlRlPweHVlECt0yvs5gRGAy2MIRt3jP
y5cuNBJg+RK/cG7HAlfl6FJVdNIKN6xbJ6YCBt3a4e9Ocy13pgDiduZokRzvVZs2
7JyfbzQdITgE+q5wlDJFehCLDOgTrlSihewVACfAjkj4EpDpNF3wyw3mFE0dGdw7
vswIJbKXA/V/9a6lLgGHTzA4YppZ7GwFH1me75BxAgMBAAGjggMVMIIDETAOBgNV
HQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1Ud
EwEB/wQCMAAwHQYDVR0OBBYEFEBrmMh52ZnAhI07YCvOX+bvO+nUMB8GA1UdIwQY
MBaAFBQusxe3WFbLrlAJQOYfr52LFMLGMFUGCCsGAQUFBwEBBEkwRzAhBggrBgEF
BQcwAYYVaHR0cDovL3IzLm8ubGVuY3Iub3JnMCIGCCsGAQUFBzAChhZodHRwOi8v
cjMuaS5sZW5jci5vcmcvMIHjBgNVHREEgdswgdiCHmFjbWUtdjAyLTEuYXBpLmxl
dHNlbmNyeXB0Lm9yZ4IeYWNtZS12MDItMi5hcGkubGV0c2VuY3J5cHQub3Jngh5h
Y21lLXYwMi0zLmFwaS5sZXRzZW5jcnlwdC5vcmeCHmFjbWUtdjAyLTQuYXBpLmxl
dHNlbmNyeXB0Lm9yZ4IeYWNtZS12MDItNS5hcGkubGV0c2VuY3J5cHQub3Jnghxh
Y21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnghhpbmNpZGVudC5sZXRzZW5jcnlw
dC5vcmcwTAYDVR0gBEUwQzAIBgZngQwBAgEwNwYLKwYBBAGC3xMBAQEwKDAmBggr
BgEFBQcCARYaaHR0cDovL2Nwcy5sZXRzZW5jcnlwdC5vcmcwggEFBgorBgEEAdZ5
AgQCBIH2BIHzAPEAdwBByMqx3yJGShDGoToJQodeTjGLGwPr60vHaPCQYpYG9gAA
AYGfmFpzAAAEAwBIMEYCIQCRcCINkwrzNw+z4RSBznemoVKfUck3SSspMBFwtG/O
8QIhAOssn0SLCWa60EQiytVTswbJdijzNS5A2hg7hpgD7CnFAHYARqVV63X6kSAw
taKJafTzfREsQXS+/Um4havy/HD+bUcAAAGBn5hahQAABAMARzBFAiAzoLZQtsQa
Na+EjRZLP9qQP4KXqcUUXyeelrzokaKYaQIhAMuEN+0yNBFwchFi9D7rLDRAdZU2
OAvDxelxIUnexcCoMA0GCSqGSIb3DQEBCwUAA4IBAQBwbh2rwHyEpp7vLjcn2iSd
kfJ3bOq8DmfmI8F4viPPYM+pofGyLax/FceQkBZYL6pqn/mJY12AeG4LpH1IIFGV
ZIm4vXPODjILLDVHUMiL1YqAZP4chen87xrYLzN6vDlSkaGwz6IjAwOcOGYFDwRl
16pM1wP+tB4Kn+3bFXxjY3TqKWeXQ18SrgnsFfqTAf9dQnI7+N4ATNrTQAVKD89I
TyAMWZREd5SEVb+FUczbVNHNU0S8+MyKeLCgWWBe2OBC/M48OBGxPuCX5aZKLIhy
prO1e+MWph8ce4SV/hJSwvVnXX/jOc+mkUo7gIQ4SKoJW4YbKjxHy2Nms0yhQkNX
-----END CERTIFICATE-----
subject=/CN=acme-v02.api.letsencrypt.org
issuer=/C=US/O=Let's Encrypt/CN=R3
---
No client certificate CA names sent
Peer signing digest: SHA256
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 3352 bytes and written 433 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: 28748B1B81AE4A4878860BCAC1DD564E99253FB81BAF20243677BB826DB4943B
    Session-ID-ctx:
    Master-Key: 6CAC9742521BCE79C354CE48FD9F91E16DBDDB380F64439ECCB17C6DE02AAD60A6CF6BBCB9C881A2E4575437C296758C
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1657202514
    Timeout   : 300 (sec)
    Verify return code: 20 (unable to get local issuer certificate)
---
closed
1 Like
10

Thanks. I am puzzled. The certs shown by the openssl are what we should see. And, they are not self-signed. So, I don't understand the cause of the error.

I see the openssl error about "unable to get local issuer" but that might be quirk of that particular openssl not using the Windows CA store properly. So, I'm ignoring that and just looking at the actual certs which are fine (also noting your cur/invoke-webrequest was fine).

Could you have some new kind of firewall, possibly on the server, which is affecting requests from certbot program that would not affect other programs? It would be a new program or new setting since you last got a good cert so in last 3 months.

There is an option that might allow you to renew.
BUT IT COULD BE DANGEROUS.
The self-signed cert error is a sign that something odd is between certbot and the Let's Encrypt server. It could be a MITM attack and ignoring it could create security problems. As a debug aid though, you could try:

cerbot renew --dry-run --no-verify-ssl

@Osiris I don't know python well but is there some debug code that could be easily added to the certbot scripts to display the self-signed cert details? That likely would tell us what is interfering.

9 Likes
11

I think I'd start met looking at the certifi root store. If I read the log correctly, Certbot is using urllib3 for the connection and urllib3 uses certifi as the certificate root store.

@RedUp Could you please run the following command on the command line?:

python -m certifi

It should output the location of the root store used by Certbot. I can't find a way to show the certificates used in the connection using urllib3 unfortunately. A Wireshark dump should do the trick tho :wink:

8 Likes
12

As it turned out there was a firewall issue. I don't control that side of things but my client was also watching this thread and spoke to their IT provider. They'd recently enabled some form of Deep Packet Inspection, which when they disabled it for port 443 the issue cleared up.

Thanks everyone for your help and for teaching me a few things here. I'm not sure on the etiquette if this post makes sense to mark as the solution or if it'd be @MikeMcQ suggesting examining the firewall?

5 Likes
13

Glad it's sorted. Would your client share the vendor or kind of firewall product? This is likely to come up again and it is helpful to be able to give specific advice.

The "Deep Packet Inspection" is a MITM but a friendly one in this case. Well, friendly in the sense of not intending to cause harm even though it was interfering.

As for "solution" it is usually granted to the person who identified the problem. I'm happy to take it but if you want it that's fine too :slight_smile:

9 Likes
14

a.k.a. HTTPS inspection.
They simply need to exclude LE domains from such inspection.
[as they should for Banks and Health care, etc.]

8 Likes
15

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.