Previously working connection to LE now fails- related to recent root cert exp.?

Help
1

My domain is: voyant.com

I ran this command: certbot certonly -n
--authenticator certbot-pdns:auth
-d voyant.com
--agree-tos
--email "...."
--expand

It produced this output:
ConnectionError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)'),))

My web server is (include version): N/A

My hosting provider, if applicable, is: PDNS

I can login to a root shell on my machine (yes or no, or I don't know): YES

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): NO

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 1.7.0

I know the IdenTrust Root CA used by LE expired at the end of Sept. Unsure if this is related to that. I don't seem to get a cert error otherwise, and am otherwise able to reach acme-v02.api.letsencrypt.org

This is a great FOSS community, and I appreciate the volunteer help!

More info bellow.

My cerbot server is not behind FW or proxy. It's on Ubuntu 16.04 VM.

I can ping and hit the SSL port of that server...

$ nmap acme-v02.api.letsencrypt.org -p 443

Starting Nmap 7.01 ( https://nmap.org ) at 2021-10-05 15:16 GMT
Nmap scan report for acme-v02.api.letsencrypt.org (172.65.32.248)
Host is up (0.00098s latency).
Other addresses for acme-v02.api.letsencrypt.org (not scanned): 2606:4700:60:0:f53d:5624:85c7:3a2c
PORT    STATE SERVICE
443/tcp open  https

Nmap done: 1 IP address (1 host up) scanned in 0.60 seconds

$ ping acme-v02.api.letsencrypt.org
PING ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com (172.65.32.248) 56(84) bytes of data.
64 bytes from 172.65.32.248: icmp_seq=1 ttl=58 time=0.781 ms
64 bytes from 172.65.32.248: icmp_seq=2 ttl=58 time=0.818 ms
64 bytes from 172.65.32.248: icmp_seq=3 ttl=58 time=0.806 ms

It does resolve to a different hostname (...pacloudflare.com), but I think that's a non factor.

$ dig +trace acme-v02.api.letsencrypt.org

; <<>> DiG 9.10.3-P4-Ubuntu <<>> +trace acme-v02.api.letsencrypt.org
;; global options: +cmd
.			80600	IN	NS	d.root-servers.net.
.			80600	IN	NS	b.root-servers.net.
.			80600	IN	NS	h.root-servers.net.
.			80600	IN	NS	k.root-servers.net.
.			80600	IN	NS	c.root-servers.net.
.			80600	IN	NS	a.root-servers.net.
.			80600	IN	NS	j.root-servers.net.
.			80600	IN	NS	e.root-servers.net.
.			80600	IN	NS	l.root-servers.net.
.			80600	IN	NS	f.root-servers.net.
.			80600	IN	NS	m.root-servers.net.
.			80600	IN	NS	i.root-servers.net.
.			80600	IN	NS	g.root-servers.net.
;; Received 239 bytes from 209.32.32.32#53(209.32.32.32) in 0 ms

org.			172800	IN	NS	d0.org.afilias-nst.org.
org.			172800	IN	NS	a0.org.afilias-nst.info.
org.			172800	IN	NS	c0.org.afilias-nst.info.
org.			172800	IN	NS	a2.org.afilias-nst.info.
org.			172800	IN	NS	b0.org.afilias-nst.org.
org.			172800	IN	NS	b2.org.afilias-nst.org.
org.			86400	IN	DS	26974 8 2 4FEDE294C53F438A158C41D39489CD78A86BEB0D8A0AEAFF14745C0D 16E1DE32
org.			86400	IN	RRSIG	DS 8 1 86400 20211018050000 20211005040000 14748 . AisUt7GjM6u2BMr10OmvEiCDWiWJpOkClutWfx98f5nmSBbMRm2ksoks 4Res0XfeZJyDUxsFiHNAb95gYKDoL66hloa3oMHvqU1pjtopm4Tos3Ti mKbo8vKgRQ/ulsZVzfUV2nWjrBRizCAtfz6jP51s8icmKI6306HiVHha UlL/tns42TtcqBT1CRx5uRbGdo437EcRbeP8+wKwckymJqWce1eNEHmz UOlJfkr68v7ZCaq+/dawFcIbhKAhcA086nM9x/moxznKYf9YtRRxkNfm cWK8UHCVS7ANdLFNszTFOLEgIfzfZS0G+BwJQjdvS+AcUMDHjym26RtZ iAMCOQ==
;; Received 794 bytes from 198.41.0.4#53(a.root-servers.net) in 9 ms

letsencrypt.org.	86400	IN	NS	vera.ns.cloudflare.com.
letsencrypt.org.	86400	IN	NS	owen.ns.cloudflare.com.
1i870vj5h429vj9pci7ar6e9gki74tr7.org. 86400 IN NSEC3 1 1 10 332539EE7F95C32A 1I885400UG2KGQG7SFLBNJLA7HJET9FC NS SOA RRSIG DNSKEY NSEC3PARAM
1i870vj5h429vj9pci7ar6e9gki74tr7.org. 86400 IN RRSIG NSEC3 8 2 86400 20211026142532 20211005132532 6555 org. k9rAeZKryUbO3bJPke+jBjWlsYe7yUAWJaOUzQPtcAhwyGp7pA6qnyqz 7wKYjs6YbcsLDxIvhUFJGWCDDFLGaoXlU4oEIk55+xNwM0K4I2JzzHHw ik0FVVtnDLFIvJzwQ15ag6uBHwsgHUA3ISUvF4cIz2Y0eqMh5TlQRB0b tZU=
ok5ilot7cjd0kv9uo0defrei5slj1p2f.org. 86400 IN NSEC3 1 1 10 332539EE7F95C32A OK5R0R9H99MNQ10GR83549E2RUTL02N7 NS DS RRSIG
ok5ilot7cjd0kv9uo0defrei5slj1p2f.org. 86400 IN RRSIG NSEC3 8 2 86400 20211022152845 20211001142845 6555 org. H9OxjFrAHWFpkf9iTDgRoUn+8ySKP4ZMT43dbObTaiU/8Ru9Vxde8jtR b0MRnRTO5No+Fk13uZnayzXSJzyFdOVs4tyDFnkV2cshqDmf2Bw0OT76 hLqYwDyOV3Y1Kg1ws04PWQhkiiGcTefoFPEPq9JCUPmNNl0FdravQUFW Uko=
;; Received 613 bytes from 199.19.54.1#53(b0.org.afilias-nst.org) in 41 ms

acme-v02.api.letsencrypt.org. 7200 IN	CNAME	prod.api.letsencrypt.org.
prod.api.letsencrypt.org. 300	IN	CNAME	ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com.
;; Received 139 bytes from 172.64.33.219#53(owen.ns.cloudflare.com) in 3 ms



$ curl -v https://acme-v02.api.letsencrypt.org/directory
*   Trying 172.65.32.248...
* Connected to acme-v02.api.letsencrypt.org (172.65.32.248) port 443 (#0)
* found 156 certificates in /etc/ssl/certs/ca-certificates.crt
* found 624 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_256_GCM_SHA384
* 	 server certificate verification OK
* 	 server certificate status verification SKIPPED
* 	 common name: acme-v02.api.letsencrypt.org (matched)
* 	 server certificate expiration date OK
* 	 server certificate activation date OK
* 	 certificate public key: RSA
* 	 certificate version: #3
* 	 subject: CN=acme-v02.api.letsencrypt.org
* 	 start date: Thu, 30 Sep 2021 00:30:21 GMT
* 	 expire date: Wed, 29 Dec 2021 00:30:20 GMT
* 	 issuer: C=US,O=Let's Encrypt,CN=R3
* 	 compression: NULL
* ALPN, server accepted to use http/1.1
> GET /directory HTTP/1.1
> Host: acme-v02.api.letsencrypt.org
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx
< Date: Tue, 05 Oct 2021 14:36:15 GMT
< Content-Type: application/json
< Content-Length: 658
< Connection: keep-alive
< Cache-Control: public, max-age=0, no-cache
< X-Frame-Options: DENY
< Strict-Transport-Security: max-age=604800
<
{
  "aNeZxt9caaQ": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
  "keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
    "website": "https://letsencrypt.org"
  },
  "newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
  "revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert"
* Connection #0 to host acme-v02.api.letsencrypt.org left intact
}




$ dig acme-v02.api.letsencrypt.org

; <<>> DiG 9.10.3-P4-Ubuntu <<>> acme-v02.api.letsencrypt.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 14610
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;acme-v02.api.letsencrypt.org.	IN	A

;; ANSWER SECTION:
acme-v02.api.letsencrypt.org. 5117 IN	CNAME	prod.api.letsencrypt.org.
prod.api.letsencrypt.org. 300	IN	CNAME	ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com.
ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com. 300 IN A 172.65.32.248

;; Query time: 7 msec
;; SERVER: 209.32.32.32#53(209.32.32.32)
;; WHEN: Tue Oct 05 14:47:38 GMT 2021
;; MSG SIZE  rcvd: 155



$ curl -v https://acme-v02.api.letsencrypt.org/
*   Trying 172.65.32.248...
* Connected to acme-v02.api.letsencrypt.org (172.65.32.248) port 443 (#0)
* found 156 certificates in /etc/ssl/certs/ca-certificates.crt
* found 624 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_256_GCM_SHA384
* 	 server certificate verification OK
* 	 server certificate status verification SKIPPED
* 	 common name: acme-v02.api.letsencrypt.org (matched)
* 	 server certificate expiration date OK
* 	 server certificate activation date OK
* 	 certificate public key: RSA
* 	 certificate version: #3
* 	 subject: CN=acme-v02.api.letsencrypt.org
* 	 start date: Thu, 30 Sep 2021 00:27:02 GMT
* 	 expire date: Wed, 29 Dec 2021 00:27:01 GMT
* 	 issuer: C=US,O=Let's Encrypt,CN=R3
* 	 compression: NULL
* ALPN, server accepted to use http/1.1
> GET / HTTP/1.1
> Host: acme-v02.api.letsencrypt.org
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx
< Date: Tue, 05 Oct 2021 14:47:55 GMT
< Content-Type: text/html
< Content-Length: 2174
< Last-Modified: Wed, 18 Aug 2021 16:36:13 GMT
< Connection: keep-alive
< ETag: "611d36fd-87e"
< X-Frame-Options: DENY
< Strict-Transport-Security: max-age=604800
<
<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <meta http-equiv="X-UA-Compatible" content="IE=edge">
  <meta name="viewport" content=
  "width=device-width, initial-scale=1">

  <title>Boulder: The Let's Encrypt CA</title>
  <link href=
  "//maxcdn.bootstrapcdn.com/bootstrap/3.2.0/css/bootstrap.min.css"
  rel="stylesheet" type="text/css">
  <link href=
  "//maxcdn.bootstrapcdn.com/font-awesome/4.2.0/css/font-awesome.min.css"
  rel="stylesheet" type="text/css">
</head>

<body>
  <div class="container-fluid">
    <div class="row">
      <div class="col-xs-6 text-right">
        <p style="font-size: 90px;">
        <i class="fa fa-barcode"></i></p>
      </div>

      <div class="col-xs-6 text-left">
        <h1>Boulder<br>
        <small>The Let's Encrypt CA</small></h1>
      </div>
    </div>

    <div class="row">
      <div class="col-xs-8 col-xs-offset-2 text-center">
        <h3>This is an <a href="https://github.com/letsencrypt/acme-spec/">ACME</a> Certificate Authority running <a href="https://github.com/letsencrypt/boulder">Boulder</a>.</h3>
        <p>This is a <em>programmatic</em> endpoint, an API for a computer to talk to. You should probably be using a specialized client to utilize the service, and not your web browser. See <a href="https://letsencrypt.org/"><tt>https://letsencrypt.org/</tt></a> for help.</p>
        <p>If you're trying to use this service, note that the starting point, <em>the directory</em>, is available at this URL: <a href="https://acme-v02.api.letsencrypt.org/directory"><tt>https://acme-v02.api.letsencrypt.org/directory</a></tt>.</p>
      </div>
    </div>
    <div class="row">
      <div class="col-xs-4 col-xs-offset-2 text-center">
        <p><a href="https://letsencrypt.status.io" title="Twitter">
          <i class="fa fa-area-chart"></i>
          Service Status (letsencrypt.status.io)
        </a></p>
      </div>
      <div class="col-xs-4 text-center">
        <p><a href="https://twitter.com/letsencrypt" title="Twitter">
          <i class="fa fa-twitter"></i>
          Check with us on Twitter
        </a></p>
      </div>
    </div> <!-- row -->
  </div>


</body>
</html>
* Connection #0 to host acme-v02.api.letsencrypt.org left intact

And the cert

$ dpkg -l ca-certificates
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                      Version           Architecture      Description
+++-=========================-=================-=================-========================================================
ii  ca-certificates           20170717~16.04.2  all               Common CA certificates

$ openssl s_client -connect acme-v02.api.letsencrypt.org:443 -servername acme-v02.api.letsencrypt.org
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = acme-v02.api.letsencrypt.org
verify return:1
---
Certificate chain
 0 s:/CN=acme-v02.api.letsencrypt.org
   i:/C=US/O=Let's Encrypt/CN=R3
 1 s:/C=US/O=Let's Encrypt/CN=R3
   i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
---
Server certificate
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
subject=/CN=acme-v02.api.letsencrypt.org
issuer=/C=US/O=Let's Encrypt/CN=R3
---
No client certificate CA names sent
Peer signing digest: SHA256
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 3329 bytes and written 468 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 3F29BDFA442D3BF2921A4CAD295A525982C9AB93485461679B64B44D81578849
    Session-ID-ctx:
    Master-Key: ...
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1633445297
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---
closed
2

@erglazier I do not have much to add but maybe this will help.

Based on the verification error I was going to say you are missing the ISRG Root X1 from your trusted store - especially noting the older Ubuntu 16.

But, then I saw your openssl command did not have errors so now not sure. Do you by chance have your openssl cfg setup to use an alternate trust store?

FYI, the acme-v02.api.letsencrypt.org cert chain no longer includes the expired DST Root X3. Note the LE websites still do for compatibility with older Androids. So, the chain is just:


Certificate chain
 0 s:/CN=acme-v02.api.letsencrypt.org
   i:/C=US/O=Let's Encrypt/CN=R3
 1 s:/C=US/O=Let's Encrypt/CN=R3
   i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1

For some reason this is not verifying for certbot. Hmmm

What is your openssl version? I am wondering why it worked.

Here is a thread about Ubuntu root store. You are very knowledgeable so maybe something will inspire you :slight_smile:

Post back if not resolved. There is more to look at if need be.

3

Please show the outputs of:
wget --delete-after https://acme-v02.api.letsencrypt.org/directory
curl -Iki4 https://acme-v02.api.letsencrypt.org/directory
curl -Iki6 https://acme-v02.api.letsencrypt.org/directory

I suspect that your IPv6 path is unusable.

4

Thank you Mike! I appreciate your time and feedback! (and I'm not that knowledgeable :))

2 Likes
5

Thanks for the help.

$ wget --delete-after https://acme-v02.api.letsencrypt.org/directory
--2021-10-05 21:47:34--  https://acme-v02.api.letsencrypt.org/directory
Resolving acme-v02.api.letsencrypt.org (acme-v02.api.letsencrypt.org)... 172.65.32.248, 2606:4700:60:0:f53d:5624:85c7:3a2c
Connecting to acme-v02.api.letsencrypt.org (acme-v02.api.letsencrypt.org)|172.65.32.248|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 658 [application/json]
Saving to: 'directory.tmp'

directory.tmp                 100%[===============================================>]     658  --.-KB/s    in 0s

2021-10-05 21:47:34 (43.6 MB/s) - 'directory.tmp' saved [658/658]

Removing directory.tmp.

:~
$ curl -Iki4 https://acme-v02.api.letsencrypt.org/directory
HTTP/1.1 200 OK
Server: nginx
Date: Tue, 05 Oct 2021 21:48:34 GMT
Content-Type: application/json
Content-Length: 658
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
Replay-Nonce: 0002dApt35GhKtnDQGvLbOUTWqK5jYgLlkPGBYGdZzDuwYs
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800


:~
$ curl -Iki6 https://acme-v02.api.letsencrypt.org/directory
curl: (7) Couldn't connect to server

:~
$
1 Like
6

There is your problem:

As I had... predicted:

1 Like
7

@rg305 Rudy, what made you suspect IPv6? I did not see it in nslookup so never questioned it. Just looking to learn. Thanks

Non-authoritative answer:
Name:   voyant.com
Address: 104.199.124.219
8

It's all there: In black and white.

I guess we could have confirmed it within the LE logs.

1 Like
9

Good find, and thanks. I guess, since it was working, and then it was not.. what possibly changed in terms of IPv6? The server itself hasn't had any network changes in terms of enable/disabling IPv6. I guess I am unsure what the fix is, at this point.

1 Like
10

Ask your Internet Service Provider (ISP).

11

I don't have an IPv6 interface up on my cerbot server.

12

Does it use IPv6?
Is it used via IPv6?

13

It does not / is not

14

Then remove the IPv6 address from it.

15

Thanks. I guess I need to brush up on my nmap outputs. Since it did not scan it I never considered it could. Especially when nslookup said was IPv4 only.

16

agreed

17

Sorry, I may not be well versed enough in all of this. I actually did try to ping that address via ip6 (before I opened the ticket). I got this- not really sure what it means.

certbot@ply-certbot01:~
$ ping6 acme-v02.api.letsencrypt.org
connect: Cannot assign requested address

I just checked, and IPv6 is disabled from a server perspective, and the one network interface only has an IPv4 address on it.

I guess a public DNS resolver resolves both the 4 and 6 addresses for acme-v02.api.letsencrypt.org, but that's outside of my scope.

You said remove the IPv6 address from it, but I guess I on't know what "it" is. Does an IPv6 DNS record need to be removed, and if so, I don't control the api.letsencrypt.org domain. I'm guessing I'm way off base.

1 Like
18

@erglazier Let's gather more info. Let's look at these:

curl -4 ifconfig.co
curl -6 ifconfig.co

More importantly, can you upload the letsencrypt log? It will show the IP it tried to connect with. And, lots and lots of other stuff. Best to upload rather than paste it. The log is in /var/log/letsencrypt and they roll so get one showing the error. I am pretty sure of the folder and rolling - my server is down for service right now so can't double check.

1 Like
21

Inbound doesn't have to match (the actual) outbound.

ifconfig | grep -Ei 'add|inet'

I'm starting to doubt the IPv6 thing...
I'd like to see that log too.

22

Thank you for continuing to pursue this, Mike and Rudy.

I will upload the logs too...

$ curl -4 ifconfig.co
137.192.81.7


$ curl -6 ifconfig.co
curl: (7) Couldn't connect to server


$ ifconfig | grep -Ei 'add|inet'
ens192    Link encap:Ethernet  HWaddr 00:50:56:97:51:57
          inet addr:10.7.55.28  Bcast:10.7.55.255  Mask:255.255.254.0
          inet addr:127.0.0.1  Mask:255.0.0.0