Trouble renewing expired certs

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:
gs615-icesat-pz.gsfc.nasa.gov
icesat-2.gsfc.nasa.gov
icesat.gsfc.nasa.gov

I ran this command: certbot renew --preferred-chain "ISRG Root X1"

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/gs615-icesat-pz.gsfc.nasa.gov-0001.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Renewing an existing certificate for gs615-icesat-pz.gsfc.nasa.gov

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
  Domain: gs615-icesat-pz.gsfc.nasa.gov
  Type:   connection
  Detail: 129.164.142.40: Fetching https://gs615-icesat-pz.gsfc.nasa.gov/.well-known/acme-challenge/WhogDE9IKDFyY4DtXf7eHBHysoWYWADOR7irLJDuUk8: Timeout during connect (likely firewall problem)

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

Failed to renew certificate gs615-icesat-pz.gsfc.nasa.gov-0001 with error: Some challenges have failed.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/icesat-2.gsfc.nasa.gov-0001.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Renewing an existing certificate for icesat-2.gsfc.nasa.gov

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
  Domain: icesat-2.gsfc.nasa.gov
  Type:   connection
  Detail: 129.164.142.40: Fetching https://icesat-2.gsfc.nasa.gov/.well-known/acme-challenge/-4CHOw7wWL-4JvtGCkU9gE5_Tmfils44tQWSrq30XQ4: Timeout during connect (likely firewall problem)

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

Failed to renew certificate icesat-2.gsfc.nasa.gov-0001 with error: Some challenges have failed.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/icesat.gsfc.nasa.gov.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Renewing an existing certificate for icesat.gsfc.nasa.gov

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
  Domain: icesat.gsfc.nasa.gov
  Type:   connection
  Detail: 129.164.142.40: Fetching https://icesat.gsfc.nasa.gov/.well-known/acme-challenge/1TTnbCSKxrIMFo31pgLyPYLJylxUpeoIPxJJoiv9FT4: Timeout during connect (likely firewall problem)

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

Failed to renew certificate icesat.gsfc.nasa.gov with error: Some challenges have failed.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
All renewals failed. The following certificates could not be renewed:
  /etc/letsencrypt/live/gs615-icesat-pz.gsfc.nasa.gov-0001/fullchain.pem (failure)
  /etc/letsencrypt/live/icesat-2.gsfc.nasa.gov-0001/fullchain.pem (failure)
  /etc/letsencrypt/live/icesat.gsfc.nasa.gov/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
3 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

My web server is (include version): Apache/2.4.41 (Ubuntu)

The operating system my web server runs on is (include version): Ubuntu 20.04.5 LTS

My hosting provider, if applicable, is: NA

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 1.32.2

Additional Info: The server is behind a squid proxy.

1 Like
2

Hmmm. Well, the IPv6 address in your AAAA record isn't allowing connections.

The Let's Encrypt servers favor IPv6 but the IP shown in the error is your IPv4 address. For some timeouts to IPv6 the LE servers retry with IPv4 so maybe that is happening here.

I don't have good explanation right now why IPv4 would show in the failure message. But, if you have an AAAA record it should be a functional IP address. So, that's one thing to look at. Also use Let's Debug test site to see this (link here)

curl -I6 -m6 gs615-icesat-pz.gsfc.nasa.gov
curl: (28) Failed to connect to gs615-icesat-pz.gsfc.nasa.gov port 80 after 3002 ms: Connection timed out

curl -I4 -m6 gs615-icesat-pz.gsfc.nasa.gov
HTTP/1.1 301 Moved Permanently
Server: Apache
Location: https://gs615-icesat-pz.gsfc.nasa.gov/
6 Likes
3

Thanks. I've got to run now, but will look into this later this evening.

4 Likes
4

That looks like something is blocking the LE challenge requests.
But that must be a real specific block list.
@MikeMcQ and my IPs are able to reach your system [although only via IPv4].
Not sure why the IPv6 is not getting through, but that is not the reason for the failure.

5 Likes
5

It's an interesting quirk.

If they got IPv6 working then Let's Encrypt may just start fully working. Yet, an as-yet undefined issue of certain IPv4 requests being blocked would go unexplained. LE is only trying IPv4 because IPv6 failed.

That is, the IPv4 block wouldn't continue to affect LE but might affect something else.

5 Likes
6

Well, I've enabled the IPv6 on the interface. Apparently an automatic security configuration set a kernel parameter to disable IPv6. But when I run the same certbot command (with --dry-run this time) I get the same output.

Can you verify that you can now see my IPv6 address? I'm not really up to speed on using v6.

1 Like
7

Let's Debug is showing ERRORs for all 3 domain names

  1. https://letsdebug.net/gs615-icesat-pz.gsfc.nasa.gov/1333829
  2. https://letsdebug.net/icesat-2.gsfc.nasa.gov/1333830
  3. https://letsdebug.net/icesat.gsfc.nasa.gov/1333832

Best Practice - Keep Port 80 Open for all IP Addresses; that is both IPv4 and IPv6 Addresses.

2 Likes
8

When debugging connectivity issues like this or proxypass configurations, I like to do the following:

  • serve some URL(s) on the host under /.well-known/acme-challenge
  • use a remote connection to access the URL(s)

Then I iterate on the server/gateway configs until the URLs are correctly routed.

I open sourced a small python script that can act as a fake server, but it requires the Pyramid package: peter_sslers/fake_server.py at main · aptise/peter_sslers · GitHub

In your case, traceroute is showing me that a connection hits the initial NASA gateway ip (192.150.30.70 is allocated to NASA via whois.arin.net), but the latter hops show no info. A similar traceroute situation happens with gsfc.nasa.gov, which is also unresponsive to web requests. This suggests to me that you may have some internal firewalls/gateways that are getting in the way.

Edit: A lot of things could cause the above, but this sort of network activity where a subdomain works while the parent doesn't, and traceroute/ping are entirely unhelpful, are often caused by some firewall/gateway in the route that is intercepting the traffic to /.well-known/acme-challenge and not relaying it upstream.

4 Likes
9

I can see that domain on both IPv4 and IPv6 right now. And, re-running Let's Debug shows an OK message now too (contrary to what Bruce saw just a bit ago).

4 Likes
10

I must just check things at wrong times. :thinking:

3 Likes
11

Yeah, sorry, rebooted a couple times in the process of getting the IPv6 address activated. The debug site now shows all URLs OK, but the certbot command is now timing out on each:

# certbot --dry-run renew --preferred-chain "ISRG Root X1"
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/gs615-icesat-pz.gsfc.nasa.gov-0001.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Failed to renew certificate gs615-icesat-pz.gsfc.nasa.gov-0001 with error: HTTPSConnectionPool(host='acme-staging-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f619d37a820>, 'Connection to acme-staging-v02.api.letsencrypt.org timed out. (connect timeout=45)'))

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/icesat-2.gsfc.nasa.gov-0001.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Failed to renew certificate icesat-2.gsfc.nasa.gov-0001 with error: HTTPSConnectionPool(host='acme-staging-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f619d3754c0>, 'Connection to acme-staging-v02.api.letsencrypt.org timed out. (connect timeout=45)'))

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/icesat.gsfc.nasa.gov.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Failed to renew certificate icesat.gsfc.nasa.gov with error: HTTPSConnectionPool(host='acme-staging-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f619d3d8a60>, 'Connection to acme-staging-v02.api.letsencrypt.org timed out. (connect timeout=45)'))

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
All simulated renewals failed. The following certificates could not be renewed:
  /etc/letsencrypt/live/gs615-icesat-pz.gsfc.nasa.gov-0001/fullchain.pem (failure)
  /etc/letsencrypt/live/icesat-2.gsfc.nasa.gov-0001/fullchain.pem (failure)
  /etc/letsencrypt/live/icesat.gsfc.nasa.gov/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
3 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

The .../.well-known/.acme-challenge directory exists under each of the corresponding webroots, if that matters.

1 Like
12

Here is what I presently see with curl

>curl -4 -Ii http://gs615-icesat-pz.gsfc.nasa.gov/.well-known/acme-challenge/testfile
HTTP/1.1 301 Moved Permanently
Date: Thu, 12 Jan 2023 20:15:56 GMT
Server: Apache
Location: https://gs615-icesat-pz.gsfc.nasa.gov/.well-known/acme-challenge/testfile
Content-Type: text/html; charset=iso-8859-1

>curl -6 -Ii http://gs615-icesat-pz.gsfc.nasa.gov/.well-known/acme-challenge/testfile
HTTP/1.1 301 Moved Permanently
Date: Thu, 12 Jan 2023 20:16:05 GMT
Server: Apache
Location: https://gs615-icesat-pz.gsfc.nasa.gov/.well-known/acme-challenge/testfile
Content-Type: text/html; charset=iso-8859-1

>curl -4 -Ii http://icesat-2.gsfc.nasa.gov/.well-known/acme-challenge/testfile
HTTP/1.1 301 Moved Permanently
Date: Thu, 12 Jan 2023 20:16:31 GMT
Server: Apache
Location: https://icesat-2.gsfc.nasa.gov/.well-known/acme-challenge/testfile
Content-Type: text/html; charset=iso-8859-1

>curl -6 -Ii http://icesat-2.gsfc.nasa.gov/.well-known/acme-challenge/testfile
HTTP/1.1 301 Moved Permanently
Date: Thu, 12 Jan 2023 20:16:39 GMT
Server: Apache
Location: https://icesat-2.gsfc.nasa.gov/.well-known/acme-challenge/testfile
Content-Type: text/html; charset=iso-8859-1

>curl -4 -Ii http://icesat.gsfc.nasa.gov/.well-known/acme-challenge/testfile
HTTP/1.1 301 Moved Permanently
Date: Thu, 12 Jan 2023 20:17:02 GMT
Server: Apache
Location: https://icesat.gsfc.nasa.gov/.well-known/acme-challenge/testfile
Content-Type: text/html; charset=iso-8859-1

>curl -6 -Ii http://icesat.gsfc.nasa.gov/.well-known/acme-challenge/testfile
HTTP/1.1 301 Moved Permanently
Date: Thu, 12 Jan 2023 20:17:08 GMT
Server: Apache
Location: https://icesat.gsfc.nasa.gov/.well-known/acme-challenge/testfile
Content-Type: text/html; charset=iso-8859-1

Edit and with the redirect for the first one.

>curl -4 -k -Ii https://gs615-icesat-pz.gsfc.nasa.gov/.well-known/acme-challenge/testfile
HTTP/1.1 404 Not Found
Date: Thu, 12 Jan 2023 20:20:17 GMT
Server: Apache
Strict-Transport-Security: max-age=63072000; includeSubdomains;
Content-Type: text/html; charset=iso-8859-1

>curl -6 -k -Ii https://gs615-icesat-pz.gsfc.nasa.gov/.well-known/acme-challenge/testfile
HTTP/1.1 404 Not Found
Date: Thu, 12 Jan 2023 20:20:28 GMT
Server: Apache
Strict-Transport-Security: max-age=63072000; includeSubdomains;
Content-Type: text/html; charset=iso-8859-1

>curl -4 -k -Ii https://gs615-icesat-pz.gsfc.nasa.gov/.well-known/acme-challenge/testfile -A "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
HTTP/1.1 404 Not Found
Date: Thu, 12 Jan 2023 20:20:49 GMT
Server: Apache
Strict-Transport-Security: max-age=63072000; includeSubdomains;
Content-Type: text/html; charset=iso-8859-1

>curl -6 -k -Ii https://gs615-icesat-pz.gsfc.nasa.gov/.well-known/acme-challenge/testfile -A "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
HTTP/1.1 404 Not Found
Date: Thu, 12 Jan 2023 20:20:53 GMT
Server: Apache
Strict-Transport-Security: max-age=63072000; includeSubdomains;
Content-Type: text/html; charset=iso-8859-1
2 Likes
13

Put a test file in there, like /.well-known/.acme-challenge/test.txt, and set the contents to be a uniquely identifiable string in it - such as the domain or server name. Then try to load that file from a connection outside of your network.

3 Likes
14

OK, just did that, and I can reach them from my mobile cell network.

3 Likes
15

Sorry, meant to reply to jvanasco. The files I created are "test.txt" with the URL in each.

BTW, the certbot errors state

"Max retries exceeded with url: /directory (Caused by ConnectTimeoutError(...."

Is it literally looking for a directory named "directory", or is that in a generic sense?

3 Likes
16

That is a problem outbound from your server talking to the Let's Encrypt servers.

What happens with:

curl -4 https://acme-v02.api.letsencrypt.org/directory
curl -6 https://acme-v02.api.letsencrypt.org/directory
4 Likes
17

More specifically, in the ACME protocol an initial request goes to the "/directory" endpoint of the server.

The combination of your posts above suggest to me that you are been suffering from random network connectivity issues – both inbound and outbound. I would reach out to your IT or infrastructure team to see if anything is going on.

4 Likes
18

NEVERMIND! After rebooting to get the IPv6 working, and logging back in, I forgot to set the proxy variables in my bash shell, hence all the timeouts.

I greatly appreciate all the help and apologize for wasting your time on the proxy-related timeouts. The initial cause of the renewal failure does appear to have been the IPv6 outage (MikeMcQ was the first to call that out) even though the site was running fine on just IPv4.

6 Likes
19

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.