Erroneous connection timeouts when renewing TLS cert

lordvadr · February 8, 2024, 9:03pm

I ran this command: /bin/certbot renew --must-staple --post-hook /sbin/keyfix

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/cjoster.com.conf

Renewing an existing certificate for cjoster.com and 5 more domains

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
Domain: cjoster.com
Type: connection
Detail: 35.163.241.194: Fetching https://cjoster.com/.well-known/acme-challenge/HlNAT1tUSaaAl8801JUkGHSE2604AhRHtyP9yUR-ZMY: Timeout during connect (likely firewall problem)

Domain: mail.cjoster.com
Type: connection
Detail: 35.163.241.194: Fetching https://mail.cjoster.com/.well-known/acme-challenge/ILndFmtHsxEsV0OqLyGTBZL4rijAkKNbe-TVMxQIbnM: Timeout during connect (likely firewall problem)

Domain: mail.osterfam.com
Type: connection
Detail: 35.163.241.194: Fetching https://mail.osterfam.com/.well-known/acme-challenge/Ur5BtbBmYxzIJtOjQSBImRqsnwtfRqpPv-q70D-CnTM: Timeout during connect (likely firewall problem)

Domain: osterfam.com
Type: connection
Detail: 35.163.241.194: Fetching https://osterfam.com/.well-known/acme-challenge/jW3awIaYMY-Hm4bO_96VMz_FEMNG06-mgo7GgIQZatI: Timeout during connect (likely firewall problem)

Domain: www.cjoster.com
Type: connection
Detail: 35.163.241.194: Fetching https://www.cjoster.com/.well-known/acme-challenge/N33MyZE99MC04sCgL3zaPJ0SN9wczCLxd3WuCfZyBEg: Timeout during connect (likely firewall problem)

Domain: www.osterfam.com
Type: connection
Detail: 35.163.241.194: Fetching https://www.osterfam.com/.well-known/acme-challenge/ni7XKMYQV79WpMsjjM1HDoj_r_5AK0N98pD4j-xtfsg: Timeout during connect (likely firewall problem)

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

Failed to renew certificate cjoster.com with error: Some challenges have failed.

All renewals failed. The following certificates could not be renewed:
/etc/letsencrypt/live/cjoster.com/fullchain.pem (failure)

Hook 'post-hook' ran with error output:

cp -f /etc/letsencrypt/live/cjoster.com/privkey.pem /etc/pki/tls/private/cjoster.com.key
chmod 400 /etc/pki/tls/private/cjoster.com.key
cp -f /etc/letsencrypt/live/cjoster.com/fullchain.pem /etc/pki/tls/certs/cjoster.com.pem
cat /etc/letsencrypt/live/cjoster.com/privkey.pem
chown root.mail /etc/pki/cyrus-imapd/privkey.pem
chmod 640 /etc/pki/cyrus-imapd/privkey.pem
cat /etc/letsencrypt/live/cjoster.com/fullchain.pem
cat /etc/letsencrypt/live/cjoster.com/cert.pem
systemctl restart sendmail
systemctl restart cyrus-imapd
1 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

My web server is (include version): Apache, httpd-2.4.57-5.el9.x86_64

The operating system my web server runs on is (include version):

[root@cjoster lordvadr]# cat /etc/os-release
NAME="Rocky Linux"
VERSION="9.1 (Blue Onyx)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.1"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Rocky Linux 9.1 (Blue Onyx)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:9::baseos"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-9"
ROCKY_SUPPORT_PRODUCT_VERSION="9.1"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.1"

My hosting provider, if applicable, is: AWS EC2

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

[root@cjoster lordvadr]# certbot --version
certbot 2.6.0

[root@cjoster lordvadr]# !nmap
nmap -Pn -p80,443 cjoster.com
Starting Nmap 7.92 ( https://nmap.org ) at 2024-02-08 21:02 UTC
Nmap scan report for cjoster.com (35.163.241.194)
Host is up (0.00022s latency).
Other addresses for cjoster.com (not scanned): 2600:1f14:d0:a000:fc80:95a1:a7a9:3c65

PORT STATE SERVICE
80/tcp open http
443/tcp open https

(from external host)
[cjo@mcauliffe src]$ !nmap
nmap -Pn -p80,443 cjoster.com
Starting Nmap 7.93 ( https://nmap.org ) at 2024-02-08 15:02 CST
Nmap scan report for cjoster.com (35.163.241.194)
Host is up (0.061s latency).
Other addresses for cjoster.com (not scanned): 2600:1f14:d0:a000:fc80:95a1:a7a9:3c65

PORT STATE SERVICE
80/tcp open http
443/tcp open https

Nmap done: 1 IP address (1 host up) scanned in 0.29 seconds

Bruce5051 · February 8, 2024, 9:16pm

Hello @lordvadr, welcome to the Let's Encrypt community.

Using the online tool Let's Debug yields these results https://letsdebug.net/cjoster.com/1797379

AAAANotWorking
ERROR
cjoster.com has an AAAA (IPv6) record (2600:1f14:d0:a000:fc80:95a1:a7a9:3c65) but a test request to this address over port 80 did not succeed. Your web server must have at least one working IPv4 or IPv6 address. You should either ensure that validation requests to this domain succeed over IPv6, or remove its AAAA record.
A timeout was experienced while communicating with cjoster.com/2600:1f14:d0:a000:fc80:95a1:a7a9:3c65: Get "http://cjoster.com/.well-known/acme-challenge/letsdebug-test": dial tcp [2600:1f14:d0:a000:fc80:95a1:a7a9:3c65]:80: i/o timeout

Trace:
@0ms: Making a request to http://cjoster.com/.well-known/acme-challenge/letsdebug-test (using initial IP 2600:1f14:d0:a000:fc80:95a1:a7a9:3c65)
@0ms: Dialing 2600:1f14:d0:a000:fc80:95a1:a7a9:3c65
@10001ms: Experienced error: dial tcp [2600:1f14:d0:a000:fc80:95a1:a7a9:3c65]:80: i/o timeout

IssueFromLetsEncrypt
ERROR
A test authorization for cjoster.com to the Let's Encrypt staging service has revealed issues that may prevent any certificate for this domain being issued.
35.163.241.194: Fetching https://cjoster.com/.well-known/acme-challenge/kd_vRqeaE9KZZjEcq_vFFuZ0JDhoW5u7ac67-S3pgMM: Timeout during connect (likely firewall problem)

Your DNS has both IPv4 & IPv6 Addresses for the domain name, that is nice.
However both IPv4 & IPv6 Addresses need to respond the same. Yet they do not.

IPv6 shows ports 80 & 443 are showing FILTERED..

>nmap -6 -Pn -p80,443 cjoster.com
Starting Nmap 7.94 ( https://nmap.org ) at 2024-02-08 21:14 UTC
Nmap scan report for cjoster.com (2600:1f14:d0:a000:fc80:95a1:a7a9:3c65)
Host is up.
Other addresses for cjoster.com (not scanned): 35.163.241.194

PORT    STATE    SERVICE
80/tcp  filtered http
443/tcp filtered https

Nmap done: 1 IP address (1 host up) scanned in 3.43 seconds

IPv4 shows ports 80 & 443 are showing OPEN.

>nmap -4 -Pn -p80,443 cjoster.com
Starting Nmap 7.94 ( https://nmap.org ) at 2024-02-08 21:15 UTC
Nmap scan report for cjoster.com (35.163.241.194)
Host is up (0.0062s latency).
Other addresses for cjoster.com (not scanned): 2600:1f14:d0:a000:fc80:95a1:a7a9:3c65

PORT    STATE SERVICE
80/tcp  open  http
443/tcp open  https

Nmap done: 1 IP address (1 host up) scanned in 0.05 seconds

Osiris · February 8, 2024, 9:59pm

You might want to consider using the --deploy-hook option, as it looks like your script is only meant to be used when a newly issued certificate needs deploying.

Also, I'm wondering what the purpose of those three cat commands in your script is? Maybe the entire command is truncated by Certbot, I dunno, but now it looks like it just cats everything to the CLI, including the private key?

Also also note that if you've used one option like --must-staple or --deploy-hook once on a succesful renewal, those options are stored and you can simply run certbot renew.

Bruce5051 · February 8, 2024, 10:04pm

And SSL Server Test: cjoster.com (Powered by Qualys SSL Labs) shows "Unable to connect to the server" for the IPv6 Address.

lordvadr · February 9, 2024, 2:13pm

The script is idempotent if no changes were made, but thanks for the pointer. I don't think --deploy-hook was a thing when I first wrote the script. Either that, or I missed it. Not sure.

lordvadr · February 9, 2024, 2:15pm

Thank you for pointing this out. This is post migration to a new host and you're right, I never did stitch up IPv6 because, well, I don't have IPv6 anymore (new ISP at home) and have no use for it. Given that the errors showed only IPv4 addresses, I had a hunch, but that didn't seem obvious.

That's an interesting assertion that the v4 and v6 hosts have to respond identically. I mean, I understand why they usually will, but I can foresee circumstances where they would not.

Thank you again.

MikeMcQ · February 9, 2024, 2:26pm

At least for the Let's Encrypt HTTP Challenge ... LE Server uses the IPv6 address when present. If that fails with a clean timeout it will retry with IPv4. Any other errors with IPv6 (like faulty server response) won't be retried. (the doc link here)

You saw the IPv4 in the error because this timeout retry occurred. And, I agree it is confusing to see that rather than the IPv6 address.

system · March 10, 2024, 2:27pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Unable renewing certificate Help	17	766	May 13, 2024
Renew cert timeout on ipv6 Help	7	501	August 9, 2022
Certbot Stuck in the Same Renew Error Help	8	2525	January 11, 2018
Renew cert failure likely because of IPv6 Help	18	4450	June 30, 2017
Timeout during connect (likely firewall problem) Help	9	1603	October 30, 2019

Erroneous connection timeouts when renewing TLS cert

Related topics