Certificate renewal failing - IPv6?

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: oslc.org

I ran this command: certbot renew

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/smtp.oslc.org.conf

Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator standalone, Installer None
Renewing an existing certificate for smtp.oslc.org
Performing the following challenges:
http-01 challenge for smtp.oslc.org
Waiting for verification...
Challenge failed for domain smtp.oslc.org
http-01 challenge for smtp.oslc.org
Cleaning up challenges
Failed to renew certificate smtp.oslc.org with error: Some challenges have failed.

All renewals failed. The following certificates could not be renewed:
/etc/letsencrypt/live/smtp.oslc.org/fullchain.pem (failure)

1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:

  • The following errors were reported by the server:

    Domain: smtp.oslc.org
    Type: connection
    Detail: During secondary validation: 209.181.62.139: Fetching
    http://smtp.oslc.org/.well-known/acme-challenge/7Of-F1rS9B79u7uKRGQu15R0dGsHuhWrO7JZV3vQ5gI:
    Timeout during connect (likely firewall problem)

    To fix these errors, please make sure that your domain name was
    entered correctly and the DNS A/AAAA record(s) for that domain
    contain(s) the right IP address. Additionally, please check that
    your computer has a publicly routable IP address and that no
    firewalls are preventing the server from communicating with the
    client. If you're using the webroot plugin, you should also verify
    that you are serving files from the webroot path you provided.

My web server is (include version): Postfix 3.5.24

The operating system my web server runs on is (include version): Debian 11

My hosting provider, if applicable, is: N/A (self-hosted)

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): 1.12.0

2

What makes you think this would be an IPv6 issue? The IP address mentioned in the error message is IPv4.

I also can't connect to smtp.oslc.org on port 80. Is there a firewall blocking that port?

2 Likes
3

Additional info: This is one of three servers we have that are using certbot but the only one that is failing. Cert renewals have worked fine up until now, and we haven't changed anything in our environment as it relates to this server (DNS, firewall, etc).

One thing strange I noticed in the letsencrypt.log were these lines:

2024-04-16 16:39:53,057:DEBUG:acme.standalone:Successfully bound to :80 using IPv6
2024-04-16 16:39:53,057:DEBUG:acme.standalone:Certbot wasn't able to bind to :80 using IPv4, this is often expected due to the dual stack nature of IPv6 socket implementations.

We have IPv6 disabled so I don’t see how certbot is binding to IPv6. I tried the following command, but that didn’t help.

/usr/bin/certbot renew --standalone --http-01-address <our server's IP>

4

There's probably something else already listening on (IPv4) port 80. You can check using:

sudo netstat -nap | grep LISTEN | grep :80
1 Like
5

Thanks for the reply Osiris. Running the netstat command returns no results. Nothing else is listening on 80.

1 Like
6

Hmm, right, nevermind.. With dualstack IPv4/IPv6 the lines from the log are just fine. It's also listening on IPv4 even if there's nothing actually configured for IPv6.

The fact I can't connect to port 80 suggests a firewall issue as mentioned earlier.

3 Likes
7

The fact that it's failed during secondary validation suggests that this is yet another instance of the geoblocking thing and new validation perspectives in different countries.

I should really push to get that new documentation page published!

3 Likes
8

Yup. The IP address from OP seems to be situated in the US and from my perspective, Europe, it's down.

Weirdly, port 443 is working just fine.

2 Likes
9

@garrisont For now, take a quick look at other threads about renewals failing and new validation perspectives—this issue is common this week and is well-explained in other threads on the forum.

2 Likes
10

Will do---thank you both!

2 Likes
11

I can't reach it from an AWS East Coast region EC2 system. There is no IPv6 address in the DNS so I don't see how that is involved at least for requests from the public internet.

curl -i4 http://smtp.oslc.org
curl: (7) Failed to connect to smtp.oslc.org port 80 after 115 ms: 
Connection refused

I can reach it using HTTPS just like @Osiris so this looks like a general firewall block of port 80

2 Likes
12

We don't have port 80 open on this server. We let certbot open the port temporarily during renewals.

1 Like
13

Do the other two servers still work? That would isolate the problem

You can run sudo certbot renew --dry-run to test without affecting production certs

3 Likes
14

Ah. I just did a dry run on all our other certbot servers (turns out we have five) and the only one that didn't fail during second validation was the server that is hosted by AWS. All the others we host on-prem.

2 Likes
15

That's a different "open" than with firewalls. Usually when no service is listening, a Linux system will send out an ICMP reply with "connection refused". That's totally different from what we're currently seeing: a complete lack of response resulting in a timeout.

2 Likes
16

I found that our firewall was only allowing traffic from US locations, and saw that the secondary validation traffic from SWE and SGP was being dropped. Once I adjusted this, cert renewal worked. Thanks all for your responses.

2 Likes