Timeout during connect (likely firewall problem) after changing internet provider

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:
fotografeer.nl

I ran this command:
/home/letsencrypt/certbot-auto renew --force-renewal

It produced this output:
Your system is not supported by certbot-auto anymore.
certbot-auto and its Certbot installation will no longer receive updates.
You will not receive any bug fixes including those fixing server compatibility
or security problems.
Please visit https://certbot.eff.org/ to check for other alternatives.
Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/fotografeer.nl.conf

Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for fotografeer.nl
http-01 challenge for www.fotografeer.nl
Waiting for verification...
Challenge failed for domain fotografeer.nl
Challenge failed for domain www.fotografeer.nl
http-01 challenge for fotografeer.nl
http-01 challenge for www.fotografeer.nl
Cleaning up challenges
Attempting to renew cert (fotografeer.nl) from /etc/letsencrypt/renewal/fotografeer.nl.conf produced an unexpected error: Some challenges have failed.. Skipping.
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/fotografeer.nl/fullchain.pem (failure)

All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/fotografeer.nl/fullchain.pem (failure)

1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:

My web server is (include version):
Server version: Apache/2.4.25 (Debian)
Server built: 2019-02-18T19:55:40

The operating system my web server runs on is (include version):
PRETTY_NAME="ReadyNASOS 6.10.6"
NAME="Debian GNU/Linux"
VERSION_ID="8"
VERSION="8 (jessie)"
ID=debian
HOME_URL="http://www.debian.org/"
SUPPORT_URL="Debian -- Support"
BUG_REPORT_URL="https://bugs.debian.org/"

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don't know):
yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
certbot 1.9.0

1 Like
2

Please note that "--force-renewal" doesn't magically skip the entire validation procedure from Let's Encrypts side. That validation procedure is mandatory! Would be quite the security leak if I could get a certificate for whitehouse.gov by simply adding --force-renewal :smiley: So please, don't use that option if you don't know what it does. It could give you all sorts of trouble such as running into rate limits.

Further more, it might be due to the fact your websites DNS A record points to an IP address from Freedom Internet (jeej!), but your IPv6 AAAA resource record still points to an IPv6 address from XS4ALL (boo, KPN :frowning:) You probably made the sensible switch from XS4ALL to Freeedom internet, right? Do you also have IPv6 at Freedom? If so, you probably should update the AAAA resource record in your TransIP zone editor to the Freedom IPv6 address. If not, you probably should just delete the AAAA resource record entirely.

4 Likes
3

I'm normally running without --force-renewal, but the certificate has expired. Thanks for finding the issue with the IPv6 AAAA resource record. That should also point to freedom, since I completely switched to freedom.nl. I will check my settings at my domain registrar. Maybe the AAAA is not (properly) configured yet.

2 Likes
4

What connection do you see between the use of --force-renewal and an expired cert?

4 Likes
5

nslookup was showing the IPv6 address from my old provider. I checked my domain name registrar settings and found that it was indeed the old IPv6 address. I now updated it to the correct one and am waiting for it to cascade down to other DNS servers.
Maybe I'm misinterpreting the use of --force-renewal. Will stop using it.

2 Likes
6

Still don't see what that has to do with --force-renewal--perhaps you don't understand what that option does. The --force-renewal option is only used to tell certbot to request a new duplicate (i.e., "renewed") cert, even when the existing cert has not expired, and indeed is not close (by default, within 30 days) to expiring. It has no relevance to a cert that's already expired, or is about to expire (again, by default, that means it would expire in 30 days or less).

It doesn't really hurt much to use this option from the command line, but it's very dangerous (and wasteful) indeed to use it in a cron job.

4 Likes
7

I was working under the assumption that I needed to force renewal when a certificate is expired. After checking the details (with certbot-auto --help renew), I now see that my assumption was not correct. Thanks for pointing that out.
I only used --force-renewal from command-line for testing. It is not used in the crontab.

3 Likes
8

Your client would renew the certificate by itself without any "forcing" if it has expired.

Anyway, after fixing your AAAA record, you should be good to go! If not, please let us know to check everything again :slight_smile:

5 Likes
9

...but not if it continually fails due to, e.g., a bad AAAA record...

3 Likes
10

My provider's DNS is slow to pickup the change. Will verify tomorrow.

1 Like
11

Yeah, DNS can be very slow to propogate unfortunately. And even if one half of the world has the correct value, you might find that other parts of the world are still lagging behind. And as Let's Encrypt uses multiple vantage points to validate the challenge, you might run into that issue.

In any case, from where I'm sitting, I'm seeing the Freedom Internet-IPv6-address on your domain. But I'm from the same country, so that doesn't say anything about global propogation as said earlier.

3 Likes
12

So the DNS servers that Let's Encrypt uses to validate the challenge may take as long as 24 to be updated. Then it is always better to wait a least a day before trying to renew after change of DNS A/AAAA record(s), I guess?

That lagging that you mention is currently happening. Via dns.google (8.8.8.8), nslookup allready shows both IPv6 and IPv4 addresses. Via dns.freedom.nl, nslookup currently only shows the IPv4.

nslookup fotografeer.nl 8.8.8.8
nslookup fotografeer.nl dns.freedom.nl

1 Like
13

No, not really, there's nothing to update. Let's Encrypt always traverses the DNS from the root zone up to the authorative DNS server of the hostname. It does have a cache for resolved hostnames, but it's very short. I believe just a few minutes.

However, the authorative DNS servers of your website might take a long time to update. Not every DNS provider is slow, some are very fast. Others take a while, sometimes hours or even a day indeed.

DNS servers like 8.8.8.8 or the DNS resolver of your ISP might cache results for a relative long time, so querying those servers might not provide current information.

You're probably good to go already :slight_smile:

3 Likes
14

You're probably good to go already :slight_smile:

A different error message show up now: "Error getting validation data"

# /home/letsencrypt/certbot-auto renew
Your system is not supported by certbot-auto anymore.
certbot-auto and its Certbot installation will no longer receive updates.
You will not receive any bug fixes including those fixing server compatibility
or security problems.
Please visit https://certbot.eff.org/ to check for other alternatives.
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/fotografeer.nl.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for fotografeer.nl
http-01 challenge for www.fotografeer.nl
Waiting for verification...
Challenge failed for domain fotografeer.nl
Challenge failed for domain www.fotografeer.nl
http-01 challenge for fotografeer.nl
http-01 challenge for www.fotografeer.nl
Cleaning up challenges
Attempting to renew cert (fotografeer.nl) from /etc/letsencrypt/renewal/fotografeer.nl.conf produced an unexpected error: Some challenges have failed.. Skipping.
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/fotografeer.nl/fullchain.pem (failure)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/fotografeer.nl/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: fotografeer.nl
   Type:   connection
   Detail: Fetching
   https://fotografeer.nl/.well-known/acme-challenge/jtiOWOBNAQh_KdBgHxS8zBJHGnnf-7_Voi9vPUM4bSE:
   Error getting validation data

   Domain: www.fotografeer.nl
   Type:   connection
   Detail: Fetching
   https://www.fotografeer.nl/.well-known/acme-challenge/m-0PKGgeMQTHK354qN04Kv_eFunGnoW6k8Fix2bNV18:
   Error getting validation data

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.
1 Like
15

That's weird, no extra info from the LE server about what went wrong.. That's no help at all :confused:

I do see permission denied errors when I try to connect to your IPv6 address on port 80 or 443.. Maybe a firewall?

2 Likes
16

The HTTP challenge requests are being redirected to HTTPS.
I'd say that was a missed opportunity to have answered that request.

3 Likes
17

You may be on to something. Looking at the logs from when I was still with my old provider, I can see that certbot-auto always used HTTP for the challenge.

http://fotografeer.nl/.well-known/acme-challenge/...

But with my new provider, certbot-auto uses HTTPS for the challenge.

https://fotografeer.nl/.well-known/acme-challenge/...

Not sure why that is different. The server settings do redirect HTTP to HTTPS, but that has never been a problem when still with old provider.

This is the current rewrite setting:

        RewriteEngine on
                ReWriteCond %{SERVER_PORT} !^443$
                RewriteRule ^/(.*) https://%{HTTP_HOST}/$1 [NC,R,L]

If I switch off the Apache Rewrite engine, the challenges goes through and new certificate is loaded. But the idea for having redirect to HTTPS is to force all users to visit the site via HTTPS only. I don't want to switch that off permanently. Probablly need to find a way to not rewrite for the challenges, something along the lines of what is mentioned in: lets encrypt - I have a rewrite in an apache httpd conf file, that breaks certbot. Is there a way to change it so that it doesn't? - Server Fault

1 Like
18

Thanks for your support! To recap:

Issues

  1. AAAA record in DNS registar was still pointing to old provider.
  2. Apache rewrite engine forwarded the ACME challenge to HTTPS, causing it to fail.

Fixes

  1. Corrected AAAA record to point to the IPv6 address at the new provider.
  2. Changed the Apache rewrite condition to skip rewrite for ACME challenges. See below. After restart of apache2 service, the renew worked.
RewriteEngine on
        RewriteCond %{REQUEST_URI} !^/\.well-known/acme-challenge/
        RewriteRule ^/(.*) https://%{HTTP_HOST}/$1 [NC,R,L]
2 Likes
19

While this might have solved your issue regarding the issuance of LE certificates, this should not have been necessary and, to me, signals that the original issue preventing you from getting a cert is still active and could impact other users with e.g. IPv6.

Yes, I saw that. Redirecting to HTTPS is perfectly fine and should work.

What missed opportunity?

2 Likes
20

Yes, as long as the server (1) is listening there and (2) has a formally valid certificate (even a self signed one).

3 Likes