Timeout on renew

[Hananja]

Hi guys,

I'm stuck in a problem that I cannot solve. On a server that used to run without problems, I now get timeout errors. However, the challenge query is successful. I have checked this with Wireshark. Where is the connection problem that is causing the timeout? I am at a loss and would appreciate any help. Here are the details:

My domain is: admin.szut.de

I ran this command:
$ sudo certbot renew --dry-run
$ sudo certbot certonly --apache --dry-run -d admin.szut.de

I tried certbot renew and certonly with or without --dry-run. The error message is always the same.

It produced this output:

administrator@demosios:~$ sudo certbot certonly --apache --dry-run -d admin.szut.de
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for admin.szut.de
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. admin.szut.de (http-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://admin.szut.de/.well-known/acme-challenge/ILMBS9VxOtPXO7bfOzxTRtpK--x98RpQdTx5tJsQtAs: Timeout during connect (likely firewall problem)

IMPORTANT NOTES:

• The following errors were reported by the server:

  Domain: admin.szut.de
  Type: connection
  Detail: Fetching
  http://admin.szut.de/.well-known/acme-challenge/ILMBS9VxOtPXO7bfOzxTRtpK--x98RpQdTx5tJsQtAs:
  Timeout during connect (likely firewall problem)

  To fix these errors, please make sure that your domain name was
  entered correctly and the DNS A/AAAA record(s) for that domain
  contain(s) the right IP address. Additionally, please check that
  your computer has a publicly routable IP address and that no
  firewalls are preventing the server from communicating with the
  client. If you're using the webroot plugin, you should also verify
  that you are serving files from the webroot path you provided.

My web server is (include version): apache 2.4.29

The operating system my web server runs on is (include version): Ubuntu 18.04.5 LTS

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 0.31.0

The Capture of HTTP challenge traffic is:

GET /.well-known/acme-challenge/ILMBS9VxOtPXO7bfOzxTRtpK--x98RpQdTx5tJsQtAs HTTP/1.1
Host: admin.szut.de
User-Agent: Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)
Accept: /
Accept-Encoding: gzip
Connection: close

HTTP/1.1 200 OK
Date: Thu, 10 Jun 2021 12:26:08 GMT
Server: Apache/2.4.29 (Ubuntu)
Last-Modified: Thu, 10 Jun 2021 12:26:04 GMT
ETag: "57-5c4687eb34f5b"
Accept-Ranges: bytes
Content-Length: 87
Connection: close

ILMBS9VxOtPXO7bfOzxTRtpK--x98RpQdTx5tJsQtAs.FqSWek3yvVGFqNmCDKx6-zD95hCPtaFthOkyu3yQnno

Searching the web I only found hints that the server might not be reachable via HTTP, but the challenge query obviously works. Where can a timeout still occur? Thanks for any idea.

Best regards

Hananja

[_az]

@Hananja

The Capture of HTTP challenge traffic is:

This can be misleading! You should be seeing 4 requests for the same challenge file, from different IP addresses. If any of them is dropped, it can present itself with the timeout error.

Could you post the Apache access log lines for that ILMBS9VxOtPXO7bfOzxTRtpK--x98RpQdTx5tJsQtAs resource?

[Hananja]

There is only one entry with this ID in access.log:

18.196.102.134 - - [10/Jun/2021:14:26:08 +0200] "GET /.well-known/acme-challenge/ILMBS9VxOtPXO7bfOzxTRtpK--x98RpQdTx5tJsQtAs HTTP/1.1" 200 308 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

So you think, that the other three ones are blocked somewhere? What is the best way to debug this?

[_az]

@Hananja

that the other three ones are blocked somewhere

Yes, that seems likely.

For reference, here are the IPs that show up when I try against one of my domains just now:

34.221.186.243 - - [17/Jun/2021:17:53:40 +1000] "GET /.well-known/acme-challenge/iYdafu-QeDQcuRdcapt1F6_NPAktyYcOlzCQYpH2P_I HTTP/1.1" 404 118 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
64.78.149.164 - - [17/Jun/2021:17:53:40 +1000] "GET /.well-known/acme-challenge/iYdafu-QeDQcuRdcapt1F6_NPAktyYcOlzCQYpH2P_I HTTP/1.1" 404 118 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
3.142.122.14 - - [17/Jun/2021:17:53:40 +1000] "GET /.well-known/acme-challenge/iYdafu-QeDQcuRdcapt1F6_NPAktyYcOlzCQYpH2P_I HTTP/1.1" 404 118 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
18.184.29.122 - - [17/Jun/2021:17:53:40 +1000] "GET /.well-known/acme-challenge/iYdafu-QeDQcuRdcapt1F6_NPAktyYcOlzCQYpH2P_I HTTP/1.1" 404 118 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"

As you can see, mine don't even include the IP that you have in your access log. This is because the IP address pool of the validation servers tends to change over time.

But what you might try is to see whether you can find 64.78.149.164 (which is the primary validation server, the other 3 ones are secondaries) in your iptables or hosts.deny. Or maybe try see in a packet capture whether you are receiving any SYN packets from that host in the first place.

[MyraFigueroa]

pls test

curl https://acme-v02.api.letsencrypt.org

[griffin]

@MyraFigueroa

Hi

Please see my response to your same suggestion here:

Timeout during connect (likely firewall problem)

@…

[JimPas]

I believe you have a problem with your dns and nameserver... or lack of a nameserver for your admin subdomain. No problem with your apex domain and www subdomain with their DigiCert CA.

NS Lookup - Lookup Nameserver of any Domain
Sorry no record found!

Can't find auth dns server for admin.szut.de

And for your IP address ... DNS Lookup - Check DNS Records
No records found, not even an A record.

DNS Propagation Test for admin.szut.de - Explore the results
No NS found. However, this tool did show an A record propagated around the world with the exception of Japan, South Africa and Argentina. Still, checking the DNS PTR to reverse lookup your domain (admin.szut.de) against your IP address fails to find anything as well.

You may want to check your records to see if any nameservers are shown for your admin subdomain.

[Hananja]

This works fine. I got HTML with a description page.

[Hananja]

Thank you for your hints on DNS.

Can't find auth dns server for admin.szut.de

But there are two for szut.de. Is a NS entry for admin.szut.de necessary?

With the Propagation tool, I got results for Type A for admin.szut.de and www.admin.szut.de with only two problems (timeouts in Paris and Buenos Aires and the last fails also with google.de).

[JimPas]

@Hananja

Is a NS entry for admin.szut.de necessary?

No, sorry for the confusion. However, you should have a CNAME record for your admin subdomain that points to your apex domain (szut.de). That way the subdomain will be associated with the IP address for your apex domain... or see below.

I see the server IP address for szut.de is 78.46.3.115, but for your subdomain at admin.szut.de it's 195.37.105.110 - two different servers. Was this intentional - different content being served by each server? Is there a proxy set now that wasn't there when you first got your cert?

You can try setting the IP for admin.szut.de to 78.46.3.115 and see if you can reach your subdomain. Your problem sounds like it is one of two things: 1) a firewall blocking (which has been mentioned already), or 2) the wrong IP address for your subdomain (time outs/not found errors).

@Hananja

To fix these errors, please make sure that your domain name was
entered correctly

and the DNS A/AAAA record(s) for that domain contain(s) the right IP address.

I'm still getting a time out trying to reach your admin.szut.de site as of 20:08 UTC.

[Hananja]

Hi folks

It turned out that the guys operating the firewall were doing some weird things that they didn't talk about and would have been better off leaving alone. I suspect it was because of some geoblocking. Now it works again.

Thanks for any help and sorry for the inconvenience. Just be glad you don't work in a government office.