Network is unreachable

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: *.sipengines.com

I ran this command: certbot -v renew --dry-run

It produced this output: 2025-05-16 14:24:15,461:DEBUG:certbot._internal.main:certbot version: 2.3.0202 - Pastebin.com

My web server is (include version): using haproxy 2.2.9-2

The operating system my web server runs on is (include version): debian 11

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): 2.3.0

Hello,
I have been using certbot to auto-renew a wildcard certificate via a python dns-constellix authenticator for several months.
Yesterday the certificate expired and I realized the service certbot service was stopped on the server. There seemed to be a new certificate named sipengines.com.conf-0001 in addition to the previously working sipengines.com.conf. I don't know why that would be?
Restarting the certbot service failed and it was caused by this new rogue certificate. I deleted the new certificate and restarted the service successfully.

However now I am testing renewals but it is failing due network unreachable. I don't know why I am not able to reach the servers all of a sudden. We are using CF as DNS 1.1.1.1

I also used your letsdebug.net with *.sipengines.com and sipengines.com and both returned:

All OK!
No issues were found with sipengines.com. If you are having problems with creating an SSL certificate, please visit the Let's Encrypt Community forums and post a question there.

Some other tests I have performed:

curl -4 https://ifconfig.co
38.67.242.144

curl -v https://acme-staging-v02.api.letsencrypt.org
*   Trying 172.65.46.172:443...
*   Trying 2606:4700:60:0:f41b:d4fe:4325:6026:443...
* Immediate connect fail for 2606:4700:60:0:f41b:d4fe:4325:6026: Network is unreachable
* connect to 172.65.46.172 port 443 failed: Connection timed out
* Failed to connect to acme-staging-v02.api.letsencrypt.org port 443: Connection timed out
* Closing connection 0
curl: (28) Failed to connect to acme-staging-v02.api.letsencrypt.org port 443: Connection timed out

It's behaving like our IP above may be blocked?

2

usually this is because of some ephemeral network routing issue at your datacenter or a provider. try doing a traceroute to see if the connection is dropping somewhere.

traceroute 172.65.46.172
2 Likes
3

Hi, thanks for the reply.

traceroute 172.65.46.172
traceroute to 172.65.46.172 (172.65.46.172), 30 hops max, 60 byte packets
 1  38.67.242.130 (38.67.242.130)  0.164 ms  0.176 ms  0.113 ms
 2  be6708.rcr52.bct01.atlas.cogentco.com (38.142.194.225)  0.825 ms  0.777 ms  0.800 ms
 3  be6827.ccr82.mia03.atlas.cogentco.com (154.54.160.221)  28.237 ms  2.379 ms  2.188 ms
 4  be2258.ccr41.mia03.atlas.cogentco.com (154.54.168.85)  1.686 ms  1.677 ms  1.593 ms
 5  38.88.164.218 (38.88.164.218)  2.980 ms  2.356 ms 38.88.165.38 (38.88.165.38)  26.721 ms
 6  108.162.211.228 (108.162.211.228)  1.421 ms 108.162.211.12 (108.162.211.12)  2.695 ms 108.162.211.236 (108.162.211.236)  2.595 ms
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

Doesn't seem to be a routing issue even though the traceroute doesn't get responses all the way through.

I have done the same traceroute on another server and the results are the same, and that other server is successfully able to run curl -v https://acme-v02.api.letsencrypt.org/directory.

4

That's an entry node in to the Cloudflare network, which fronts the API, so your connection is good.

LetsEncrypt should not have any blocks in place here. Hopefully someone else where will have an idea on how to proceed.

3 Likes
5

Let's Encrypt doesn't have that older style IP block anymore and even when they did the symptom was not a timeout.

What do these show?

curl https://www.cloudflare.com/cdn-cgi/trace

And retry the traceroute like this please (uses TCP instead of UDP)

sudo traceroute -T -p 443 acme-v02.api.letsencrypt.org
3 Likes
6

There seems to be an intermittent access issue caused by some routers in our path from our IP scope.

I get a response 1 out of every 4 or 5 attempts to the curl command below. Most of the time it hangs for several minutes and finally will come back with a reply.

curl https://www.cloudflare.com/cdn-cgi/trace
fl=980f42
h=www.cloudflare.com
ip=38.67.242.144
ts=1747416706.069
visit_scheme=https
uag=curl/7.74.0
colo=MIA
sliver=none
http=http/2
loc=US
tls=TLSv1.3
sni=plaintext
warp=off
gateway=off
rbi=off
kex=X25519

By contrast the suggested traceroute command below seems to always respond.
however, what I did notice is that on hop 3, I will occasionally see 1 or 2 asterisk (*) so it seems a different route is taken to complete the traceroute?

sudo traceroute -T -p 443 acme-v02.api.letsencrypt.org
traceroute to acme-v02.api.letsencrypt.org (172.65.32.248), 30 hops max, 60 byte packets
 1  38.67.242.130 (38.67.242.130)  0.150 ms  0.114 ms  0.095 ms
 2  be6708.rcr52.bct01.atlas.cogentco.com (38.142.194.225)  0.762 ms  0.837 ms  0.709 ms
 3  * be6827.ccr82.mia03.atlas.cogentco.com (154.54.160.221)  3.152 ms  3.139 ms
 4  be2258.ccr41.mia03.atlas.cogentco.com (154.54.168.85)  1.777 ms  1.741 ms  2.217 ms
 5  38.88.165.38 (38.88.165.38)  2.050 ms 38.88.164.218 (38.88.164.218)  5.718 ms  2.735 ms
 6  108.162.211.238 (108.162.211.238)  2.683 ms 108.162.211.28 (108.162.211.28)  8.094 ms 108.162.211.14 (108.162.211.14)  2.430 ms
 7  172.65.32.248 (172.65.32.248)  1.840 ms  2.575 ms *

sudo traceroute -T -p 443 acme-v02.api.letsencrypt.org
traceroute to acme-v02.api.letsencrypt.org (172.65.32.248), 30 hops max, 60 byte packets
 1  38.67.242.130 (38.67.242.130)  0.182 ms  0.108 ms  0.141 ms
 2  be6708.rcr52.bct01.atlas.cogentco.com (38.142.194.225)  0.928 ms  0.838 ms  0.889 ms
 3  be6827.ccr82.mia03.atlas.cogentco.com (154.54.160.221)  3.002 ms  3.203 ms  3.311 ms
 4  be2258.ccr41.mia03.atlas.cogentco.com (154.54.168.85)  1.814 ms  1.798 ms  2.237 ms
 5  38.88.165.38 (38.88.165.38)  47.953 ms 38.88.164.218 (38.88.164.218)  3.062 ms  4.167 ms
 6  108.162.211.28 (108.162.211.28)  4.920 ms 108.162.211.236 (108.162.211.236)  1.789 ms 108.162.211.26 (108.162.211.26)  8.432 ms
 7  172.65.32.248 (172.65.32.248)  2.156 ms *  2.268 ms

sudo traceroute -T -p 443 acme-v02.api.letsencrypt.org
traceroute to acme-v02.api.letsencrypt.org (172.65.32.248), 30 hops max, 60 byte packets
 1  38.67.242.130 (38.67.242.130)  0.873 ms  0.829 ms  0.795 ms
 2  be6708.rcr52.bct01.atlas.cogentco.com (38.142.194.225)  710.254 ms  710.261 ms  710.190 ms
 3  * * be6827.ccr82.mia03.atlas.cogentco.com (154.54.160.221)  2.064 ms
 4  be2258.ccr41.mia03.atlas.cogentco.com (154.54.168.85)  2.188 ms  2.107 ms  2.123 ms
 5  38.88.164.218 (38.88.164.218)  19.594 ms  18.408 ms  19.491 ms
 6  108.162.211.232 (108.162.211.232)  2.645 ms 108.162.211.26 (108.162.211.26)  2.627 ms 108.162.211.232 (108.162.211.232)  2.721 ms
 7  172.65.32.248 (172.65.32.248)  2.108 ms * *
1 Like
7

Failing to reach that Cloudflare endpoint helps narrow the problem.

By coincidence, one of my test servers also connects to that same Cloudflare center (colo=MIA) (Miami). I don't have any problems with that same test (repeatedly) so not likely something in Cloudflare's center. Their status site doesn't indicate any problems either. Any number of "things" between you and them could be the problem. Probably closer to you than closer to them given the scale they operate at. A major outage near their centers would "light up" the boards of their monitoring tools. I say "probably" as it's the best guess - not a guarantee :slight_smile:

Usually that traceroute would fail too but given you can't reach Cloudflare's center or Let's Encrypt the comms problem isn't unique to LE.

If the problem just started recently you may just wait until tomorrow and see if it resolves. Communications problems do happen from time to time. Otherwise you may need to start working with your ISP.

2 Likes
8

It likely didn't start recently since the cert expired yesterday and can be renewed as of 30 days to expiry, so this has likely been going on for > 30 days.
Thanks for the reply Mike.

1 Like
9

As for this problem, Certbot makes a -0001 (and -0002 ...) if you make changes to the set of domain names in the cert and don't choose to expand the original (or shrink it as the case may be).

Your cert from yesterday for the sipengine wildcard had these two names, for example.

X509v3 Subject Alternative Name: 
                DNS:*.brytecall.com
                DNS:*.sipengines.com

But, the wildcard you got last Feb had

X509v3 Subject Alternative Name: 
                DNS:*.brytecall.com
                DNS:*.itlevel3.com
                DNS:*.sipengines.com
2 Likes
10

Possibly. Other things can cause that too. You could look at old logs in /var/log/letsencrypt folder and see.

Could the dropped domain name been the reason for failures the past 30d for example?

2 Likes
11

Presently I got a WARNING with Let's Debug here https://letsdebug.net/sipengines.com/2449017

12

FWIW, if I were you, I would emergency triage this situation by ordering a cert with a DNS-01 challenge from my laptop to get the domain back online properly, then go back to investigating the networking issue.

2 Likes
13

Ohh crap, so the -0001 is the correct request, and I discarded that :roll_eyes:

I only need *.sipengines.com and *.brytecall.com

What do you recommend I do to ensure only those wildcard domains are recreated now that I messed with the config?

1 Like
14

You should be able to just re-run the Certbot command as you want it. It should prompt you about the change in domain names.

Or, to be sure you update a certain one do:

sudo certbot certonly --cert-name X -d name1 -d name2 (all the rest ...)

Where X is the Certificate Name from sudo certbot certificates

2 Likes
15

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.