Certbot http challenge problem, maybe VPN issue?

Help
1

I’ve been using certbot for many years on my home server that hosts a few domains and runs an email server. I cannot get a static IP directly through my internet provider, so I have a static IP VPN set up on this server for that purpose.

About 6 months ago the certbot automatic renewal failed. I tried renewing manually but it failed the http challenges. I’ve shown the result below for the hostname jezzubu.kehoe.org, which is the FQDN of the server and has a generic apache site on it (the actual websites are on virtual hosts). I have checked jezzubu.kehoe.org on https://letsdebug.net and it said all was OK. I followed the apache access logs and saw one successful retrieval of the /.well-known/acme-challenge/… file when I run the certbot command. How many retrievals should I expect?

I’m wondering if the IP address I have from my VPN provider has been tagged as Chinese/VPN and being blocked from one of the Letsencrypt challenge servers. The VPN endpoint is in Phoenix, AZ, but sometimes when I open Google on a browser on a VM that shares that IP, the google login prompt is in Chinese (not always, it flips back and forth from English). I believe I’m the only one using this IP, but maybe a block of IP’s had been tagged, or maybe the private IP is not as private as advertised. Also, before this cert issue I started having an issue with incoming SMTP connections on port 25 being blocked through the VPN connection from some origins (not all origins); I had to set up a secondary MX on a cheap VPS to get around that.

Please help, I’m limping along with self-signed certificates but that’s causing all sorts of other issues.

My domain is: jezzubu.kehoe.org

I ran this command: sudo certbot --apache -d jezzubu.kehoe.org

It produced this output:

Requesting a certificate for jezzubu.kehoe.org

Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:
Domain: jezzubu.kehoe.org
Type: connection
Detail: During secondary validation: 23.81.125.21: Fetching http://jezzubu.kehoe.org/.well-known/acme-challenge/RKxQrUaEPP9lrpL77t7hRkeyyNCXRsbme9rRck4lCPg: Timeout during connect (likely firewall problem)

Hint: The Certificate Authority failed to verify the temporary Apache configuration changes made by Certbot. Ensure that the listed domains point to this Apache server and that it is accessible from the internet.

Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

My web server is (include version): Apache 2.4.52-1ubuntu

The operating system my web server runs on is (include version): Ubuntu 22.04.5 LTS

My hosting provider, if applicable, is: self-hosted

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 3.0.0

2

there will be five iirc 3 in main + two usa server and one in europe and asia
as your error says during secondary validation I'd guess its about geoblocking firewall I guess

1 Like
3

For success five. You can succeed with 4 but if that is a pattern that points to a likely geographic firewall blocking certain worldwide regions. One may occasionally fail for comms reasons or what-not.

Are you blocking all requests from Amazon AWS origins?

Because the Let's Encrypt primary center makes the first request and is not in AWS. All the others are. The error says "Secondary" failure so is not a problem from the primary center. All the secondaries failing would account for you seeing only one log entry.

And, Let's Debug issues two kinds of tests. One from its own server which succeeds and which is not AWS. The other uses Let's Encrypt staging system but it only checks the primary center which we know succeeds from your Certbot message.

I can't reach that domain from my own AWS region on US East Coast.

So, AWS seems the common denominator (so far anyway).

2 Likes
4

Hmm, interesting. I'm not blocking anything on my end, but I'm wondering if AWS is refusing to connect to an IP tagged as being on a VPN. I ran certbot again and confirmed that only one retrieval is getting through, from a 23.178.x.x IP.

This could also explain the SMTP issue I was having earlier. Gmail was one of the services that wouldn't deliver to this VPN IP; don't know if they use AWS but maybe use a common filtering service. Is there any recourse for this other than using a different VPN? Unfortunately I bought two years of the service just before this stuff started happening. Could the VPN service be actually sharing my supposedly "private" IP or is it just guilty of being in a known block of VPN addresses?

Thanks for the help.

1 Like
5

No. AWS is not blocking outbound traffic from a server in its environ.

Are you sure the VPN has no firewall blocks?

What was the IP address that came through? It is not a secret especially it that is the original requester IP.

You mention China. Is this behind the great wall of china firewall?

2 Likes
6

Are you sure the VPN has no firewall blocks?
I'm not sure how to verify this, obviously some traffic is getting through on port 80. When I had the SMTP issue, I did contact the VPN support about it but that didn't lead to a fix. I suppose I could try again with these additional clues.

What was the IP address that came through? It
Varies slightly each time I run certbot, although always only one gets through each run. Here are a few IPs see in the logs from the 'Let's Encrypt validation server':
23.178.112.214
23.178.112.212
66.133.109.36
The Let's Debug query came from here:
65.21.146.168

< You mention China. Is this behind the great wall of china firewall?
I'm in Seattle, WA and the VPN endpoint is in Phoenix, AZ. I think the VPN provider serves a lot of customers in China, and I've seen evidence that my IP, or block of IP's including mine, has been somehow tagged as Chinese, for example when I browse the web using my Private IP, Google occasionally presents the Chinese login screen. The China Firewall may have tagged this IP as VPN and blocking it, but would that alone prevent certbot from running?

Thanks again for your help.

1 Like
7

Actually, to clarify, it's the (annoying, IMHO) google login popup prompt in Firefox that's coming up in Chinese when I'm on my VPN IP. I'd post a screenshot but it's in English at the moment...

8

Those are IP of the primary Let's Encrypt center which is not AWS. The other is the Let's Debug test server as I described. None of the secondary LE centers which are in AWS are reaching you.

And, as noted, I cannot reach your "home" page from my own AWS EC2 test server. I can reach you from another test server I have just not the AWS one.

This isn't Let's Encrypt unique. You need to work with your VPN provider and find out why these are blocked. Maybe they had a problem with too many bots or something and block inbound requests from AWS IP. I don't know.

I can't reach your home page:

curl -i -m10 http://jezzubu.kehoe.org
curl: (28) Connection timed out after 10001 milliseconds
5 Likes
9

Thanks for the info and troubleshooting help, I really do appreciate it and this helps me make more-targeted and informed queries to my VPN provider.

I might consider ditching the VPN altogether and instead see if I can tunnel into the VPS to use its static IP

3 Likes
10

I don't understand exactly why you need the VPN. It didn't matter for this problem as HTTP requests must work and your comms config must allow it.

You said you needed the VPN to provide a 'static IP' so I wasn't sure if you had a public IP of your own (that just changed) or did not have a public IP at all. The former would just need a DDNS service but the latter something more (like a VPN).

Assuming the latter you might look at Cloudflare. There is a tunnel feature that allows connections when you don't have a public IP. Tunnel | Zero Trust App Connector

If you do have a public IP that just changes there are many DDNS services. Cloudflare describes options with their system here: Dynamically update DNS records | Cloudflare DNS docs

5 Likes