Certbot report possible firewall rule issue (--expand)

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:
mll7.com
mserv.pw
mserv.top

I ran this command:
certbot certonly --expand --rsa-key-size=4096 -n --webroot -w /var/www/html/www -d mll7.com --webroot -w /var/www/html/www -d www.mll7.com --webroot -w /var/www/html/www -d vpn.mll7.com --webroot -w /var/www/html/www -d mail.mll7.com --webroot -w /var/www/html/www -d meet.mll7.com --webroot -w /var/www/html/www -d jonathanfalk.mll7.com --webroot -w /var/www/html/www -d mserv.pw --webroot -w /var/www/html/www -d www.mserv.pw --webroot -w /var/www/html/www -d vpn.mserv.pw --webroot -w /var/www/html/www -d mail.mserv.pw --webroot -w /var/www/html/www -d meet.mserv.pw --webroot -w /var/www/html/www -d jonathanfalk.mserv.pw --webroot -w /var/www/html/www -d mserv.top --webroot -w /var/www/html/www -d www.mserv.top --webroot -w /var/www/html/www -d vpn.mserv.top --webroot -w /var/www/html/www -d mail.mserv.top --webroot -w /var/www/html/www -d meet.mserv.top --webroot -w /var/www/html/www -d jonathanfalk.mserv.top

It produced this output:
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for jonathanfalk.mll7.com
http-01 challenge for meet.mll7.com
http-01 challenge for mll7.com
http-01 challenge for vpn.mll7.com
http-01 challenge for www.mll7.com
Using the webroot path /var/www/html/www for all unmatched domains.
Waiting for verification...
Challenge failed for domain jonathanfalk.mll7.com
Challenge failed for domain meet.mll7.com
Challenge failed for domain mll7.com
Challenge failed for domain vpn.mll7.com
Challenge failed for domain www.mll7.com
http-01 challenge for jonathanfalk.mll7.com
http-01 challenge for meet.mll7.com
http-01 challenge for mll7.com
http-01 challenge for vpn.mll7.com
http-01 challenge for www.mll7.com
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:

My web server is (include version):
apache2 2.4.41-4ubuntu3.1
nginx-full 1.18.0-0ubuntu1.2

The operating system my web server runs on is (include version):
Ubuntu 20.04.2 LTS

My hosting provider, if applicable, is:
N/A

I can login to a root shell on my machine (yes or no, or I don't know):
Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
certbot 0.40.0

cat /var/log/apache2/access.log | grep "www.letsencrypt.org" | grep "03/Jun/2021:19:28"

=========================================================================================
3.142.122.14 - - [03/Jun/2021:19:28:19 -0500] "GET /.well-known/acme-challenge/Na90W3C9ZWtHQyAGHD5iVO5REumS4tKfOPMWh4YvW74 HTTP/1.1" 200 336 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
3.142.122.14 - - [03/Jun/2021:19:28:19 -0500] "GET /.well-known/acme-challenge/qzWsnYCmiXFXz_JiRSX9U9CxN-mQUkKNW0vX5e-F_eo HTTP/1.1" 200 336 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
18.184.114.154 - - [03/Jun/2021:19:28:19 -0500] "GET /.well-known/acme-challenge/qzWsnYCmiXFXz_JiRSX9U9CxN-mQUkKNW0vX5e-F_eo HTTP/1.1" 200 336 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
3.19.56.43 - - [03/Jun/2021:19:28:19 -0500] "GET /.well-known/acme-challenge/JQdXR10eVA_z4qrZvjsMA4x6iLev2qV5kDUB40bJk-Q HTTP/1.1" 200 336 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
34.221.255.206 - - [03/Jun/2021:19:28:19 -0500] "GET /.well-known/acme-challenge/Na90W3C9ZWtHQyAGHD5iVO5REumS4tKfOPMWh4YvW74 HTTP/1.1" 200 336 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
54.189.22.122 - - [03/Jun/2021:19:28:19 -0500] "GET /.well-known/acme-challenge/qzWsnYCmiXFXz_JiRSX9U9CxN-mQUkKNW0vX5e-F_eo HTTP/1.1" 200 336 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
3.120.130.29 - - [03/Jun/2021:19:28:19 -0500] "GET /.well-known/acme-challenge/Na90W3C9ZWtHQyAGHD5iVO5REumS4tKfOPMWh4YvW74 HTTP/1.1" 200 336 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
52.39.4.59 - - [03/Jun/2021:19:28:19 -0500] "GET /.well-known/acme-challenge/JQdXR10eVA_z4qrZvjsMA4x6iLev2qV5kDUB40bJk-Q HTTP/1.1" 200 336 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
18.116.86.117 - - [03/Jun/2021:19:28:19 -0500] "GET /.well-known/acme-challenge/bYkw8xdMgX9_7ToySwQbycEt3hF04pB78iZtoRsuGcE HTTP/1.1" 200 336 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
52.39.4.59 - - [03/Jun/2021:19:28:19 -0500] "GET /.well-known/acme-challenge/bYkw8xdMgX9_7ToySwQbycEt3hF04pB78iZtoRsuGcE HTTP/1.1" 200 336 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
3.122.178.200 - - [03/Jun/2021:19:28:19 -0500] "GET /.well-known/acme-challenge/JQdXR10eVA_z4qrZvjsMA4x6iLev2qV5kDUB40bJk-Q HTTP/1.1" 200 336 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
3.19.56.43 - - [03/Jun/2021:19:28:19 -0500] "GET /.well-known/acme-challenge/SFA_SDf478fwPn7XqRQmHVj72-Pdbar6lg4cTh0vXNQ HTTP/1.1" 200 336 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
3.120.130.29 - - [03/Jun/2021:19:28:19 -0500] "GET /.well-known/acme-challenge/bYkw8xdMgX9_7ToySwQbycEt3hF04pB78iZtoRsuGcE HTTP/1.1" 200 336 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
52.39.4.59 - - [03/Jun/2021:19:28:19 -0500] "GET /.well-known/acme-challenge/SFA_SDf478fwPn7XqRQmHVj72-Pdbar6lg4cTh0vXNQ HTTP/1.1" 200 336 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

curl http://jonathanfalk.mll7.com/.well-known/acme-challenge/qzWsnYCmiXFXz_JiRSX9U9CxN-mQUkKNW0vX5e-F_eo

qzWsnYCmiXFXz_JiRSX9U9CxN-mQUkKNW0vX5e-F_eo.NjVU1ttmm7KWthKactl13VRE-23Jdwc2sdvQK_XlGS4

Let me know if you want the letsencrypt.log file as it is too long to paste here...

Let's Encrypt can connect to other Vultr servers in that IP range, so it seems likely that it is actually a firewall issue on your server.

Do you have any IPs blocked in iptables? What's the output of:

sudo iptables-save

Sometimes software like fail2ban or similar can trigger these blocks.

It can be slightly misleading to read the logs because Let's Encrypt makes requests from multiple validation servers at once.

I notice that 66.133.109.36 is missing from your logs, which is one of the currently active validation IPs. Worth checking in your iptables for that.

Keep in mind that Let's Encrypt does not publish its validation IPs and that they do change.

1 Like

Hi,

Thanks for prompt followup.

Here is the output from iptables-save

iptables-save

Generated by iptables-save v1.8.4 on Thu Jun 3 20:56:16 2021

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [41:3660]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -m state --state NEW -m recent --set --name DEFAULT --mask 255.255.255.255 --rsource
-A INPUT -p tcp -m tcp --dport 22 -m state --state NEW -m recent --update --seconds 60 --hitcount 10 --name DEFAULT --mask 255.255.255.255 --rsource -j DROP
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT
-A INPUT -p tcp -m state --state NEW,ESTABLISHED -m tcp --dport 21 -j ACCEPT
-A INPUT -p tcp -m state --state RELATED,ESTABLISHED -m tcp --dport 15000:15100 -j ACCEPT
-A INPUT -s 192.168.12.6/32 -p tcp -m multiport --dports 111,2049 -j ACCEPT
-A INPUT -j DROP
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j DROP
COMMIT

Completed on Thu Jun 3 20:56:16 2021

And regarding the 66.133.109.36, it had been flagged by HostsDeny so cleared that one.

1 Like

That seems to have done the trick, because I no longer see the timeout for your domain:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Simulating a certificate request for jonathanfalk.mll7.com

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
  Domain: jonathanfalk.mll7.com
  Type:   unauthorized
  Detail: Invalid response from https://jonathanfalk.mll7.com/.well-known/acme-challenge/S-Nk579Rbx77s4a83kIdjc-0zxFBj1oeIjCfOuuz8fY [45.32.211.10]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p"

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
1 Like

Great, thanks! Tried again with dry run this time and it was successful so ran a production which failed:

Dry run

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for jonathanfalk.mll7.com
http-01 challenge for jonathanfalk.mserv.pw
http-01 challenge for jonathanfalk.mserv.top
http-01 challenge for meet.mll7.com
http-01 challenge for meet.mserv.pw
http-01 challenge for meet.mserv.top
http-01 challenge for mll7.com
http-01 challenge for mserv.pw
http-01 challenge for mserv.top
http-01 challenge for vpn.mll7.com
http-01 challenge for vpn.mserv.pw
http-01 challenge for vpn.mserv.top
http-01 challenge for www.mll7.com
http-01 challenge for www.mserv.pw
http-01 challenge for www.mserv.top
http-01 challenge for mail.mll7.com
http-01 challenge for mail.mserv.pw
http-01 challenge for mail.mserv.top
Using the webroot path /var/www/html/www for all unmatched domains.
Waiting for verification...
Cleaning up challenges

IMPORTANT NOTES:

  • The dry run was successful.

Production

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for jonathanfalk.mll7.com
http-01 challenge for meet.mll7.com
http-01 challenge for mll7.com
http-01 challenge for vpn.mll7.com
http-01 challenge for www.mll7.com
Using the webroot path /var/www/html/www for all unmatched domains.
Waiting for verification...
Challenge failed for domain jonathanfalk.mll7.com
Challenge failed for domain meet.mll7.com
Challenge failed for domain mll7.com
Challenge failed for domain vpn.mll7.com
Challenge failed for domain www.mll7.com
http-01 challenge for jonathanfalk.mll7.com
http-01 challenge for meet.mll7.com
http-01 challenge for mll7.com
http-01 challenge for vpn.mll7.com
http-01 challenge for www.mll7.com
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:

1 Like

Ran one more time and this time it worked:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for jonathanfalk.mll7.com
http-01 challenge for meet.mll7.com
http-01 challenge for mll7.com
http-01 challenge for vpn.mll7.com
http-01 challenge for www.mll7.com
Using the webroot path /var/www/html/www for all unmatched domains.
Waiting for verification...
Cleaning up challenges

IMPORTANT NOTES:

  • Congratulations! Your certificate and chain have been saved at:
    /etc/letsencrypt/live/mserv.pw/fullchain.pem
    Your key file has been saved at:
    /etc/letsencrypt/live/mserv.pw/privkey.pem
    Your cert will expire on 2021-09-02. To obtain a new or tweaked
    version of this certificate in the future, simply run certbot
    again. To non-interactively renew all of your certificates, run
    "certbot renew"

  • If you like Certbot, please consider supporting our work by:

    Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate
    Donating to EFF: https://eff.org/donate-le

Whatever was different this time made it successful - thanks a lot for helping troubleshooting this!

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.