Cannot renew certificate due to challenges failed

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: swarm.siteinteldev.com

I ran this command: sudo certbot --apache

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache

Which names would you like to activate HTTPS for?


1: swarm.siteinteldev.com


Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel):
Cert is due for renewal, auto-renewing...
Renewing an existing certificate for swarm.siteinteldev.com
Performing the following challenges:
http-01 challenge for swarm.siteinteldev.com
Waiting for verification...
Challenge failed for domain swarm.siteinteldev.com
http-01 challenge for swarm.siteinteldev.com
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:

  • The following errors were reported by the server:

    Domain: swarm.siteinteldev.com
    Type: connection
    Detail: Fetching
    http://swarm.siteinteldev.com/.well-known/acme-challenge/vltyvVd2LYaGKlSHfvy9-aCxgMecTht5I133CLYP5o0:
    Timeout during connect (likely firewall problem)

    To fix these errors, please make sure that your domain name was
    entered correctly and the DNS A/AAAA record(s) for that domain
    contain(s) the right IP address. Additionally, please check that
    your computer has a publicly routable IP address and that no
    firewalls are preventing the server from communicating with the
    client. If you're using the webroot plugin, you should also verify
    that you are serving files from the webroot path you provided.

My web server is (include version):
Server version: Apache/2.4.29 (Ubuntu)
Server built: 2020-08-12T21:33:25

The operating system my web server runs on is (include version): Ubuntu 18.04.5 LTS (GNU/Linux 5.4.0-1039-azure x86_64)

My hosting provider, if applicable, is: Azure VM

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 1.15.0

Additional information:

https://swarm.siteinteldev.com is accessible, but http://swarm.siteinteldev.com is not. It's okay to allow http traffic if needed. Azure security group for the VM already has port 80 open.

Tried to access the well known URL via HTTPS, but got 404:
https://swarm.siteinteldev.com/.well-known/acme-challenge/vltyvVd2LYaGKlSHfvy9-aCxgMecTht5I133CLYP5o0

2 Likes

Hi @martinl, and welcome to the LE community forum :slight_smile:

And yet, port 80 is still unreachable from the Internet.
You will need to ensure port 80 is reachable to validate via HTTP.

As for the 404 via HTTPS, try placing a test text file in that expected challenge location. Then see if that file can be reached. If not, then you might not have the right location or your web server config is not doing what you expected.

3 Likes

Thank you rg305. I've been reading one of your old threads in the forum.

I was trying to find out why port 80 didn't work but couldn't figure it out so far. These are what I did:

Azure security group rules. Verified that I do have both 443 and 80 incoming open. Also did a test to double confirm that I'm modifying the right security group: first removed the 443 rule, verified that https was no longer accessible, then added a new 443 rule, and verified that https is accessible again. Also re-created the port 80 rule, but http still not accessible (either using telnet or a browser).

Checked firewall as below. But I'm not sure if ufw is the only possible firewall on ubuntu.
sudo ufw status
Status: inactive

Ran: sudo lsof -iTCP -sTCP:LISTEN -P

Apache seems to be listening on both 80 and 443.

COMMAND     PID            USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
systemd-r   902 systemd-resolve   13u  IPv4    19745      0t0  TCP localhost:53 (LISTEN)
sshd       1404            root    3u  IPv4    23563      0t0  TCP *:22 (LISTEN)
sshd       1404            root    4u  IPv6    23574      0t0  TCP *:22 (LISTEN)
redis-ser  9815        perforce    6u  IPv4   277758      0t0  TCP localhost:7379 (LISTEN)
apache2   15373            root    4u  IPv6 24369079      0t0  TCP *:80 (LISTEN)
apache2   15373            root    6u  IPv6 24369083      0t0  TCP *:443 (LISTEN)
master    16229            root   13u  IPv4  3078154      0t0  TCP localhost:25 (LISTEN)
master    16229            root   14u  IPv6  3078155      0t0  TCP ip6-localhost:25 (LISTEN)
apache2   16500        www-data    4u  IPv6 24369079      0t0  TCP *:80 (LISTEN)
apache2   16500        www-data    6u  IPv6 24369083      0t0  TCP *:443 (LISTEN)
apache2   16501        www-data    4u  IPv6 24369079      0t0  TCP *:80 (LISTEN)
apache2   16501        www-data    6u  IPv6 24369083      0t0  TCP *:443 (LISTEN)
apache2   16502        www-data    4u  IPv6 24369079      0t0  TCP *:80 (LISTEN)
apache2   16502        www-data    6u  IPv6 24369083      0t0  TCP *:443 (LISTEN)
apache2   16529        www-data    4u  IPv6 24369079      0t0  TCP *:80 (LISTEN)
apache2   16529        www-data    6u  IPv6 24369083      0t0  TCP *:443 (LISTEN)
apache2   16704        www-data    4u  IPv6 24369079      0t0  TCP *:80 (LISTEN)
apache2   16704        www-data    6u  IPv6 24369083      0t0  TCP *:443 (LISTEN)
apache2   16854        www-data    4u  IPv6 24369079      0t0  TCP *:80 (LISTEN)
apache2   16854        www-data    6u  IPv6 24369083      0t0  TCP *:443 (LISTEN)
apache2   16858        www-data    4u  IPv6 24369079      0t0  TCP *:80 (LISTEN)
apache2   16858        www-data    6u  IPv6 24369083      0t0  TCP *:443 (LISTEN)
apache2   16859        www-data    4u  IPv6 24369079      0t0  TCP *:80 (LISTEN)
apache2   16859        www-data    6u  IPv6 24369083      0t0  TCP *:443 (LISTEN)
apache2   16860        www-data    4u  IPv6 24369079      0t0  TCP *:80 (LISTEN)
apache2   16860        www-data    6u  IPv6 24369083      0t0  TCP *:443 (LISTEN)
apache2   16862        www-data    4u  IPv6 24369079      0t0  TCP *:80 (LISTEN)
apache2   16862        www-data    6u  IPv6 24369083      0t0  TCP *:443 (LISTEN)
apache2   16890        www-data    4u  IPv6 24369079      0t0  TCP *:80 (LISTEN)
apache2   16890        www-data    6u  IPv6 24369083      0t0  TCP *:443 (LISTEN)
apache2   17743        www-data    4u  IPv6 24369079      0t0  TCP *:80 (LISTEN)
apache2   17743        www-data    6u  IPv6 24369083      0t0  TCP *:443 (LISTEN)
2 Likes

It seems the problem may be within Apache (no surprise).

Let's start with:
apachectl -S
curl -4 ifconfig.co
curl -6 ifconfig.co

3 Likes
apachectl -S
AH00526: Syntax error on line 14 of /etc/apache2/sites-enabled/perforce-swarm-site.conf:
SSLCertificateFile: file '/etc/letsencrypt/live/swarm.siteinteldev.com/fullchain.pem' does not exist or is empty
Action '-S' failed.
The Apache error log may have more information.

curl -4 ifconfig.co
<html>
<head><title>522 Origin Connection Time-out</title></head>
<body bgcolor="white">
<center><h1>522 Origin Connection Time-out</h1></center>
<hr><center>cloudflare-nginx</center>
</body>
</html>

curl -6 ifconfig.co
curl: (7) Couldn't connect to server
2 Likes

Try:
sudo apachectl -S
ls -l /etc/letsencrypt/live/swarm.siteinteldev.com/

3 Likes
sudo apachectl -S

VirtualHost configuration:
*:443 swarm.siteinteldev.com (/etc/apache2/sites-enabled/perforce-swarm-site.conf:1)
*:80 is a NameVirtualHost
default server helix-swarm.internal.cloudapp.net (/etc/apache2/sites-enabled/000-default.conf:1)
port 80 namevhost helix-swarm.internal.cloudapp.net (/etc/apache2/sites-enabled/000-default.conf:1)
port 80 namevhost swarm.siteinteldev.com (/etc/apache2/sites-enabled/perforce-swarm-site.conf:19)
alias localhost
ServerRoot: "/etc/apache2"
Main DocumentRoot: "/var/www/html"
Main ErrorLog: "/var/log/apache2/error.log"
Mutex watchdog-callback: using_defaults
Mutex rewrite-map: using_defaults
Mutex ssl-stapling-refresh: using_defaults
Mutex ssl-stapling: using_defaults
Mutex ssl-cache: using_defaults
Mutex default: dir="/var/run/apache2/" mechanism=default
Mutex mpm-accept: using_defaults
PidFile: "/var/run/apache2/apache2.pid"
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
User: name="www-data" id=33
Group: name="www-data" id=33

sudo ls -l /etc/letsencrypt/live/swarm.siteinteldev.com/

total 4
-rw-r--r-- 1 root root 692 Mar 5 01:41 README
lrwxrwxrwx 1 root root 46 Mar 5 01:41 cert.pem -> ../../archive/swarm.siteinteldev.com/cert1.pem
lrwxrwxrwx 1 root root 47 Mar 5 01:41 chain.pem -> ../../archive/swarm.siteinteldev.com/chain1.pem
lrwxrwxrwx 1 root root 51 Mar 5 01:41 fullchain.pem -> ../../archive/swarm.siteinteldev.com/fullchain1.pem
lrwxrwxrwx 1 root root 49 Mar 5 01:41 privkey.pem -> ../../archive/swarm.siteinteldev.com/privkey1.pem

1 Like

Ok how about:
sudo netstat -pant | grep -E ':80|:443'
sudo cat /etc/apache2/sites-enabled/perforce-swarm-site.conf

3 Likes
sudo netstat -pant | grep -E ':80|:443'
tcp        0      0 10.3.0.5:40276          168.63.129.16:8037      ESTABLISHED 4480/NetworkWatcher
tcp        0      0 10.3.0.5:52154          168.63.129.16:80        ESTABLISHED 4480/NetworkWatcher
tcp        0      0 127.0.0.1:46388         127.0.0.1:80            TIME_WAIT   -
tcp6       0      0 :::80                   :::*                    LISTEN      15373/apache2
tcp6       0      0 :::443                  :::*                    LISTEN      15373/apache2
tcp6       1      0 127.0.0.1:80            127.0.0.1:46000         CLOSE_WAIT  23657/apache2
tcp6       1      0 127.0.0.1:80            127.0.0.1:45802         CLOSE_WAIT  23681/apache2
tcp6       1      0 127.0.0.1:80            127.0.0.1:45900         CLOSE_WAIT  23391/apache2

sudo cat /etc/apache2/sites-enabled/perforce-swarm-site.conf
<VirtualHost _default_:443>
    SSLEngine on

    ServerName swarm.siteinteldev.com
    ErrorLog "/var/log/apache2/swarm.error_log"
    CustomLog "/var/log/apache2/swarm.access_log"common
    DocumentRoot "/opt/perforce/swarm/public"

    <Directory "/opt/perforce/swarm/public">
        AllowOverride All
        Require all granted
    </Directory>

    SSLCertificateFile /etc/letsencrypt/live/swarm.siteinteldev.com/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/swarm.siteinteldev.com/privkey.pem
    Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>

<VirtualHost *:80>
    ServerName swarm.siteinteldev.com
    ServerAlias localhost
    ErrorLog "/var/log/apache2/swarm.error_log"
    CustomLog "/var/log/apache2/swarm.access_log" common
    DocumentRoot "/opt/perforce/swarm/public"

    <Directory "/opt/perforce/swarm/public">
        AllowOverride All
        Require all granted
    </Directory>

</VirtualHost>
2 Likes

Yeah, so something is stopping connections from the Internet from reaching your server.

I don't know much about Azure but I would guess that there's somehow still another Azure setting that needs to be changed to allow the incoming connection to succeed.

(In case anyone thinks the tcp6 might have something to do with it, I don't think this is the case, as I ran nc -6 -l 11111 and confirmed that this does show up as a tcp6 socket but that a connection using IPv4 still connects to it. So the tcp6 socket listening on ::: can still receive connections that were made over IPv4.)

Edit: Also, kind of like AWS, Azure is apparently giving you an internal IPv4 address which is not the same as your public address. I don't know what setting you have to use to ensure that this is mapped correctly—and I don't know whether there even is a setting for this outside of the security groups thing—but you should probably double-check that this is allowed to the extent possible. I would also suggest double-checking that 104.208.24.248 really is your own Azure-issued public IPv4 address and that you didn't make a typo when copying it into your DNS records or anything. (For me, the https connection currently doesn't give a 404, but gives the same timeout as the http connection!)

4 Likes

Thank you Rudy @rg305 and Seth @schoen . It turns out that there two Azure network security groups in the play. One is associated with the network interface of the server, and the other is associated with the subnet. The effective inbound rules are a combination of the two NSG's. The 2nd NSG allows 443 but not 80. That's the cause of the issue. The cert is now renewed. Thank you both for your help!

4 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.