CertBot, Ubuntu 16 LTS, Apache/2.4.18 (Ubuntu) stopped working

Help
1

My domain is: arno.com

I ran this command: certbot
** certbot -v certonly --apache**

It produced this output:
root@pluto# certbot -v certonly --apache
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache

Which names would you like to activate HTTPS for?
We recommend selecting either all domains, or all domains in a VirtualHost/server block.

1: arno.com
2: crystalridge.arno.com
3: facebook.crystalridge.arno.com
4: www.facebook.crystalridge.arno.com
5: www.crystalridge.arno.com
6: oc.arno.com
7: pluto.arno.com
8: ubuntu.arno.com
9: wilma.arno.com
10: www.wilma.arno.com

Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): 1 2 3 4 5 9 10
Requesting a certificate for arno.com and 6 more domains
Performing the following challenges:
http-01 challenge for arno.com
http-01 challenge for crystalridge.arno.com
http-01 challenge for facebook.crystalridge.arno.com
http-01 challenge for wilma.arno.com
http-01 challenge for www.crystalridge.arno.com
http-01 challenge for www.facebook.crystalridge.arno.com
http-01 challenge for www.wilma.arno.com
Waiting for verification...
Challenge failed for domain arno.com
Challenge failed for domain crystalridge.arno.com
Challenge failed for domain facebook.crystalridge.arno.com
Challenge failed for domain wilma.arno.com
Challenge failed for domain www.crystalridge.arno.com
Challenge failed for domain www.facebook.crystalridge.arno.com
Challenge failed for domain www.wilma.arno.com
http-01 challenge for arno.com
http-01 challenge for crystalridge.arno.com
http-01 challenge for facebook.crystalridge.arno.com
http-01 challenge for wilma.arno.com
http-01 challenge for www.crystalridge.arno.com
http-01 challenge for www.facebook.crystalridge.arno.com
http-01 challenge for www.wilma.arno.com

Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:
Domain: arno.com
Type: connection
Detail: 50.47.248.66: Fetching http://arno.com/.well-known/acme-challenge/VrFLePHYkGMEfM0RidrufjQG4TAPxx8dGgAFRU1A_Xo: Connection reset by peer

Domain: crystalridge.arno.com
Type: connection
Detail: 50.47.248.66: Fetching http://crystalridge.arno.com/.well-known/acme-challenge/LzJf4XthiJxy29tzpBzfLKL-x_4WULKymTkCdmqVqjw: Connection reset by peer

Domain: facebook.crystalridge.arno.com
Type: connection
Detail: 50.47.248.66: Fetching http://facebook.crystalridge.arno.com/.well-known/acme-challenge/XvVshVxzcwdg8xD23He8sWCa_tg75AGULDdxhQMo4hI: Connection reset by peer

Domain: wilma.arno.com
Type: connection
Detail: 50.47.248.66: Fetching http://wilma.arno.com/.well-known/acme-challenge/zondzuZsq0FVZ-JhTwLOkSBLpO7QtoeAybaVq_tAnEY: Connection reset by peer

Domain: www.crystalridge.arno.com
Type: connection
Detail: 50.47.248.66: Fetching http://www.crystalridge.arno.com/.well-known/acme-challenge/MUGd4JfeizSPFAhsUCUbBWgItxR9W7dcu2r2Nnra1pQ: Connection reset by peer

Domain: www.facebook.crystalridge.arno.com
Type: connection
Detail: 50.47.248.66: Fetching http://www.facebook.crystalridge.arno.com/.well-known/acme-challenge/RP4QS_8JDvW4A6BiaOmSdJbsHKl-uLEUM5cJ504EiD8: Connection reset by peer

Domain: www.wilma.arno.com
Type: connection
Detail: 50.47.248.66: Fetching http://www.wilma.arno.com/.well-known/acme-challenge/JFGsGAL2CR9KdLt7DQxJz1waK_siODYhvC1epjL2w5Y: Connection reset by peer

Hint: The Certificate Authority failed to verify the temporary Apache configuration changes made by Certbot. Ensure that the listed domains point to this Apache server and that it is accessible from the internet.

Cleaning up challenges
Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

My web server is (include version):
Apache/2.4.18 (Ubuntu)

The operating system my web server runs on is (include version):

cat /etc/lsb-release

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.7 LTS"

My hosting provider, if applicable, is:
Self

I can login to a root shell on my machine (yes or no, or I don't know):
yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

certbot --version

certbot 2.7.4

certbot worked well and automatically for 10 years, then suddenly stopped working, now my certificates are expired. It used to be that I could run "certbot" from the command shell and it would update. Now even this doesn't work.

I followed directions from Certbot Instructions
uninstalled my apt version of certbot, installed snapd and the snap version of certbot, and the results are the same and listed above.

I'm in the middle of building an Ubuntu 22.04 LTS system in my spare time, while working for a startup (spare time nonexistent). Any more ideas about how to get this running under Ubuntu 16.04 LTS until I can move over to 22.04? It was running fine 3 months ago from the command shell. What changed? What can I do to fix this?

2

Hi @craiga, and welcome to the LE community forum :slight_smile:

It seems that LE is unable to reach your web server via HTTP.
Let's Debug also can't: Let's Debug (letsdebug.net)

Oddly enough, I can reach it:

curl -Ii http://arno.com/
HTTP/1.1 200 OK
Date: Sat, 25 Nov 2023 07:44:05 GMT
Server: Apache/2.4.18 (Ubuntu)
Last-Modified: Tue, 29 Aug 2023 06:05:24 GMT
ETag: "21ba-6040998bfed74"
Accept-Ranges: bytes
Content-Length: 8634
Vary: Accept-Encoding
Content-Type: text/html

It would seem that there is some sort of IP checking [like a firewall rule] that is blocking only some IPs.

3

@craiga You almost certainly have a Palo Alto Networks brand firewall blocking the ACME HTTP Challenge. I can reach your domain unless I use a user-agent the same as Let's Encrypt uses (which Let's Debug also uses).

You need to change the Palo Alto firewall to allow "acme-protocol" in the Applications section. Show your network team the two requests below. Both should result in a 404 but you can see the one with the user-agent like LE fails. We have seen this problem often.

curl -i http://arno.com/.well-known/acme-challenge/Test404
HTTP/1.1 404 Not Found
Server: Apache/2.4.18 (Ubuntu)

curl -i http://arno.com/.well-known/acme-challenge/Test404 -A "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
curl: (56) Recv failure: Connection reset by peer
4

@MikeMcQ and @rg305 Perfect!

I'm a one-man show, so I'll put my "network team" hat on and say Thank you! from myself, my family [552 living], and HOA [82 households]. (people affected by this)

Yes, I run a Palo Alto firewall, and don't think I would have guessed the "acme-protocol" in the Applications section solution. Now I need to update my PA config documentation so this exact situation won't happen again.

Fortunately, or unfortunately, I develop new technology that state level actors would like to acquire. IP theft or disruption is their business model. So from a business perspective, running a firewall solution any less capable (easier) than a PA solution isn't an option.

Thank you again for your efforts. Now I can get back to saving the world. :wink:

=========Successful Run ==============
root@pluto:~# certbot -v certonly --apache
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache

Which names would you like to activate HTTPS for?
We recommend selecting either all domains, or all domains in a VirtualHost/server block.

1: arno.com
2: crystalridge.arno.com
3: facebook.crystalridge.arno.com
4: www.facebook.crystalridge.arno.com
5: www.crystalridge.arno.com
6: oc.arno.com
7: pluto.arno.com
8: ubuntu.arno.com
9: wilma.arno.com
10: www.wilma.arno.com

Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): 1 2 3 4 5 9 10
Requesting a certificate for arno.com and 6 more domains
Performing the following challenges:
http-01 challenge for arno.com
http-01 challenge for crystalridge.arno.com
http-01 challenge for facebook.crystalridge.arno.com
http-01 challenge for wilma.arno.com
http-01 challenge for www.crystalridge.arno.com
http-01 challenge for www.facebook.crystalridge.arno.com
http-01 challenge for www.wilma.arno.com
Waiting for verification...
Cleaning up challenges
Reloading apache server after certificate issuance

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/arno.com-0002/fullchain.pem
Key is saved at: /etc/letsencrypt/live/arno.com-0002/privkey.pem
This certificate expires on 2024-02-23.
These files will be updated when the certificate renews.
Certbot has set up a scheduled task to automatically renew this certificate in the background.

If you like Certbot, please consider supporting our work by: