After cert delete, can't create new cert

Hello! Like a dummy, I let my certificate expire and couldn't renew it. I deleted the cert, not realizing I was supposed to revoke it first. Since then, I've been unable to create a new cert :frowning:

Prior to my cert snafu, the site was working via https for months. I haven't touched the apache2.conf since it was first up and running.

My domain is: fulcrum.mu

I ran this command: certbot --apache -d "www.fulcrum.mu,fulcrum.mu"

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for fulcrum.mu
http-01 challenge for www.fulcrum.mu
Waiting for verification...
Challenge failed for domain fulcrum.mu
Challenge failed for domain www.fulcrum.mu
http-01 challenge for fulcrum.mu
http-01 challenge for www.fulcrum.mu
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: fulcrum.mu
   Type:   unauthorized
   Detail: Invalid response from
   http://fulcrum.mu/.well-known/acme-challenge/CMKIf_VWRGvRzsrere1GaAJcddkwicgc--BkMY6MVvU
   [143.110.234.63]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML
   2.0//EN\">\n<html><head>\n<title>404 Not
   Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p"

   Domain: www.fulcrum.mu
   Type:   unauthorized
   Detail: Invalid response from
   http://www.fulcrum.mu/.well-known/acme-challenge/4YFLDjiOzWtjKez6u2MpDiDKTvSt-danB9qfqghQGCM
   [143.110.234.63]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML
   2.0//EN\">\n<html><head>\n<title>404 Not
   Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

My web server is (include version): Server Version: Apache/2.4.41 (Ubuntu) OpenSSL/1.1.1j

The operating system my web server runs on is (include version): Ubuntu 20.04

My hosting provider, if applicable, is: Digital Ocean

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no. I manage through ssh.

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): 0.40.0

2 Likes

As an addendum, here's the appropriate sections from the apache logs.

From other_vhosts_access.log:

127.0.0.1:80 54.189.22.122 - - [12/Jun/2021:23:03:26 +0000] "GET /.well-known/acme-challenge/4YFLDjiOzWtjKez6u2MpDiDKTvSt-danB9qfqghQGCM HTTP/1.1" 404 456 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
127.0.0.1:80 64.78.149.164 - - [12/Jun/2021:23:03:26 +0000] "GET /.well-known/acme-challenge/CMKIf_VWRGvRzsrere1GaAJcddkwicgc--BkMY6MVvU HTTP/1.1" 404 452 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
127.0.0.1:80 18.116.86.117 - - [12/Jun/2021:23:03:26 +0000] "GET /.well-known/acme-challenge/CMKIf_VWRGvRzsrere1GaAJcddkwicgc--BkMY6MVvU HTTP/1.1" 404 452 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
127.0.0.1:80 3.142.122.14 - - [12/Jun/2021:23:03:26 +0000] "GET /.well-known/acme-challenge/4YFLDjiOzWtjKez6u2MpDiDKTvSt-danB9qfqghQGCM HTTP/1.1" 404 456 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
127.0.0.1:80 64.78.149.164 - - [12/Jun/2021:23:03:26 +0000] "GET /.well-known/acme-challenge/4YFLDjiOzWtjKez6u2MpDiDKTvSt-danB9qfqghQGCM HTTP/1.1" 404 456 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
127.0.0.1:80 18.197.97.115 - - [12/Jun/2021:23:03:26 +0000] "GET /.well-known/acme-challenge/4YFLDjiOzWtjKez6u2MpDiDKTvSt-danB9qfqghQGCM HTTP/1.1" 404 456 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

From error.log:

[Sat Jun 12 23:03:23.083775 2021] [mpm_prefork:notice] [pid 878872] AH00171: Graceful restart requested, doing restart
[Sat Jun 12 23:03:23.221310 2021] [mpm_prefork:notice] [pid 878872] AH00163: Apache/2.4.41 (Ubuntu) OpenSSL/1.1.1j configured -- resuming normal operations
[Sat Jun 12 23:03:23.221351 2021] [core:notice] [pid 878872] AH00094: Command line: '/usr/sbin/apache2'
[Sat Jun 12 23:03:27.592804 2021] [mpm_prefork:notice] [pid 878872] AH00171: Graceful restart requested, doing restart
[Sat Jun 12 23:03:27.736698 2021] [mpm_prefork:notice] [pid 878872] AH00163: Apache/2.4.41 (Ubuntu) OpenSSL/1.1.1j configured -- resuming normal operations
[Sat Jun 12 23:03:27.736723 2021] [core:notice] [pid 878872] AH00094: Command line: '/usr/sbin/apache2'
2 Likes

You were neither supposed to revoke it nor delete it. What made you think you were supposed to?

Well, that's interesting. It looks like you're getting validation requests from the secondary servers, but not the primary validation server. (I say this because you're supposed to get three connections from the secondary servers, and each of them to each of your two domain names (with the www. and without), and one connection from the primary server, and the error message from Let's Encrypt doesn't say "secondary" in it.)

I'd almost think it was another weird routing issue like this recent person who also had trouble with primary but not secondary validation, but in your case it looks like the primary requests are going somewhere, just a server that isn't expecting it somehow and is returning a 404.

This one is a puzzler.

3 Likes

Thanks for the reply :slight_smile:

To be honest, I'm not sure. The renewal kept failing, so I figured it'd be ok to delete the expired cert and just start over with a new one. It's quite possible the renewal failure was the same one we're seeing here with the creation of a new one.

I'm a software engineer, not a sysadmin/network admin. Definitely outside my wheelhouse on this.

3 Likes

From a technical perspective, there's no difference between a "renewed" certificate and a "new" certificate, in terms of what the request and response to Let's Encrypt's servers are. It's just a matter of convention of what your client software calls things, that you're making a "same" request as before so it's a renewal.

Certbot should just be automatically renewing things, so yes it's highly likely that it ran into the same problem renewing as you're seeing now. It looks like your first request was approx. 3 months ago, so this is your first "renewal". Did you have to do anything "weird" or unusual to get your initial certificate, or did it just all work straightforwardly.

Can you dig through your log files and see if your server has any entry anywhere else for the fourth expected validation request? It almost seems like some requests are going to your server but the fourth is going to some other server that isn't expecting it, but maybe it's actually going to some other virtual host or the like within your Apache config?

3 Likes

I seem to recall it worked without issue. I was actually quite pleased.

So, a few minutes before the log messages I shared above, I see the following:

127.0.0.1:80 34.221.255.206 - - [12/Jun/2021:22:46:28 +0000] "GET /.well-known/acme-challenge/SQ9RA-rCyRSp4h6yWmb3gYTcoWfGZq6jJdQnidsPOD4 HTTP/1.1" 404 452 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
127.0.0.1:80 64.78.149.164 - - [12/Jun/2021:22:46:28 +0000] "GET /.well-known/acme-challenge/SQ9RA-rCyRSp4h6yWmb3gYTcoWfGZq6jJdQnidsPOD4 HTTP/1.1" 404 452 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
127.0.0.1:80 18.116.86.117 - - [12/Jun/2021:22:46:29 +0000] "GET /.well-known/acme-challenge/SQ9RA-rCyRSp4h6yWmb3gYTcoWfGZq6jJdQnidsPOD4 HTTP/1.1" 404 452 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
127.0.0.1:80 18.116.86.117 - - [12/Jun/2021:22:46:29 +0000] "GET /.well-known/acme-challenge/iRInoRkRAkPj1uU85lhM9iDb17caN2z2fTuntabZe_Y HTTP/1.1" 404 456 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
127.0.0.1:80 34.221.255.206 - - [12/Jun/2021:22:46:29 +0000] "GET /.well-known/acme-challenge/iRInoRkRAkPj1uU85lhM9iDb17caN2z2fTuntabZe_Y HTTP/1.1" 404 456 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
127.0.0.1:80 18.184.29.122 - - [12/Jun/2021:22:46:29 +0000] "GET /.well-known/acme-challenge/SQ9RA-rCyRSp4h6yWmb3gYTcoWfGZq6jJdQnidsPOD4 HTTP/1.1" 404 452 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
127.0.0.1:80 64.78.149.164 - - [12/Jun/2021:22:46:29 +0000] "GET /.well-known/acme-challenge/iRInoRkRAkPj1uU85lhM9iDb17caN2z2fTuntabZe_Y HTTP/1.1" 404 456 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
127.0.0.1:80 18.197.97.115 - - [12/Jun/2021:22:46:29 +0000] "GET /.well-known/acme-challenge/iRInoRkRAkPj1uU85lhM9iDb17caN2z2fTuntabZe_Y HTTP/1.1" 404 456 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

A pair of 4 requests.

2 Likes

What's the output of:

sudo apachectl -t -D DUMP_VHOSTS
3 Likes
root@fulcrum:/var/log/apache2# apachectl -t -D DUMP_VHOSTS
VirtualHost configuration:
*:80                   127.0.0.1 (/etc/apache2/sites-enabled/000-default.conf:1)
root@fulcrum:/var/log/apache2#
2 Likes

That should really just be working, I can't see what's wrong.

You might want to force-kill Apache, just in case something has gone wrong with the service:

systemctl stop apache2
kill -9 $(pgrep apache2)

and try again.

If that doesn't work, I would like to see the contents of /etc/apache2/sites-enabled/000-default.conf.

2 Likes

Thanks so much for spending time on this.

Killed apache as described, confirmed there were no apache processes running. Tried certbot --apache -d "www.fulcrum.mu,fulcrum.mu" and got the same failure.

Restarted apache via systemctl, re-ran certbot, same result.

Here's the contents of 000-default.conf:

root@fulcrum:/etc/apache2/sites-enabled# cat 000-default.conf
<VirtualHost *:80>
        ServerAdmin webmaster@localhost
        DocumentRoot /var/www/html

        <Directory /var/www/html/>
            Options Indexes FollowSymLinks
            AllowOverride All
            Require all granted
        </Directory>

        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined

        <IfModule mod_dir.c>
            DirectoryIndex index.php index.pl index.cgi index.html index.xhtml index.htm
        </IfModule>

RewriteEngine on
RewriteCond %{SERVER_NAME} =www.fulcrum.mu [OR]
RewriteCond %{SERVER_NAME} =fulcrum.mu
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>
root@fulcrum:/etc/apache2/sites-enabled#
3 Likes

Are you sure 100% sure this is the right server and IP address?

With those rewrite rules, I should be getting redirected to HTTPS when I make a request to your domain. But I don't:

$ curl -X GET -I fulcrum.mu
HTTP/1.1 200 OK
Date: Sun, 13 Jun 2021 00:29:49 GMT
Server: Apache/2.4.41 (Ubuntu)
Vary: Accept-Encoding
Content-Length: 1388
Content-Type: text/html;charset=UTF-8

i.e. if I copy your configuration verbatim, here is how the same request looks for me:

# curl -X GET -I --resolve fulcrum.mu:80:127.0.0.1 fulcrum.mu/
HTTP/1.1 301 Moved Permanently
Date: Sun, 13 Jun 2021 00:40:07 GMT
Server: Apache/2.4.41 (Ubuntu)
Location: https://fulcrum.mu/
Content-Length: 303
Content-Type: text/html; charset=iso-8859-1

I'm also assuming you didn't redact anything from the apachectl output.

3 Likes

fulcrum.mu and www.fulcrum.mu both resolve to 143.110.234.63

No redaction. That's why I included the command prompt before and after the output.

Interestingly, I just checked netstat -an | grep LISTEN and I see ports 80 and 443 open for ipv6, but not for ipv4.

2 Likes

This is normal; on Linux tcp6 sockets will by default work for both IPv4 and IPv6.

I'm still not sure what's happening, other than something is funky with the Apache config.

You could try just get and install the certificate using webroot:

certbot run -a webroot -w /var/www/html -i apache -d fulcrum.mu -d www.fulcrum.mu

For the installer to work, you might first need to add:

ServerName fulcrum.mu
ServerAlias www.fulcrum.mu

to the VirtualHost.

3 Likes

Success! I can now access the site via https, and the ssllabs.com tests look good. :slight_smile:

Thank you so much!

Now I just need to figure out why the redirect from HTTP to HTTPS isn't working, but that's probably outside the scope of this forum.

3 Likes

Great!

Unfortunately it's going to be the same reason that certbot --apache isn't working; Apache isn't actually using that :80 virtualhost for some reason.

But you could probably chuck that redirect into /var/www/html/.htaccess as a roundabout way to make it work.

2 Likes

I'll poke around at the redirect stuff. For now, I can have the few people that are working on the stuff on that site explicitly use HTTPS until I get the redirect thing working (eg. the example at How to redirect HTTP to HTTPS Using .htaccess isn't working after I drop a .htaccess file in /var/www/html with my proper domain name replacing the yourdomain.com example.)

Thanks again for all your help!

3 Likes

I suspect that you might have multiple versions of Apache running.
OR at least the one that is running is likely using another configuration file.

2 Likes

Avoid .htaccess files whenever possible...

https://httpd.apache.org/docs/current/rewrite/avoid.html

2 Likes

My post was linked to regarding the secondary/primary servers, and since I'm not an apache guy, I don't understand the changes and output above, so I have to ask what I feel is a silly question -

Was the issue related to my post, or was the cause determined to be something else?

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.