Problem migrating away from TLS-SNI-01 with Certbot

Help
1

Hi, I got the email “Action required: Let’s Encrypt certificate renewals” and attempted to follow this guide: How to stop using TLS-SNI-01 with Certbot.

“If the version is less than 0.28, you need to upgrade your Certbot. Visit…”

Here I ran the commands

$ sudo apt-get update
$ sudo apt-get install software-properties-common
$ sudo add-apt-repository universe
$ sudo add-apt-repository ppa:certbot/certbot
$ sudo apt-get update
$ sudo apt-get install certbot python-certbot-apache

I stopped at the Get started section since I had a certificate.

$ certbot --version
certbot 0.28.0

Then I ran the recommended

$ sudo sh -c "sed -i.bak -e 's/^\(pref_challs.*\)tls-sni-01\(.*\)/\1http-01\2/g' /etc/letsencrypt/renewal/*; rm -f /etc/letsencrypt/renewal/*.bak"

And got

$ sudo certbot renew --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/naturalhazardsgroup.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator apache, Installer apache
Starting new HTTPS connection (1): acme-staging-v02.api.letsencrypt.org
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for naturalhazardsgroup.com
http-01 challenge for www.naturalhazardsgroup.com
Waiting for verification...
Cleaning up challenges
Attempting to renew cert (naturalhazardsgroup.com) from /etc/letsencrypt/renewal/naturalhazardsgroup.com.conf produced an unexpected error: Failed authorization procedure. naturalhazardsgroup.com (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://naturalhazardsgroup.com/.well-known/acme-challenge/3u-7Rn_H9vH1LxBBb0X5f_7BzmXykWwFGmzEJ2wv05g: "\n\n\n<!DOCTYPE html>\n\n<html class=\"no-js\" lang=\"en-US\">\n\n<head>\n  \n<meta charset=\"UTF-8\">\n<meta name=\"viewport\" content=\"width=dev". Skipping.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/new.naturalhazardsgroup.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator apache, Installer apache
Starting new HTTPS connection (1): acme-staging-v02.api.letsencrypt.org
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for new.naturalhazardsgroup.com
Waiting for verification...
Cleaning up challenges
Attempting to renew cert (new.naturalhazardsgroup.com) from /etc/letsencrypt/renewal/new.naturalhazardsgroup.com.conf produced an unexpected error: Failed authorization procedure. new.naturalhazardsgroup.com (http-01): urn:ietf:params:acme:error:dns :: DNS problem: NXDOMAIN looking up A for new.naturalhazardsgroup.com. Skipping.
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/naturalhazardsgroup.com/fullchain.pem (failure)
  /etc/letsencrypt/live/new.naturalhazardsgroup.com/fullchain.pem (failure)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/naturalhazardsgroup.com/fullchain.pem (failure)
  /etc/letsencrypt/live/new.naturalhazardsgroup.com/fullchain.pem (failure)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: naturalhazardsgroup.com
   Type:   unauthorized
   Detail: Invalid response from
   http://naturalhazardsgroup.com/.well-known/acme-challenge/3u-7Rn_H9vH1LxBBb0X5f_7BzmXykWwFGmzEJ2wv05g:
   "\n\n\n<!DOCTYPE html>\n\n<html class=\"no-js\"
   lang=\"en-US\">\n\n<head>\n  \n<meta charset=\"UTF-8\">\n<meta
   name=\"viewport\" content=\"width=dev"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.
 - The following errors were reported by the server:

   Domain: new.naturalhazardsgroup.com
   Type:   None
   Detail: DNS problem: NXDOMAIN looking up A for
   new.naturalhazardsgroup.com
 - Your account credentials have been saved in your Certbot
   configuration directory at /etc/letsencrypt. You should make a
   secure backup of this folder now. This configuration directory will
   also contain certificates and private keys obtained by Certbot so
   making regular backups of this folder is ideal.

There used to be a site at new.naturalhazardsgroup.com and a DNS record, this was removed at some point. I tried to clean it up maybe foolishly

$ sudo rm -r etc/letsencrypt/archive/new.naturalhazardsgroup.com
$ sudo rm -r /etc/letsencrypt/live/new.naturalhazardsgroup.com/
$ sudo rm -r /etc/letsencrypt/renewal/new.naturalhazardsgroup.com.conf

I have the remaining error:

$ sudo certbot renew --dry-run
_[removed - says I am not allowed to post more than 20 links on forum]_
1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: naturalhazardsgroup.com
   Type:   unauthorized
   Detail: Invalid response from
   http://naturalhazardsgroup.com/.well-known/acme-challenge/OnwYyDAMFKUg3v15FvFVwPxkAMJHxMIjmj70eIG_X1k:
   "\n\n\n<!DOCTYPE html>\n\n<html class=\"no-js\"
   lang=\"en-US\">\n\n<head>\n  \n<meta charset=\"UTF-8\">\n<meta
   name=\"viewport\" content=\"width=dev"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

My web server is: Apache/2.4.18
The operating system my web server runs on is: Ubuntu 16.04
My hosting provider is: DigitalOcean
I can login to a root shell on my machine: yes
I’m using a control panel to manage my site: no
The version of my client is: certbot 0.28.0

2

0.28.0 version is good.

Please show:
certbot certificates

This was a bad idea:

There are built-in commands for that:
certbot delete --cert-name {name-of-cert}

3

It says:

$ certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Found the following certs:
  Certificate Name: naturalhazardsgroup.com
    Domains: naturalhazardsgroup.com www.naturalhazardsgroup.com
    Expiry Date: 2019-05-11 22:26:49+00:00 (VALID: 89 days)
    Certificate Path: /etc/letsencrypt/live/naturalhazardsgroup.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/naturalhazardsgroup.com/privkey.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Wish I had known about the other command :frowning:

4

So far so good...
Please show:
ls -l /etc/letsencrypt/renewal/

We can still "recover"/"overcome" it.

5

Here:

$ ls -l /etc/letsencrypt/renewal/
total 4
-rw-r--r-- 1 root root 574 Feb 10 23:26 naturalhazardsgroup.com.conf
6

Not sure where it still sees this:

Try:
grep -Eri 'new.natural' /etc/apache2

7

Ah no - as I wrote “removed - says I am not allowed to post more than 20 links on forum”

That output is from the first time I ran the command

I will try to post the complete output as it is now and hope that the forum allows me. It is:

$  sudo certbot renew --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/naturalhazardsgroup.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator apache, Installer apache
Starting new HTTPS connection (1): acme-staging-v02.api.letsencrypt.org
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for naturalhazardsgroup.com
http-01 challenge for www.naturalhazardsgroup.com
Waiting for verification...
Cleaning up challenges
Attempting to renew cert (naturalhazardsgroup.com) from /etc/letsencrypt/renewal/naturalhazardsgroup.com.conf produced an unexpected error: Failed authorization procedure. naturalhazardsgroup.com (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://naturalhazardsgroup.com/.well-known/acme-challenge/XH_U1-a4kn4QYMC6AWg_T2woDQEx7mqR9iWCv9pvN9o: "\n\n\n<!DOCTYPE html>\n\n<html class=\"no-js\" lang=\"en-US\">\n\n<head>\n  \n<meta charset=\"UTF-8\">\n<meta name=\"viewport\" content=\"width=dev". Skipping.
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/naturalhazardsgroup.com/fullchain.pem (failure)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/naturalhazardsgroup.com/fullchain.pem (failure)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: naturalhazardsgroup.com
   Type:   unauthorized
   Detail: Invalid response from
   http://naturalhazardsgroup.com/.well-known/acme-challenge/XH_U1-a4kn4QYMC6AWg_T2woDQEx7mqR9iWCv9pvN9o:
   "\n\n\n<!DOCTYPE html>\n\n<html class=\"no-js\"
   lang=\"en-US\">\n\n<head>\n  \n<meta charset=\"UTF-8\">\n<meta
   name=\"viewport\" content=\"width=dev"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

EDIT:

$ grep -Eri 'new.natural' /etc/apache2
/etc/apache2/sites-available/000-default.conf:RewriteCond %{SERVER_NAME} =new.naturalhazardsgroup.com [OR]
/etc/apache2/sites-available/000-default-le-ssl.conf:ServerName new.naturalhazardsgroup.com
8

OK.
Let’s focus on that.

Please show the vhost config for that site:
http://naturalhazardsgroup.com

And also those file:
/etc/apache2/sites-available/000-default.conf
/etc/apache2/sites-available/000-default-le-ssl.conf

And to be sure which are being used:
ls -l /etc/apache2/sites-enabled/

9

Trying to remember where the vhost is… :confused:

$  cat /etc/apache2/sites-available/000-default.conf
<VirtualHost *:80>
    ServerName naturalhazardsgroup.com
    Redirect permanent / https://www.naturalhazardsgroup.com/
RewriteEngine on
RewriteCond %{SERVER_NAME} =new.naturalhazardsgroup.com [OR]
RewriteCond %{SERVER_NAME} =naturalhazardsgroup.com
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>

<VirtualHost *:80>
                ServerName www.naturalhazardsgroup.com
        DocumentRoot /var/www/html

        <Directory /var/www/html/>
            Options FollowSymLinks
            AllowOverride All
            Require all granted
        </Directory>

        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined
RewriteEngine on
RewriteCond %{SERVER_NAME} =www.naturalhazardsgroup.com
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>

$ cat /etc/apache2/sites-available/000-default.conf
<VirtualHost *:80>
ServerName naturalhazardsgroup.com
Redirect permanent / https://www.naturalhazardsgroup.com/
RewriteEngine on
RewriteCond %{SERVER_NAME} =new.naturalhazardsgroup.com [OR]
RewriteCond %{SERVER_NAME} =naturalhazardsgroup.com
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>

<VirtualHost *:80>
            ServerName www.naturalhazardsgroup.com
    DocumentRoot /var/www/html

    <Directory /var/www/html/>
        Options FollowSymLinks
        AllowOverride All
        Require all granted
    </Directory>

    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined
RewriteEngine on
RewriteCond %{SERVER_NAME} =www.naturalhazardsgroup.com
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>

root@mine-frankfurt:/var#
root@mine-frankfurt:/var# ls -l /etc/apache2/sites-enabled/
total 0
lrwxrwxrwx 1 www-data www-data 35 Jan 15  2018 000-default.conf -> ../sites-available/000-default.conf
lrwxrwxrwx 1 root     root     52 Jul 15  2018 000-default-le-ssl.conf -> /etc/apache2/sites-available/000-default-le-ssl.conf
root@mine-frankfurt:/var#
root@mine-frankfurt:/var#
root@mine-frankfurt:/var#
root@mine-frankfurt:/var# cat /etc/apache2/sites-available/000-default-le-ssl.conf
<IfModule mod_ssl.c>
<VirtualHost *:443>
    ServerAdmin webmaster@localhost
    DocumentRoot /var/www/html

    <Directory /var/www/html/>
        Options FollowSymLinks
        AllowOverride All
        Require all granted
    </Directory>

    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined
ServerName new.naturalhazardsgroup.com
Include /etc/letsencrypt/options-ssl-apache.conf
ServerAlias naturalhazardsgroup.com
SSLCertificateFile /etc/letsencrypt/live/naturalhazardsgroup.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/naturalhazardsgroup.com/privkey.pem
</VirtualHost>
</IfModule>
<IfModule mod_ssl.c>
<VirtualHost *:80>
ServerName naturalhazardsgroup.com
Redirect permanent / https://www.naturalhazardsgroup.com/
</VirtualHost>
</IfModule>
<IfModule mod_ssl.c>
<VirtualHost *:443>
            ServerName www.naturalhazardsgroup.com
    DocumentRoot /var/www/html

    <Directory /var/www/html/>
        Options FollowSymLinks
        AllowOverride All
        Require all granted
    </Directory>

    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined
Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateFile /etc/letsencrypt/live/naturalhazardsgroup.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/naturalhazardsgroup.com/privkey.pem
</VirtualHost>
</IfModule>


$  ls -l /etc/apache2/sites-enabled/
total 0
lrwxrwxrwx 1 www-data www-data 35 Jan 15  2018 000-default.conf -> ../sites-available/000-default.conf
lrwxrwxrwx 1 root     root     52 Jul 15  2018 000-default-le-ssl.conf -> /etc/apache2/sites-available/000-default-le-ssl.conf
10

Repetitive redundancy:

You need to fix this:

If there is no NEW, then remove the "new." form that line in file:
/etc/apache2/sites-available/000-default-le-ssl.conf
[and remove the ServerAlias as line well]

11

Ok, I have removed those 2 lines from the file. Should I do anything with the redundancy?

EDIT: hold on. I shouldn’t have removed the one line…

12

It's not breaking anything...
Your call.

13 
 $  cat 000-default-le-ssl.conf
    <IfModule mod_ssl.c>
    <VirtualHost *:443>
            ServerAdmin webmaster@localhost
            DocumentRoot /var/www/html

            <Directory /var/www/html/>
                Options FollowSymLinks
                AllowOverride All
                Require all granted
            </Directory>

            ErrorLog ${APACHE_LOG_DIR}/error.log
            CustomLog ${APACHE_LOG_DIR}/access.log combined
    ServerName naturalhazardsgroup.com
    Include /etc/letsencrypt/options-ssl-apache.conf
    SSLCertificateFile /etc/letsencrypt/live/naturalhazardsgroup.com/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/naturalhazardsgroup.com/privkey.pem
    </VirtualHost>
    </IfModule>
    <IfModule mod_ssl.c>
    <VirtualHost *:80>
        ServerName naturalhazardsgroup.com
        Redirect permanent / https://www.naturalhazardsgroup.com/
    </VirtualHost>
    </IfModule>
    <IfModule mod_ssl.c>
    <VirtualHost *:443>
                    ServerName www.naturalhazardsgroup.com
            DocumentRoot /var/www/html

            <Directory /var/www/html/>
                Options FollowSymLinks
                AllowOverride All
                Require all granted
            </Directory>

            ErrorLog ${APACHE_LOG_DIR}/error.log
            CustomLog ${APACHE_LOG_DIR}/access.log combined
    Include /etc/letsencrypt/options-ssl-apache.conf
    SSLCertificateFile /etc/letsencrypt/live/naturalhazardsgroup.com/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/naturalhazardsgroup.com/privkey.pem
    </VirtualHost>
    </IfModule>

I will leave the rendundant thing if not breaking anything.

14

Please remove this entire block:

<IfModule mod_ssl.c>
 <VirtualHost *:80>
  ServerName naturalhazardsgroup.com
  Redirect permanent / https://www.naturalhazardsgroup.com/
 </VirtualHost>
</IfModule>

from file:
/etc/apache2/sites-available/000-default-le-ssl.conf

15

Ok:

$  cat 000-default-le-ssl.conf
<IfModule mod_ssl.c>
<VirtualHost *:443>
        ServerAdmin webmaster@localhost
        DocumentRoot /var/www/html

        <Directory /var/www/html/>
            Options FollowSymLinks
            AllowOverride All
            Require all granted
        </Directory>

        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined
ServerName naturalhazardsgroup.com
Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateFile /etc/letsencrypt/live/naturalhazardsgroup.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/naturalhazardsgroup.com/privkey.pem
</VirtualHost>
</IfModule>
<IfModule mod_ssl.c>
<VirtualHost *:443>
                ServerName www.naturalhazardsgroup.com
        DocumentRoot /var/www/html

        <Directory /var/www/html/>
            Options FollowSymLinks
            AllowOverride All
            Require all granted
        </Directory>

        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined
Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateFile /etc/letsencrypt/live/naturalhazardsgroup.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/naturalhazardsgroup.com/privkey.pem
</VirtualHost>
</IfModule>
16

OK.

So now:
/etc/apache2/sites-available/000-default.conf
handles only the http (*:80)

and
/etc/apache2/sites-available/000-default-le-ssl.conf
handles only the https (*:443)

The http forwards to https (GOOD).

LE should follow that redirection to:
https://naturalhazardsgroup.com/.well-known/acme-challenge/{TOKEN-FILE}

Try it again now.

17

That makes sense. I don’t know how it came to be different.

18 
$ sudo certbot renew --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/naturalhazardsgroup.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator apache, Installer apache
Starting new HTTPS connection (1): acme-staging-v02.api.letsencrypt.org
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for naturalhazardsgroup.com
http-01 challenge for www.naturalhazardsgroup.com
Waiting for verification...
Cleaning up challenges

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed with reload of apache server; fullchain is
/etc/letsencrypt/live/naturalhazardsgroup.com/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

Congratulations, all renewals succeeded. The following certs have been renewed:
  /etc/letsencrypt/live/naturalhazardsgroup.com/fullchain.pem (success)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
19

Done.
“Congrats” was achieved!

20

Looking good :slight_smile:

Is it effectuated despite --dry-run? It looks as though the site certificate is valid from 10 Feb - 11 May now.

And must I restart the server?