Certbot: Unable to install the certificate into Apache after deleting the VirthualHost configuration files

My domain is: martinwurm.photo

I ran this command: certbot --apache

It produced this output:

# certbot --apache
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache

Which names would you like to activate HTTPS for?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: martinwurm.photo
2: www.martinwurm.photo
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): 
Cert not yet due for renewal

You have an existing certificate that has exactly the same domains or certificate name you requested and isn't close to expiry.
(ref: /etc/letsencrypt/renewal/martinwurm.photo.conf)

What would you like to do?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: Attempt to reinstall this existing certificate
2: Renew & replace the cert (limit ~5 per 7 days)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 1
Keeping the existing certificate
File: 
 - Could not be found to be deleted /etc/apache2/sites-available/martinwurm.photo-le-ssl.conf - Certbot probably shut down unexpectedly
An unexpected error occurred:
StopIteration
Please see the logfiles in /var/log/letsencrypt for more details.

IMPORTANT NOTES:
 - Unable to install the certificate
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/martinwurm.photo/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/martinwurm.photo/privkey.pem
   Your cert will expire on 2021-01-01. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot again
   with the "certonly" option. To non-interactively renew *all* of
   your certificates, run "certbot renew"

My web server is (include version): Apache 2.4.38

The operating system my web server runs on is (include version): Debian 10

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): 0.31.0

Background:
I previously had a site set up on the same Apache install and domain and also successfully generated and installed my certificate using Certbot. I then realised that I made a mistake in that site’s VirtualHost file (martinwurm.photo.conf) and, considering that Certbot did some magic generating the TLS-enabled equivalent (martinwurm.photo-le-ssl.conf), decided to delete both of those files, recreate martinwurm.photo.conf, and run Certbot to generate a new martinwurm.photo-le-ssl.conf.

However, this did not work as planned, and Certbot fails to install the certificate ever since, showing the error message visible in the listing above ("Could not be found to be deleted /etc/apache2/sites-available/martinwurm.photo-le-ssl.conf - Certbot probably shut down unexpectedly"). Unfortunately, I do not have a backup of the file Certbot is looking for.

What I tried
Reading the error message, I suspected that Certbot still has a reference to that now deleted file but only wants to delete it, anyway, so I created an empty file called martinwurm.photo-le-ssl.conf. This did not fix the error, however, and Certbot only notified me that "an unexpected error occurred." I also tried renaming martinwurm.photo.conf to something else and loading that site configuration into Apache instead, hoping that it would resolve some kind of naming conflict. However, the outcome was the same.

Unfortunately, I was also unable to find a similar problem on the internet, even though I am sure I can’t be the first one making that mistake. I’m sorry if I missed something there. Any help resolving this issue would be appreciated!

Relevant snippet of /var/log/letsencrypt/letsencrypt.log:

2020-10-03 18:08:03,667:DEBUG:certbot.main:certbot version: 0.31.0
2020-10-03 18:08:03,668:DEBUG:certbot.main:Arguments: ['--apache']
2020-10-03 18:08:03,668:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2020-10-03 18:08:03,683:DEBUG:certbot.log:Root logging level set at 20
2020-10-03 18:08:03,684:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2020-10-03 18:08:03,685:DEBUG:certbot.plugins.selection:Requested authenticator apache and installer apache
2020-10-03 18:08:03,810:DEBUG:certbot_apache.configurator:Apache version is 2.4.38
2020-10-03 18:08:04,094:DEBUG:certbot.plugins.selection:Single candidate plugin: * apache
Description: Apache Web Server plugin
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: apache = certbot_apache.entrypoint:ENTRYPOINT
Initialized: <certbot_apache.override_debian.DebianConfigurator object at 0x7fe188643860>
Prep: True
2020-10-03 18:08:04,095:DEBUG:certbot.plugins.selection:Selected authenticator <certbot_apache.override_debian.DebianConfigurator object at 0x7fe188643860> and installer <certbot_apache.override_debian.DebianConfigurator object at 0x7fe188643860>
2020-10-03 18:08:04,095:INFO:certbot.plugins.selection:Plugins selected: Authenticator apache, Installer apache
2020-10-03 18:08:04,100:DEBUG:certbot.main:Picked account: <Account(RegistrationResource(body=Registration(key=None, contact=(), agreement=None, status=None, terms_of_service_agreed=None, only_return_existing=None, external_account_binding=None), uri='https://acme-v02.api.letsencrypt.org/acme/acct/98301579', new_authzr_uri=None, terms_of_service=None), [redacted], Meta(creation_dt=datetime.datetime(2020, 10, 3, 14, 34, 24, tzinfo=<UTC>), creation_host='martinwurm.photography'))>
2020-10-03 18:08:04,101:DEBUG:acme.client:Sending GET request to https://acme-v02.api.letsencrypt.org/directory.
2020-10-03 18:08:04,103:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org:443
2020-10-03 18:08:04,660:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "GET /directory HTTP/1.1" 200 658
2020-10-03 18:08:04,661:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Sat, 03 Oct 2020 16:08:04 GMT
Content-Type: application/json
Content-Length: 658
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "F11mJubSeZE": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
  "keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
    "website": "https://letsencrypt.org"
  },
  "newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
  "revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert"
}
2020-10-03 18:08:06,300:INFO:certbot.renewal:Cert not yet due for renewal
2020-10-03 18:08:08,249:INFO:certbot.main:Keeping the existing certificate
2020-10-03 18:08:08,250:DEBUG:certbot.reporter:Reporting to user: Congratulations! Your certificate and chain have been saved at:
/etc/letsencrypt/live/martinwurm.photo/fullchain.pem
Your key file has been saved at:
/etc/letsencrypt/live/martinwurm.photo/privkey.pem
Your cert will expire on 2021-01-01. To obtain a new or tweaked version of this certificate in the future, simply run certbot again with the "certonly" option. To non-interactively renew *all* of your certificates, run "certbot renew"
2020-10-03 18:08:08,253:DEBUG:certbot.error_handler:Encountered exception:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/certbot/client.py", line 516, in deploy_certificate
    fullchain_path=fullchain_path)
  File "/usr/lib/python3/dist-packages/certbot_apache/configurator.py", line 334, in deploy_cert
    vhosts = self.choose_vhosts(domain)
  File "/usr/lib/python3/dist-packages/certbot_apache/configurator.py", line 358, in choose_vhosts
    return [self.choose_vhost(domain, create_if_no_ssl)]
  File "/usr/lib/python3/dist-packages/certbot_apache/configurator.py", line 540, in choose_vhost
    vhost = self.make_vhost_ssl(vhost)
  File "/usr/lib/python3/dist-packages/certbot_apache/configurator.py", line 1116, in make_vhost_ssl
    self._copy_create_ssl_vhost_skeleton(nonssl_vhost, ssl_fp)
  File "/usr/lib/python3/dist-packages/certbot_apache/configurator.py", line 1260, in _copy_create_ssl_vhost_skeleton
    ssl_vh_contents, sift = self._sift_rewrite_rules(orig_contents)
  File "/usr/lib/python3/dist-packages/certbot_apache/configurator.py", line 1332, in _sift_rewrite_rules
    line = next(contents)
StopIteration

2020-10-03 18:08:08,253:DEBUG:certbot.error_handler:Calling registered functions
2020-10-03 18:08:08,253:WARNING:certbot.reverter:File: 
 - Could not be found to be deleted /etc/apache2/sites-available/martinwurm.photo-le-ssl.conf - Certbot probably shut down unexpectedly
2020-10-03 18:08:08,254:DEBUG:certbot.reporter:Reporting to user: Unable to install the certificate
2020-10-03 18:08:08,255:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
  File "/usr/bin/certbot", line 11, in <module>
    load_entry_point('certbot==0.31.0', 'console_scripts', 'certbot')()
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 1365, in main
    return config.func(config, plugins)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 1126, in run
    _install_cert(config, le_client, domains, new_lineage)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 760, in _install_cert
    path_provider.cert_path, path_provider.chain_path, path_provider.fullchain_path)
  File "/usr/lib/python3/dist-packages/certbot/client.py", line 516, in deploy_certificate
    fullchain_path=fullchain_path)
  File "/usr/lib/python3/dist-packages/certbot_apache/configurator.py", line 334, in deploy_cert
    vhosts = self.choose_vhosts(domain)
  File "/usr/lib/python3/dist-packages/certbot_apache/configurator.py", line 358, in choose_vhosts
    return [self.choose_vhost(domain, create_if_no_ssl)]
  File "/usr/lib/python3/dist-packages/certbot_apache/configurator.py", line 540, in choose_vhost
    vhost = self.make_vhost_ssl(vhost)
  File "/usr/lib/python3/dist-packages/certbot_apache/configurator.py", line 1116, in make_vhost_ssl
    self._copy_create_ssl_vhost_skeleton(nonssl_vhost, ssl_fp)
  File "/usr/lib/python3/dist-packages/certbot_apache/configurator.py", line 1260, in _copy_create_ssl_vhost_skeleton
    ssl_vh_contents, sift = self._sift_rewrite_rules(orig_contents)
  File "/usr/lib/python3/dist-packages/certbot_apache/configurator.py", line 1332, in _sift_rewrite_rules
    line = next(contents)
StopIteration
2020-10-03 18:08:08,255:ERROR:certbot.log:An unexpected error occurred:
2 Likes

Welcome to the Let's Encrypt Community, Martin :slightly_smiling_face:

Sorry you're facing this issue. You are right that it's certainly uncommon, but nothing I haven't seen a few times and certainly does not appear to be the worst case. The directory and file structures that certbot maintains can be complex and often involve symlinks.

If things aren't too mangled, we might be able to fix things somewhat swiftly. Please try the following two commands and let us know the output:

certbot certificates

certbot delete --cert-name martinwurm.photo

Note to self: reference to certbot devs

3 Likes

Hello Jonathan!

Thanks for your reply. I forgot to mention that I did actually try deleting my certificate and generating a new one, but that didn’t solve the issue. Still, I tried your commands and here are the outputs:

# certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Found the following certs:
  Certificate Name: martinwurm.photo
    Domains: martinwurm.photo www.martinwurm.photo
    Expiry Date: 2021-01-01 14:28:06+00:00 (VALID: 89 days)
    Certificate Path: /etc/letsencrypt/live/martinwurm.photo/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/martinwurm.photo/privkey.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

.

# certbot delete --cert-name martinwurm.photo
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Deleted all files relating to certificate martinwurm.photo.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 Like

When you deleted previously, did you use the command I posted or manually delete?

2 Likes

If I’m not mistaken, I used certbot delete on its own and then got provided with a list of certificates to choose from. I just hit enter, deleting all listed certificates (although there was only one).

1 Like

When you run certbot certificates right now, what do you get?

2 Likes
certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
No certs found.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 Like
# certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
No certs found.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 Like

Perfect.

Certificate History

Let's try:

certbot run --apache

2 Likes

Same error as before:

# certbot run --apache
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache

Which names would you like to activate HTTPS for?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: martinwurm.photo
2: www.martinwurm.photo
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): 
Obtaining a new certificate
File: 
 - Could not be found to be deleted /etc/apache2/sites-available/martinwurm.photo-le-ssl.conf - Certbot probably shut down unexpectedly
An unexpected error occurred:
StopIteration
Please see the logfiles in /var/log/letsencrypt for more details.

IMPORTANT NOTES:
 - Unable to install the certificate
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/martinwurm.photo/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/martinwurm.photo/privkey.pem
   Your cert will expire on 2021-01-01. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot again
   with the "certonly" option. To non-interactively renew *all* of
   your certificates, run "certbot renew"
2 Likes

OK.

Since you're not currently rate-limited, let's try a workaround. There are most likely several lines in your configuration files that have comments about being managed by certbot. Those need to be dealt with at a point. For now though...

certbot delete --cert-name martinwurm.photo

certbot run --cert-name photo --apache -d "martinwurm.photo,www.martinwurm.photo"

3 Likes

The deletion worked as expected. The next command failed with the same error, though:

# certbot run --cert-name photo --apache -d "martinwurm.photo,www.martinwurm.photo"
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Obtaining a new certificate
File: 
 - Could not be found to be deleted /etc/apache2/sites-available/martinwurm.photo-le-ssl.conf - Certbot probably shut down unexpectedly
An unexpected error occurred:
StopIteration
Please see the logfiles in /var/log/letsencrypt for more details.

IMPORTANT NOTES:
 - Unable to install the certificate
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/photo/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/photo/privkey.pem
   Your cert will expire on 2021-01-01. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot again
   with the "certonly" option. To non-interactively renew *all* of
   your certificates, run "certbot renew"
2 Likes

Hmm...

I was hopeful, but certbot is insistent here. The certificate generation definitely succeeded per crt.sh.

Alright... let's gather some more info.

What says apachectl -S? Please put three backticks alone on lines above and below the output.

What are the contents of:
/etc/letsencrypt/live/
/etc/letsencrypt/archive/
/etc/letsencrypt/keys/
/etc/letsencrypt/renewal/
/etc/apache2/sites-enabled/
/etc/apache2/sites-available/

Just a note:
/etc/letsencrypt/archive and /etc/letsencrypt/keys contain all previous keys and certificates, while /etc/letsencrypt/live symlinks to the latest versions.

3 Likes
# apachectl -S
VirtualHost configuration:
*:80                   martinwurm.photo (/etc/apache2/sites-enabled/martinwurm.photo.conf:1)
ServerRoot: "/etc/apache2"
Main DocumentRoot: "/var/www/html"
Main ErrorLog: "/var/log/apache2/error.log"
Mutex rewrite-map: using_defaults
Mutex ssl-stapling-refresh: using_defaults
Mutex ssl-stapling: using_defaults
Mutex ssl-cache: using_defaults
Mutex default: dir="/var/run/apache2/" mechanism=default 
Mutex mpm-accept: using_defaults
Mutex watchdog-callback: using_defaults
PidFile: "/var/run/apache2/apache2.pid"
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
User: name="www-data" id=33
Group: name="www-data" id=33
# ll /etc/letsencrypt/live/
insgesamt 8
drwxr-xr-x 2 root root 4096 Okt  3 19:51 photo
-rw-r--r-- 1 root root  740 Okt  3 16:34 README
# ll /etc/letsencrypt/archive/
insgesamt 4
drwxr-xr-x 2 root root 4096 Okt  3 19:51 photo
# ll /etc/letsencrypt/keys
insgesamt 20
-rw------- 1 root root 1704 Okt  3 16:34 0000_key-certbot.pem
-rw------- 1 root root 1708 Okt  3 17:24 0001_key-certbot.pem
-rw------- 1 root root 1704 Okt  3 17:28 0002_key-certbot.pem
-rw------- 1 root root 1704 Okt  3 19:43 0003_key-certbot.pem
-rw------- 1 root root 1704 Okt  3 19:51 0004_key-certbot.pem
# ll /etc/letsencrypt/renewal
insgesamt 4
-rw-r--r-- 1 root root 484 Okt  3 19:51 photo.conf
# ll /etc/apache2/sites-enabled/
insgesamt 0
lrwxrwxrwx 1 root root 40 Okt  3 17:38 martinwurm.photo.conf -> ../sites-available/martinwurm.photo.conf
# ll /etc/apache2/sites-available/
insgesamt 16
-rw-r--r-- 1 root root 1332 Aug  8 09:47 000-default.conf
-rw-r--r-- 1 root root 6338 Aug  8 09:47 default-ssl.conf
-rw-r--r-- 1 root root 1104 Okt  3 17:27 martinwurm.photo.conf
2 Likes

What contains /etc/apache2/sites-available/martinwurm.photo.conf?

I love how clean your system is. We get some real horror shows around here.

3 Likes
<VirtualHost *:80>
        ServerName martinwurm.photo
	ServerAlias www.martinwurm.photo

        ServerAdmin webmaster@martinwurm.photo
        VirtualDocumentRoot /usr/share/wordpress

        Alias /wp-content /var/lib/wordpress/wp-content
        <Directory /usr/share/wordpress>
            Options FollowSymLinks
            AllowOverride Limit Options FileInfo
            DirectoryIndex index.php
            Require all granted
        </Directory>
        <Directory /var/lib/wordpress/wp-content>
            Options FollowSymLinks
            Require all granted
        </Directory>

	RewriteEngine On
        RewriteRule ^/wp-content/(.*)$ /srv/www/wp-content/%{HTTP_HOST}/$1
    	RewriteRule ^index\.php$ - [L]
    	RewriteCond /usr/share/wordpress%{REQUEST_URI} !-f
    	RewriteCond /usr/share/wordpress%{REQUEST_URI} !-d
    	RewriteRule . /usr/share/wordpress/index.php [L]
    	# Also needed if using PHP-FPM / Fast-CGI
    	RewriteCond %{REQUEST_URI} !^/php5-fcgi/*

        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined

</VirtualHost>
2 Likes

I just noticed that the second Directory statement should be unnecessary due to the RewriteRule down below, but that shouldn’t really make a difference here, I assume.

2 Likes

No worries there I believe.

It certainly appears the issue is in certbot's configuration files.

What contains /etc/letsencrypt/?

3 Likes
# ll /etc/letsencrypt
insgesamt 36
drwxr-xr-x 3 root root 4096 Okt  3 16:33 accounts
drwx------ 3 root root 4096 Okt  3 19:51 archive
-rw-r--r-- 1 root root  121 Mai 26  2018 cli.ini
drwxr-xr-x 2 root root 4096 Okt  3 19:51 csr
drwx------ 2 root root 4096 Okt  3 19:51 keys
drwx------ 3 root root 4096 Okt  3 19:51 live
-rw-r--r-- 1 root root 1619 Okt  3 16:33 options-ssl-apache.conf
drwxr-xr-x 2 root root 4096 Okt  3 19:51 renewal
drwxr-xr-x 5 root root 4096 Okt  3 16:33 renewal-hooks
2 Likes

Ah... my least favorite file...

What contains cli.ini?

3 Likes