Wildcard certificate renewal error

Help
1

Hello,

I’m having trouble renewing my wildcard certificates using certbot 0.22.2-1+ubuntu14.04.1+certbot+1
I’ve generated them in the first place using these commands:
virtualenv /opt/letsencrypt
source /opt/letsencrypt/bin/activate
certbot certonly --dns-route53 -d *.camlinrail.com --server https://acme-v02.api.letsencrypt.org/directory

My domain is: *.camlinrail.com
I ran this command: certbot -q renew

It produced this output from shell:
Attempting to renew cert (camlinrail.com) from /etc/letsencrypt/renewal/camlinrail.com.conf produced an unexpected error: unorderable types: NoneType() < NoneType(). Skipping.
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/camlinrail.com/fullchain.pem (failure)

It produced this output from python virtualenv (source /opt/letsencrypt/bin/activate):
Attempting to renew cert (camlinrail.com) from /etc/letsencrypt/renewal/camlinrail.com.conf produced an unexpected error: max() arg is an empty sequence. Skipping.
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/camlinrail.com/fullchain.pem (failure)

My web server is (include version): Apache 2.4.7-1ubuntu4.9
The operating system my web server runs on is (include version): Ubuntu 14.04.4 LTS
I can login to a root shell on my machine (yes or no, or I don’t know): Yes
I’m using a control panel to manage my site (no, or provide the name and version of the control panel): NO

2

If you’ve installed Certbot from the Ubuntu PPA, why are you using virtualenv? Are you following instructions from somewhere?

3

Hi,

I have installed certbot from the ppa:certbot/certbot

I am using Python virtualenv since I also installed the “DNS challenge plugin for certbot” (certbot-dns-route53) and this is the official way of installing it (https://github.com/certbot/certbot/tree/master/certbot-dns-route53). I am using certbot-dns-route53 since I host my domains on AWS Route53 DNS.

I have tried the renew the certificate from the python virtualenv and from the root shell also (see initial post) - to no avail.

4

I’m not sure whether those instructions are aimed at end-users.

Is /opt/letsencrypt the cloned git repo?

What do these say when you’re inside the virtualenv?

source /opt/letsencrypt/bin/activate

certbot --version
certbot plugins
cat /etc/letsencrypt/renewal/camlinrail.com.conf
5

/opt/letsencrypt is the Python virtualenv I’ve created, I simply placed it in /opt.

Below are the commands executed from Python virtualenv:
(letsencrypt)root@server:~# certbot --version
certbot 0.23.0

(letsencrypt)root@server:~# certbot plugins
Saving debug log to /var/log/letsencrypt/letsencrypt.log

  • dns-route53
    Description: Obtain certificates using a DNS TXT record (if you are using AWS
    Route53 for DNS).
    Interfaces: IAuthenticator, IPlugin
    Entry point: dns-route53 = certbot_dns_route53.dns_route53:Authenticator

  • standalone
    Description: Spin up a temporary webserver
    Interfaces: IAuthenticator, IPlugin
    Entry point: standalone = certbot.plugins.standalone:Authenticator

  • webroot
    Description: Place files in webroot directory
    Interfaces: IAuthenticator, IPlugin
    Entry point: webroot = certbot.plugins.webroot:Authenticator

(letsencrypt)root@server:~# cat /etc/letsencrypt/renewal/camlinrail.com.conf
# renew_before_expiry = 30 days
version = 0.23.0
archive_dir = /etc/letsencrypt/archive/camlinrail.com
cert = /etc/letsencrypt/live/camlinrail.com/cert.pem
privkey = /etc/letsencrypt/live/camlinrail.com/privkey.pem
chain = /etc/letsencrypt/live/camlinrail.com/chain.pem
fullchain = /etc/letsencrypt/live/camlinrail.com/fullchain.pem

# Options used in the renewal process
[renewalparams]
authenticator = dns-route53
installer = None
account = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
server = https://acme-v02.api.letsencrypt.org/directory

6

Hi @bzlom,

Do the logs in /var/log/letsencrypt contain Python tracebacks for these exceptions? The limited error printed to the console by Certbot doesn’t show where in the code the exceptions happened.

7

Hi schoen,

here’s the debug log output from /var/log/letsencrypt/letsencrypt.log:

2018-06-15 12:43:12,188:DEBUG:certbot.main:certbot version: 0.22.2
2018-06-15 12:43:12,189:DEBUG:certbot.main:Arguments: [’-q’]
2018-06-15 12:43:12,189:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2018-06-15 12:43:12,198:DEBUG:certbot.log:Root logging level set at 30
2018-06-15 12:43:12,198:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2018-06-15 12:43:12,215:INFO:certbot.storage:Attempting to parse the version 0.23.0 renewal configuration file found at /etc/letsencrypt/renewal/camlinrail.com.conf with version 0.22.2 of Certbot. This might not work.
2018-06-15 12:43:12,216:DEBUG:certbot.storage:No matches for target cert.
2018-06-15 12:43:12,216:DEBUG:certbot.storage:No matches for target privkey.
2018-06-15 12:43:12,216:WARNING:certbot.renewal:Attempting to renew cert (camlinrail.com) from /etc/letsencrypt/renewal/camlinrail.com.conf produced an unexpected error: unorderable types: NoneType() < NoneType(). Skipping.
2018-06-15 12:43:12,217:DEBUG:certbot.renewal:Traceback was:
Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/certbot/renewal.py”, line 413, in handle_renewal_request
renewal_candidate.ensure_deployed()
File “/usr/lib/python3/dist-packages/certbot/storage.py”, line 768, in ensure_deployed
if self.has_pending_deployment():
File “/usr/lib/python3/dist-packages/certbot/storage.py”, line 787, in has_pending_deployment
smallest_current = min(self.current_version(x) for x in ALL_FOUR)
TypeError: unorderable types: NoneType() < NoneType()

2018-06-15 12:43:12,217:ERROR:certbot.renewal:All renewal attempts failed. The following certs could not be renewed:
2018-06-15 12:43:12,217:ERROR:certbot.renewal: /etc/letsencrypt/live/camlinrail.com/fullchain.pem (failure)
2018-06-15 12:43:12,217:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
File “/usr/bin/certbot”, line 11, in
load_entry_point(‘certbot==0.22.2’, ‘console_scripts’, ‘certbot’)()
File “/usr/lib/python3/dist-packages/certbot/main.py”, line 1266, in main
return config.func(config, plugins)
File “/usr/lib/python3/dist-packages/certbot/main.py”, line 1179, in renew
renewal.handle_renewal_request(config)
File “/usr/lib/python3/dist-packages/certbot/renewal.py”, line 443, in handle_renewal_request
len(renew_failures), len(parse_failures)))
certbot.errors.Error: 1 renew failure(s), 0 parse failure(s)

8

Huh, that’s actually code that I wrote a few years ago. :slight_smile: It doesn’t look like the problem is likely to be directly related to the DNS plugin.

Did you modify or rename any of the files or directories in /etc/letsencrypt somehow?

9

Hi @schoen,

I did remove some files from /etc/letsencrypt/live/ folder, maybe even renamed one of them - not sure about renaming to be honest. And definitely removed some files from /etc/letsencrypt/renewal during the wildcard certificate generation tests.

P. S. I just remembered that I’ve renamed some files in /etc/letsencrypt/archive/camlinrail.com and then had to recreate the softlinks in /etc/letsencrypt/live/camlinrail.com folder - might that be an issue? The softlinks in /etc/letsencrypt/live/camlinrail.com folder point to the correct files in /etc/letsencrypt/archive/camlinrail.com.

10

Yes, Certbot uses those names to keep track of the state of the certificate. The status information is partially stored in the exact filenames in the archive directory. If you change them at all, it can produce these renewal failures.

11

@schoen Hi, thanks for the reply, what would I need to do to fix it in this case?

13

@craigomez, how would that article help? It’s not about Certbot or how Certbot stores files on disk. Also, it’s not about Let’s Encrypt but rather about a different CA, and probably about half of those errors either don’t exist in Let’s Encrypt or are called something else!

@bzlom, you could potentially delete all of /etc/letsencrypt and start over (but you’d lose any other certificates that you have, and you would also have to first remove references to the certificates in your web server configuration).

Or you could post the output of ls -lR /etc/letsencrypt here and we can try to suggest how you might be able to fix it.

14

Hi @schoen,

I’ve pasted the ls -lR output for /etc/letencrypt/

.:
total 36
drwx------ 4 root root 4096 Jun 12 11:33 accounts
drwx------ 4 root root 4096 Jun 18 10:18 archive
-rw-r–r-- 1 root root 121 Mar 21 04:54 cli.ini
drwxr-xr-x 2 root root 4096 Jun 12 15:27 csr
drwx------ 2 root root 4096 Jun 12 15:27 keys
drwx------ 4 root root 4096 Jun 12 15:28 live
-rw-r–r-- 1 root root 1619 Jun 15 10:33 options-ssl-apache.conf
drwxr-xr-x 2 root root 4096 Jun 15 10:23 renewal
drwxr-xr-x 5 root root 4096 Apr 6 10:39 renewal-hooks

./accounts:
total 8
drwx------ 3 root root 4096 Jun 12 11:33 acme-v01.api.letsencrypt.org
drwx------ 3 root root 4096 Apr 6 10:39 acme-v02.api.letsencrypt.org

./accounts/acme-v01.api.letsencrypt.org:
total 4
drwx------ 3 root root 4096 Jun 12 11:34 directory

./accounts/acme-v01.api.letsencrypt.org/directory:
total 4
drwx------ 2 root root 4096 Jun 12 11:34 174309b8dfdb7a037655e3de5e041d29

./accounts/acme-v01.api.letsencrypt.org/directory/174309b8dfdb7a037655e3de5e041d29:
total 12
-rw-r–r-- 1 root root 69 Jun 12 11:34 meta.json
-r-------- 1 root root 1632 Jun 12 11:34 private_key.json
-rw-r–r-- 1 root root 770 Jun 12 11:34 regr.json

./accounts/acme-v02.api.letsencrypt.org:
total 4
drwx------ 3 root root 4096 Apr 6 10:39 directory

./accounts/acme-v02.api.letsencrypt.org/directory:
total 4
drwx------ 2 root root 4096 Apr 6 10:39 4003220037df849b9a0e085941077827

./accounts/acme-v02.api.letsencrypt.org/directory/4003220037df849b9a0e085941077827:
total 12
-rw-r–r-- 1 root root 69 Apr 6 10:39 meta.json
-r-------- 1 root root 1632 Apr 6 10:39 private_key.json
-rw-r–r-- 1 root root 626 Apr 6 10:39 regr.json

./archive:
total 8
drwxr-xr-x 2 root root 4096 Jun 18 10:27 camlinrail.com
drwxr-xr-x 2 root root 4096 Jun 12 15:28 hiddendomain.com

./archive/camlinrail.com:
total 16
-rw-r–r-- 1 root root 2155 Apr 6 10:40 cert.pem
-rw-r–r-- 1 root root 1647 Apr 6 10:40 chain.pem
-rw-r–r-- 1 root root 3802 Apr 6 10:40 fullchain.pem
-rw-r–r-- 1 root root 1704 Apr 6 10:40 privkey.pem

./archive/hiddendomain.com:
total 16
-rw-r–r-- 1 root root 2159 Jun 12 15:28 cert1.pem
-rw-r–r-- 1 root root 1647 Jun 12 15:28 chain1.pem
-rw-r–r-- 1 root root 3806 Jun 12 15:28 fullchain1.pem
-rw-r–r-- 1 root root 1704 Jun 12 15:28 privkey1.pem

./csr:
total 60
-rw-r–r-- 1 root root 928 Apr 6 10:39 0000_csr-certbot.pem
-rw-r–r-- 1 root root 928 Jun 12 10:57 0001_csr-certbot.pem
-rw-r–r-- 1 root root 928 Jun 12 11:02 0002_csr-certbot.pem
-rw-r–r-- 1 root root 928 Jun 12 11:03 0003_csr-certbot.pem
-rw-r–r-- 1 root root 928 Jun 12 11:10 0004_csr-certbot.pem
-rw-r–r-- 1 root root 924 Jun 12 11:24 0005_csr-certbot.pem
-rw-r–r-- 1 root root 928 Jun 12 11:24 0006_csr-certbot.pem
-rw-r–r-- 1 root root 928 Jun 12 11:31 0007_csr-certbot.pem
-rw-r–r-- 1 root root 932 Jun 12 11:37 0008_csr-certbot.pem
-rw-r–r-- 1 root root 928 Jun 12 13:29 0009_csr-certbot.pem
-rw-r–r-- 1 root root 928 Jun 12 14:04 0010_csr-certbot.pem
-rw-r–r-- 1 root root 928 Jun 12 14:35 0011_csr-certbot.pem
-rw-r–r-- 1 root root 928 Jun 12 14:37 0012_csr-certbot.pem
-rw-r–r-- 1 root root 928 Jun 12 14:57 0013_csr-certbot.pem
-rw-r–r-- 1 root root 928 Jun 12 15:27 0014_csr-certbot.pem

./keys:
total 60
-rw------- 1 root root 1704 Apr 6 10:39 0000_key-certbot.pem
-rw------- 1 root root 1704 Jun 12 10:57 0001_key-certbot.pem
-rw------- 1 root root 1704 Jun 12 11:02 0002_key-certbot.pem
-rw------- 1 root root 1704 Jun 12 11:03 0003_key-certbot.pem
-rw------- 1 root root 1704 Jun 12 11:10 0004_key-certbot.pem
-rw------- 1 root root 1704 Jun 12 11:24 0005_key-certbot.pem
-rw------- 1 root root 1708 Jun 12 11:24 0006_key-certbot.pem
-rw------- 1 root root 1708 Jun 12 11:31 0007_key-certbot.pem
-rw------- 1 root root 1704 Jun 12 11:37 0008_key-certbot.pem
-rw------- 1 root root 1704 Jun 12 13:29 0009_key-certbot.pem
-rw------- 1 root root 1704 Jun 12 14:04 0010_key-certbot.pem
-rw------- 1 root root 1708 Jun 12 14:35 0011_key-certbot.pem
-rw------- 1 root root 1704 Jun 12 14:37 0012_key-certbot.pem
-rw------- 1 root root 1704 Jun 12 14:57 0013_key-certbot.pem
-rw------- 1 root root 1704 Jun 12 15:27 0014_key-certbot.pem

./live:
total 8
drwxr-xr-x 2 root root 4096 Apr 9 15:26 camlinrail.com
drwxr-xr-x 2 root root 4096 Jun 12 15:28 hiddendomain.com

./live/camlinrail.com:
total 4
lrwxrwxrwx 1 root root 37 Apr 9 15:26 cert.pem -> …/…/archive/camlinrail.com/cert.pem
lrwxrwxrwx 1 root root 38 Apr 9 15:26 chain.pem -> …/…/archive/camlinrail.com/chain.pem
lrwxrwxrwx 1 root root 42 Apr 9 15:26 fullchain.pem -> …/…/archive/camlinrail.com/fullchain.pem
lrwxrwxrwx 1 root root 40 Apr 9 15:26 privkey.pem -> …/…/archive/camlinrail.com/privkey.pem
-rw-r–r-- 1 root root 543 Apr 6 10:40 README

./live/hiddendomain.com:
total 4
lrwxrwxrwx 1 root root 40 Jun 12 15:28 cert.pem -> …/…/archive/hiddendomain.com/cert1.pem
lrwxrwxrwx 1 root root 41 Jun 12 15:28 chain.pem -> …/…/archive/hiddendomain.com/chain1.pem
lrwxrwxrwx 1 root root 45 Jun 12 15:28 fullchain.pem -> …/…/archive/hiddendomain.com/fullchain1.pem
lrwxrwxrwx 1 root root 43 Jun 12 15:28 privkey.pem -> …/…/archive/hiddendomain.com/privkey1.pem
-rw-r–r-- 1 root root 543 Jun 12 15:28 README

./renewal:
total 8
-rw-r–r-- 1 root root 532 Apr 6 10:40 camlinrail.com.conf
-rw-r–r-- 1 root root 542 Jun 12 15:28 hiddendomain.com.conf

./renewal-hooks:
total 12
drwxr-xr-x 2 root root 4096 Apr 6 10:39 deploy
drwxr-xr-x 2 root root 4096 Apr 6 10:39 post
drwxr-xr-x 2 root root 4096 Apr 6 10:39 pre

./renewal-hooks/deploy:
total 0

./renewal-hooks/post:
total 0

./renewal-hooks/pre:
total 0

15

There's a difference between the camlinrail.com and hiddendomain.com files: The files in archive are supposed to have a number in the name, like foo1.pem, but they don't. Certbot is crashing trying to find a number in names like foo.pem.

(The names of the symlinks in live are supposed to not have numbers. They're correct.)

You need to rename all 4 files in /etc/letsencrypt/archive/camlinrail.com and change the symlinks in /etc/letsencrypt/live/camlinrail.com.

2 Likes
16

Yes, this worked. Issue solved - that was exactly it. After certificate renewal new files are being created in the archive folder, but now are followed with number 2, like foo2.pem. And the softlinks in live are automatically changed to point to these new foo2.pem files.

Awesome, everything works now. Thanks a lot of the help.

17

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.