Certbot renew issue

[GridPaneBobby]

Hey all! We’re seeing a weird issue that is coming up, and I haven’t been able to find a lot of information about it. It started recently:

Attempting to renew cert (customerdomain.com) from /etc/letsencrypt/renewal/customerdomain.com.conf produced an unexpected error: unhashable type: ‘list’. Skipping.

Yes, I realize I didn’t include the real domain, and I understand the logic here, but domain is probably irrelevant. This issue started yesterday and I’ve been tracking it.

When we get this error, this is what is in the renewal conf:

renew_before_expiry = 30 days

version = 0.31.0
archive_dir = /etc/letsencrypt/archive/customerdomain.com
cert = /etc/letsencrypt/live/customerdomain.com/cert.pem
privkey = /etc/letsencrypt/live/customerdomain.com/privkey.pem
chain = /etc/letsencrypt/live/customerdomain.com/chain.pem
fullchain = /etc/letsencrypt/live/customerdomain.com/fullchain.pem

Options used in the renewal process

[renewalparams]
account = id
must_staple = True
authenticator = webroot
server = https://acme-v02.api.letsencrypt.org/directory
[[webroot_map]]
customerdomain.com = /var/www/customerdomain.com/htdocs
www.customerdomain.com = /var/www/customerdomain.com/htdocs
webroot_path = /var/www/customerdomain.com/htdocs,

If we remove the comma on that last line, everything goes to working. It’s never had a problem with that comma, but that comma is placed there presumably by certbot.

My web server is (include version): nginx 1.16.1

The operating system my web server runs on is (include version): Ubuntu 18.04

My hosting provider, if applicable, is: Not applicable

I can login to a root shell on my machine (yes or no, or I don’t know): Yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): GridPane

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot): We’re using certbot-auto version 1.0.0

[mnordhoff]

Has that configuration file been manually edited (beyond the obvious)? The webroot_path setting, if it exists, should be in the renewalparams section of the file, not the webroot_map section of the file.

[GridPaneBobby]

No, it hasn’t been manually edited that I’m aware of, and it’s happened now on several customers, so I don’t think it’s customers doing this. I moved the line up to renewalparams, and it renewed but the odd thing is, I can leave it under map section, and remove comma and it renews then as well.

[mnordhoff]

The webroot_path setting has a comma because it’s a list instead of a single string.

According to a comment in the code, it was a string in ancient versions of Certbot, so that’s still supported for backwards compatiblity.

But the webroot_map section isn’t supposed to contain lists, and evidently putting lists in it doesn’t work. Presumably Certbot (currently) ignores unexpected strings.

Edit:

I just ran “sudo certbot certonly --staging --webroot -w /srv/www/certbot -d $RANDOM.clover.letsencrypt.mattnordhoff.net” with the same version of Certbot and the certificate had this renewal file:

# renew_before_expiry = 30 days
version = 1.0.0
archive_dir = /etc/letsencrypt/archive/24385.clover.letsencrypt.mattnordhoff.net
cert = /etc/letsencrypt/live/24385.clover.letsencrypt.mattnordhoff.net/cert.pem
privkey = /etc/letsencrypt/live/24385.clover.letsencrypt.mattnordhoff.net/privkey.pem
chain = /etc/letsencrypt/live/24385.clover.letsencrypt.mattnordhoff.net/chain.pem
fullchain = /etc/letsencrypt/live/24385.clover.letsencrypt.mattnordhoff.net/fullchain.pem

# Options used in the renewal process
[renewalparams]
authenticator = webroot
account = d9d5905b53680c43f84f8683517b1877
webroot_path = /srv/www/certbot,
server = https://acme-staging-v02.api.letsencrypt.org/directory
[[webroot_map]]
24385.clover.letsencrypt.mattnordhoff.net = /srv/www/certbot

If your Certbot put the webroot_path setting in the webroot_map section under some sort of normal circumstances, it’s a bug. But someone or something could also have done it accidentally while manually editing the files…