Multiple issues, unexpected errors and general chaos


#1

My domain is: nickyowen.com (and others, see below)

I ran this command: Ubuntu’s renew cronjob:

(root) CMD (test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(43200))' && certbot -q renew)

It produced this output:

Jun 27 12:31:34 alice certbot[8167]: Attempting to renew cert (www.nickyowen.com) from /etc/letsencrypt/renewal/www.nickyowen.com.conf produced an unexpected error: Failed authorization procedure. relaxforlife.co.uk (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://relaxforlife.co.uk/.well-known/acme-challenge/KK3KhulH5ROvTnqBUutmQnvi-tFkNCJ5Lq_S95N9jgE: "<!DOCTYPE html>
Jun 27 12:31:34 alice certbot[8167]: <html lang="en-GB">
Jun 27 12:31:34 alice certbot[8167]: <head>
Jun 27 12:31:34 alice certbot[8167]: #011<meta charset="UTF-8"/>
Jun 27 12:31:34 alice certbot[8167]: #011<meta name="viewport" content="width=device-width, initial-", nickyowen.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://nickyowen.com/.well-known/acme-challenge/7PImP7t957IJV0CrY9togZk9aBqa32T2OPsYBxMWZlA: "<!DOCTYPE html>
Jun 27 12:31:34 alice certbot[8167]: <html lang="en-GB">
Jun 27 12:31:34 alice certbot[8167]: <head>
Jun 27 12:31:34 alice certbot[8167]: #011<meta charset="UTF-8"/>
Jun 27 12:31:34 alice certbot[8167]: #011<meta name="viewport" content="width=device-width, initial-", www.nickyowen.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.nickyowen.com/.well-known/acme-challenge/YEB68GN3NJmp1JSEDkNAxNK_sSZDs4tkxmR4o9H66oY: "<!DOCTYPE html>
Jun 27 12:31:34 alice certbot[8167]: <html lang="en-GB">
Jun 27 12:31:34 alice certbot[8167]: <head>
Jun 27 12:31:34 alice certbot[8167]: #011<meta charset="UTF-8"/>
Jun 27 12:31:34 alice certbot[8167]: #011<meta name="viewport" content="width=device-width, initial-", www.relaxforlife.co.uk (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.relaxforlife.co.uk/.well-known/acme-challenge/LtldmgugCyyw8QN4q_-W4lUxEq534GU3sBbKeSUs5VQ: "<!DOCTYPE html>
Jun 27 12:31:34 alice certbot[8167]: <html lang="en-GB">
Jun 27 12:31:34 alice certbot[8167]: <head>
Jun 27 12:31:34 alice certbot[8167]: #011<meta charset="UTF-8"/>
Jun 27 12:31:34 alice certbot[8167]: #011<meta name="viewport" content="width=device-width, initial-". Skipping.
Jun 27 12:32:07 alice certbot[8167]: All renewal attempts failed. The following certs could not be renewed:
Jun 27 12:32:07 alice certbot[8167]:   /etc/letsencrypt/live/www.nickyowen.com/fullchain.pem (failure)
Jun 27 12:32:07 alice certbot[8167]: 1 renew failure(s), 0 parse failure(s)

My web server is (include version): Apache 2.4.29-1ubuntu4.1

The operating system my web server runs on is (include version): Ubuntu 18.04 LTS

My hosting provider, if applicable, is: Jump Networks Ltd UK

I can login to a root shell on my machine (yes or no, or I don’t know): Yes (with root access)

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): No


Also, when I run certbot certificates (no arguments), I get this before the list:

Attempting to parse the version 0.26.0.dev0 renewal configuration file found at /etc/letsencrypt/renewal/tes.bakerbates.com.conf with version 0.25.0 of Certbot. This might not work.
Attempting to parse the version 0.26.0.dev0 renewal configuration file found at /etc/letsencrypt/renewal/www.contentclear.co.uk.conf with version 0.25.0 of Certbot. This might not work.
Attempting to parse the version 0.26.0.dev0 renewal configuration file found at /etc/letsencrypt/renewal/www.thedragnet.org.conf with version 0.25.0 of Certbot. This might not work.
Renewal configuration file /etc/letsencrypt/renewal/kateability.com.conf produced an unexpected error: expected /etc/letsencrypt/live/kateability.com/cert.pem to be a symlink. Skipping.
Renewal configuration file /etc/letsencrypt/renewal/www.bluett.com.conf produced an unexpected error: expected /etc/letsencrypt/live/www.bluett.com/cert.pem to be a symlink. Skipping.
Attempting to parse the version 0.26.0.dev0 renewal configuration file found at /etc/letsencrypt/renewal/www.bakerbates.com.conf with version 0.25.0 of Certbot. This might not work.
Renewal configuration file /etc/letsencrypt/renewal/www.kateability.com.conf produced an unexpected error: expected /etc/letsencrypt/live/www.kateability.com/cert.pem to be a symlink. Skipping.
Renewal configuration file /etc/letsencrypt/renewal/www.hatters.org.uk.conf produced an unexpected error: expected /etc/letsencrypt/live/www.hatters.org.uk/cert.pem to be a symlink. Skipping.

When certbot tries to renew, all the SSL sites go offline as far as I can tell too (erroring with a failed to connect error in the browser). They then come back up a bit later.

Some background:

Having run all my sites without issue for a couple of years, this week I have migrated them to another server. In the course of which I simply copied the contents of /etc/letsencrypt over to the new machine. The old machine was running an early version of certbot (I think). It was maintained by somebody else and I know little about it. I installed the certbot python3-certbot-apache 0.25.0-2+ubuntu18.04.1+certbot+1 package from standard Ubuntu repos on the new machine.

There seem to be so many issues though and I only have about 25 SSL sites. Would it be better simply to erase everything and start again somehow?


#2

This is not a released version of Certbot. It’s hard to know how this was achieved apart from running Certbot from git, perhaps.

You could give this a shot:

certbot renew -i apache -a apache --dry-run

It is non-destructive and might help figure out whether your configuration directory is salvageable or not.

For the symlink errors, I think you will either need to manually repair them or simply start again for all of your sites with an empty /etc/letsencrypt/ .


#3

Thanks - the dry-run indicated most are fine. It reported four with broken symlinks so I’ll try manually fixing those.

Interestingly though, the cert for www.nickyowen.com (the one that barfed on the cronjob yesterday) says it would be renewed OK:

 The following certs were successfully renewed: 
/etc/letsencrypt/live/www.nickyowen.com/fullchain.pem (success)

BTW if I still run into problems, is nuking it all from orbit and starting again done like this?

  1. Remove all files in /etc/letsencrypt
  2. Remove all *-le-ssl.conf files from Apache config and put those configs back to standard port 80 *.conf files
  3. Re-install certbot-apache
  4. Run certbot --apache -d domainname.com for each domain I want to protect.

#4

Changing the authenticator with -a apache may have been all that’s needed to make it start working again.

I would really try to fix the symlinks first. It’s usually okay to start again when you have a couple of virtual hosts, but I seriously dread the prospect of 25.

Your plan seems fine, but it’s not necessary to re-install certbot-apache, just removing /etc/letsencrypt/ is enough.

You might be able to get away with just unconfiguring the 4 broken sites and then deleting them from Certbot, saving the other 21:

certbot delete --cert-name kateability.com

but I’m not sure whether it’ll succeed or not with broken symlinks.


#5

Thanks - one last question: does that mean I can try a one-off manual renewal with:

certbot renew -i apache -a apache --cert-name www.nickyowen.com

and then let the cronjob (which doesn’t appear to use -a apache) take over from there?


#6

Yes, calling renew manually like that save the authenticator to the renewal parameters file in /etc/letsencrypt/renewal/, so you don’t need to alter the cronjob.


#7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.