Certbot renew stopped working at all


#1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:
www.perlenschweine.de
(The error occurs since last version update on this and all other domains hosted on this server)

I ran this command:
certbot renew

It produced this output:
Processing /etc/letsencrypt/renewal/www.perlenschweine.de.conf


Cert is due for renewal, auto-renewing…
Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate
Attempting to renew cert (www.perlenschweine.de) from /etc/letsencrypt/renewal/www.perlenschweine.de.conf produced an unexpected error: urn:ietf:params:acme:error:rateLimited :: There were too many requests of a given type :: Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/rate-limits/. Skipping.
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/www.perlenschweine.de/fullchain.pem (failure)

My web server is (include version):
ii apache2 2.4.7-1ubuntu4.21
The operating system my web server runs on is (include version):
Ubuntu 14.04
My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don’t know):
Yes
I’m using a control panel to manage my site (no, or provide the name and version of the control panel):
no
The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot):
certbot 0.28.0


#2

Hi @Arheddis

your current error

means: You have to wait one hour (max. 5 failed authorizations / hour) or you use the test system.

But the real problem may be ( https://check-your-website.server-daten.de/?q=perlenschweine.de ):

Domainname Http-Status redirect Sec. G
http://perlenschweine.de/
85.214.246.32 302 https://perlenschweine.de// 0.014 A
http://www.perlenschweine.de/
85.214.246.32 302 https://www.perlenschweine.de// 0.013 A
https://perlenschweine.de/
85.214.246.32 403 1.337 N
Forbidden
Certificate error: RemoteCertificateNameMismatch
https://perlenschweine.de// -14 10.027 T
Timeout - The operation has timed out
https://www.perlenschweine.de/
85.214.246.32 200 5.307 A
https://www.perlenschweine.de// -14 10.020 T
Timeout - The operation has timed out
http://perlenschweine.de/.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de
85.214.246.32 -14 10.027 T
Timeout - The operation has timed out
http://www.perlenschweine.de/.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de
85.214.246.32 -14 10.037 T
Timeout - The operation has timed out

Earlier, you may have used tls-sni-01 - validation. But this is deprecated, so certbot uses http-01 validation. Then an open port 80 is required. Certbot creates a file in /.well-known/acme-challenge, Letsencrypt checks this file.

But checking a file there -> timeout.

What’s your configuration?

And there is a wrong redirect, one / too much is added.


#3

You can try using “certbot renew --dry-run” for now. The staging system has separate (and higher) rate limits, so you’ll be able to see the real reason it fails.


#4

Ah, ok.
Now http is served: See https://www.perlenschweine.de/.well-known/acme-challenge/acme
But certbot renew --dryrun produces:
Processing /etc/letsencrypt/renewal/www.perlenschweine.de.conf


Cert is due for renewal, auto-renewing…
Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for www.perlenschweine.de
Cleaning up challenges
Attempting to renew cert (www.perlenschweine.de) from /etc/letsencrypt/renewal/www.perlenschweine.de.conf produced an unexpected error: Unable to insert label!. Skipping.


#5

The command is --dry-run


#6

Sorry, there is still a redirect to https, have to check, sorry for that.


#7

This isn’t a problem, Letsencrypt follows redirects and ignores certificate errors.

If the redirect is correct (same subfolder), that shouldn’t be a problem.


#8

Ok. Command issued was “certbot renew --dry-run” (typo in post …)
It produced the error above.


#9

First of all: THX for your assistance!
Here’s the log:
2019-01-26 17:06:59,061:WARNING:certbot.renewal:Attempting to renew cert (www.perlenschweine.de) from /etc/letsencrypt/renewal/www.perlenschweine.de.conf produced an unexpected error: Unable to insert label!. Skipping.
2019-01-26 17:06:59,062:DEBUG:certbot.renewal:Traceback was:
Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/certbot/renewal.py”, line 430, in handle_renewal_request
main.renew_cert(lineage_config, plugins, renewal_candidate)
File “/usr/lib/python3/dist-packages/certbot/main.py”, line 1168, in renew_cert
renewed_lineage = _get_and_save_cert(le_client, config, lineage=lineage)
File “/usr/lib/python3/dist-packages/certbot/main.py”, line 116, in _get_and_save_cert
renewal.renew_cert(config, domains, le_client, lineage)
File “/usr/lib/python3/dist-packages/certbot/renewal.py”, line 305, in renew_cert
new_cert, new_chain, new_key, _ = le_client.obtain_certificate(domains, new_key)
File “/usr/lib/python3/dist-packages/certbot/client.py”, line 335, in obtain_certificate
orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
File “/usr/lib/python3/dist-packages/certbot/client.py”, line 371, in _get_order_and_authorizations
authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
File “/usr/lib/python3/dist-packages/certbot/auth_handler.py”, line 75, in handle_authorizations
resp = self._solve_challenges(aauthzrs)
File “/usr/lib/python3/dist-packages/certbot/auth_handler.py”, line 132, in _solve_challenges
resp = self.auth.perform(all_achalls)
File “/usr/lib/python3/dist-packages/certbot_apache/configurator.py”, line 2280, in perform
http_response = http_doer.perform()
File “/usr/lib/python3/dist-packages/certbot_apache/http_01.py”, line 72, in perform
self._mod_config()
File “/usr/lib/python3/dist-packages/certbot_apache/http_01.py”, line 97, in _mod_config
self._set_up_include_directives(vh)
File “/usr/lib/python3/dist-packages/certbot_apache/http_01.py”, line 172, in _set_up_include_directives
vhost.path, “Include”, self.challenge_conf_pre)
File “/usr/lib/python3/dist-packages/certbot_apache/parser.py”, line 346, in add_dir_beginning
self.aug.insert(first_dir, “directive”, True)
File “/usr/lib/python3/dist-packages/augeas.py”, line 369, in insert
raise ValueError(“Unable to insert label!”)
ValueError: Unable to insert label!


#10

This means that you’ve found a bug in Certbot that relates to something unusual about your Apache configuration. Could you share your Apache configuration with us somehow?

@joohoi, have you ever seen this Augeas error before?


#11

Hi Mr. schoen,

how to share my config? I don’t like the idea of posting my config files to the public… Any upload site or per mail?

THX!


#12

You could send it to my forum username at eff.org.


#13

Mail is on it’s way.


#14

Are you sure you didn’t use

-dry-run

Then it would be the -d option with ry-run as domain name, this label is too short to create a certificate.


#15

No, it was --dry-run
Double hyphen, checked my bash history.


#16

But, strange enough, the “label” error has disappeared over night. Ran certbot renew --dry-run and got:
HTTP Status 403, forbidden on http://www.perlenschweine.de/.well-known/acme-challenge/tIfjydYkqHu1XO8UjF2pmudv9IabT6oKPgj1DDEpXn4

wget on this URL gives 404. That’s ok, there is no file with this name, certbot never created it. I guess, certbot should.
certbot runs as root, it schould be able to create a file in apaches file system.


#17

Solved it!
Had to do some debugging… -vvv flag did it.
This problem was missing access rights for apache user www-data on
/var/lib/letsencrypt (was: root:root, 750)
For all out there in the wild, here’s a short description what’s going on with letsencrypt’s http authentication:

  1. certbot aquires a one time challenge key from letsencrypt.org
  2. This key is placed in /var/lib/letsencrypt/http-challenges
  3. The apache config is temporarily rewritten by certbot to redirect corresponding requests to that dir.
  4. A graceful restart of apache is performed, so that the rewritten apache config gets activated.
  5. letsencrypt performs a http get on .well-known/acme-challenge/ witch gets redirected to /var/lib/letsencrypt/http-challenges/. That’s why apache needs read access to that path!
  6. The authentication get’s confirmed, the cert is updated and the original apache config get’s restored.

Solution (as root): cd /var/lib; chown -R root:www-data letsencrypt


Certbot must check read access for web-server user on renewal
#18

Happy to read that it works now.

If someone uses webroot as authenticator, no redirect is required. Instead the real webroot / DocumentRoot is used.


#19

But what if there is no usable webroot? Nextcloud for example stops updating if there are any unknown artifacts in it’s webroot. Will .well_known/ be deleted after a renewal?

Anyways, i’m online and secure again. THX a lot!


#20

This is a curious thing. Nextcloud should ignore such standard-directories.

If required, you can run a --post-hook command.

--post-hook POST_HOOK
                        Command to be run in a shell after attempting to
                        obtain/renew certificates. Can be used to deploy
                        renewed certificates, or to restart any servers that
                        were stopped by --pre-hook. This is only run if an
                        attempt was made to obtain/renew a certificate. If
                        multiple renewed certificates have identical post-
                        hooks, only one will be run. (default: None)