Problem with certificate renewal

Help
1

I realize this is a common question and I am sorry that despite reading a ton of related threads, I am still unable to resolve my problem.

They used to renew when I was on Ubuntu 16.04, but after upgrading to 18.04 failed, I was forced to reformat the disk and redo everything from scratch.

Currently using WordPress 4.9.8 on my sites.

OS: Ubuntu 18.04.1 with latest updates as of 08:00 GMT 11-Dec-2018
certbot: 0.26.1
apache2: 2.4.29

I am (fairly) certain certbot renew --dry-run worked back in September 2018.

I use Cloudflare. I verified that the SSL option is set to ‘Full SSL (strict)’

The ‘authenticator’ is set to ‘apache’ in my /renewal/conf file

I did not have an ‘acme-challenge’ directory, so I created one. I set the ownership to ‘www-data:www-data’ and the permissions to 777 (not happy about that).

I added the file ‘goodbye’ to the folder to ensure it could be accessed.

I also added a ‘.htaccess’ to turn the ‘RewriteEngine’ off in that folder.

Around 08:00 GMT 11-Dec-2018, I also added an ‘AAAA’ record to point to my ip6 address.

Executing: certbot renew --dry-run results in:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/complete-concrete-concise.com.conf

Cert is due for renewal, auto-renewing…
Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for faqbite.com
http-01 challenge for www.faqbite.com
http-01 challenge for complete-concrete-concise.com
http-01 challenge for www.complete-concrete-concise.com
Waiting for verification…
Cleaning up challenges
Attempting to renew cert (complete-concrete-concise.com) from /etc/letsencrypt/renewal/complete-concrete-concise.com.conf produced an unexpected error: Failed authorization procedure. www.complete-concrete-concise.com (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.complete-concrete-concise.com/.well-known/acme-challenge/Q9KlKi5_np590W-eBQozBw93v-W8rvKMqnotVrQYzOU: “\n\n404 Not Found\n<script src=”/cdn-cgi/apps/head/vDH", complete-concrete-concise.com (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://complete-concrete-concise.com/.well-known/acme-challenge /rp0A8itFOH9Wamakm8Z_kWXkb0gv9010VNjLBWQ4vG4: “\n\n404 Not Found\n<script src=”/cdn-cgi/apps/head/vDH". Skipping.
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/complete-concrete-concise.com/fullchain.pem (failure)

** DRY RUN: simulating ‘certbot renew’ close to cert expiry
** (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/complete-concrete-concise.com/fullchain.pem (failure)
** DRY RUN: simulating ‘certbot renew’ close to cert expiry
** (The test certificates above have not been saved.)

1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:

My apache config file for the site is:

# The ServerName directive sets the request scheme, hostname and port that # the server uses to identify itself. This is used when creating # redirection URLs. In the context of virtual hosts, the ServerName # specifies what hostname must appear in the request's Host: header to # match this virtual host. For the default virtual host (this file) this # value is not decisive as it is used as a last resort host regardless. # However, you must set it for any further virtual host explicitly. #ServerName www.example.com 
    ServerAdmin webmaster@localhost
    ServerName complete-concrete-concise.com
    ServerAlias www.complete-concrete-concise.com
    DocumentRoot /var/www/complete-concrete-concise.com/public_html
    #DocumentRoot /var/www/html

    # Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
    # error, crit, alert, emerg.
    # It is also possible to configure the loglevel for particular
    # modules, e.g.
    #LogLevel info ssl:warn

    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined

    # For most configuration files from conf-available/, which are
    # enabled or disabled at a global level, it is possible to
    # include a line for only one particular virtual host. For example the
    # following line enables the CGI configuration for this host only
    # after it has been globally disabled with "a2disconf".
    #Include conf-available/serve-cgi-bin.conf
    RewriteEngine on

Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateFile /etc/letsencrypt/live/complete-concrete-concise.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/complete-concrete-concise.com/privkey.pem

2

Please show:
/etc/letsencrypt/renewal/complete-concrete-concise.com.conf

3

What is that full path?

4

The contents of the file (with the account number redacted):

renew_before_expiry = 30 days

version = 0.26.1
archive_dir = /etc/letsencrypt/archive/complete-concrete-concise.com
cert = /etc/letsencrypt/live/complete-concrete-concise.com/cert.pem
privkey = /etc/letsencrypt/live/complete-concrete-concise.com/privkey.pem
chain = /etc/letsencrypt/live/complete-concrete-concise.com/chain.pem
fullchain = /etc/letsencrypt/live/complete-concrete-concise.com/fullchain.pem

Options used in the renewal process

[renewalparams]
account = [redacted]
authenticator = apache
installer = apache
server = https://acme-v02.api.letsencrypt.org/directory

5

That look ok/good…

6

The full path is:

/var/www/complete-concrete-concise.com/public_html/.well-known/acme-challenge

7

That also look ok/good.
Please place a sample test file “1234” in the challenge folder:
echo "works" > /var/www/complete-concrete-concise.com/public_html/.well-known/acme-challenge/1234

8

Done. There is also a sample file ‘goodbye’ - which I had placed there earlier.

9

I see:
You have reached test file 1234
after http 301 redirect to https

10

Yes, I redirect all http requests to https.

<VirtualHost *:80>
# The ServerName directive sets the request scheme, hostname and port that
# the server uses to identify itself. This is used when creating
# redirection URLs. In the context of virtual hosts, the ServerName
# specifies what hostname must appear in the request’s Host: header to
# match this virtual host. For the default virtual host (this file) this
# value is not decisive as it is used as a last resort host regardless.
# However, you must set it for any further virtual host explicitly.
#ServerName www.example.com

    ServerAdmin webmaster@localhost
    ServerName complete-concrete-concise.com
    ServerAlias www.complete-concrete-concise.com
    DocumentRoot /var/www/complete-concrete-concise.com/public_html
    #DocumentRoot /var/www/html

    # Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
    # error, crit, alert, emerg.
    # It is also possible to configure the loglevel for particular
    # modules, e.g.
    #LogLevel info ssl:warn

    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined

    # For most configuration files from conf-available/, which are
    # enabled or disabled at a global level, it is possible to
    # include a line for only one particular virtual host. For example the
    # following line enables the CGI configuration for this host only
    # after it has been globally disabled with "a2disconf".
    #Include conf-available/serve-cgi-bin.conf

RewriteEngine on
RewriteCond %{SERVER_NAME} =www.complete-concrete-concise.com [OR]
RewriteCond %{SERVER_NAME} =complete-concrete-concise.com
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]

11

Certbot follows redirections so that shouldn’t be a problem.
Do yo use anything in the logs?
Try this too:
certbot renew --dry-run --cert-name complete-concrete-concise.com --webroot -w /var/www/complete-concrete-concise.com/public_html

1 Like
12

Ok, that produces very encouraging results:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/complete-concrete-concise.com.conf

Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator webroot, Installer apache
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for complete-concrete-concise.com
http-01 challenge for faqbite.com
http-01 challenge for www.complete-concrete-concise.com
http-01 challenge for www.faqbite.com
Using the webroot path /var/www/complete-concrete-concise.com/public_html for all unmatched domains.
Waiting for verification...
Cleaning up challenges

new certificate deployed with reload of apache server; fullchain is
/etc/letsencrypt/live/complete-concrete-concise.com/fullchain.pem

** DRY RUN: simulating 'certbot renew' close to cert expiry
** (The test certificates below have not been saved.)

Congratulations, all renewals succeeded. The following certs have been renewed:
/etc/letsencrypt/live/complete-concrete-concise.com/fullchain.pem (success)
** DRY RUN: simulating 'certbot renew' close to cert expiry
** (The test certificates above have not been saved.)

13

Well it was a “dry-run”.
Remove that parameter and you should be fine with:
Congratulations, all renewals succeeded. The following certs have been renewed:…

14

Is there anything I need to change with my scripts to ensure it works without problems in future?

15

The problem is with the apache plugin and how it tries to handle the challenge requests.
It is somehow incompatible with your setup.
Using --webroot turns off the apache installer.

16

Once you have obtained a new cert using webroot, the renewal conf will auto update to that method.
All you need to do is:
cerbot renew
or
certbot renew --quiet

17

Thank you very much for all your help!!!

Too bad there is a problem with the apache plugin. As I said, I was pretty sure it was working back in September, but …

Glad to know the renewal conf is updated to use that last successful renewal method.

18

The plugin works (most of the times).
It is just not 100% capable of making every possible config out there work.
Yours is one of those "special" ones that confuses it and well... fails.

Yes, there are several ways to complete the challenge request.
There is no one-size-fits-all; but we all do seem to find our ways :slight_smile:

19

That’s good to know.

I was pretty sure when I set up my directory structure that it was ‘non-standard’. Maybe I will think about making it more ‘standard’, but … since it is working, it is probably safer to leave it alone.

1 Like
20

The method used by the Certbot Apache plugin to perform challenges changed between then and now due to the upcoming removal (in mid-February) of the old method that it used to obtain certificates. If this default hadn't been changed, we would have a flood of people having their renewals unexpectedly break for this reason on a single day. We were hoping to get people to find out about the problems, if any, before that day.

This change should account for the difference that you saw in the success or failure of the Apache plugin on different occasions. I'm sorry that the new method turned out not to work in your configuration. If you have time and interest, we could try to get some more information from you that might help us understand what went wrong and fix it for other people.

The --webroot method that @rg305 suggested is a reasonable workaround if you're not interested in further debugging. Note that if you ever stop serving static files from under /.well-known (for example by installing a CMS or web app that covers all URL paths on the site), the --webroot method will also stop working unless you exempt /.well-known and allow it to continue serving static files.

1 Like