Interestingly enough, I have another domain on this server which renews just fine, so I’m obviously doing something wrong on this particular one. I’ve read lots and tried lots but have so far been unable to see what I’m doing wrong.
-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/scammell.co.uk.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for scammell.co.uk
http-01 challenge for www.scammell.co.uk
Waiting for verification...
Cleaning up challenges
Attempting to renew cert (scammell.co.uk) from /etc/letsencrypt/renewal/scammell.co.uk.conf produced an unexpected error: Failed authorization procedure. scammell.co.uk (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://scammell.co.uk/.well-known/acme-challenge/r59KeXZJcP2szS_CwC-Sr9sWoL8c1J_t3kR8BJs-gSE: "<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>", www.scammell.co.uk (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.scammell.co.uk/.well-known/acme-challenge/FMGcqIY0n9lQX9kxtiWg2Yy9kf8rPo_4Z7hoRj-lhXM: "<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>". Skipping.
My web server is (include version): nginx/1.13.12
The operating system my web server runs on is (include version): Ubuntu 16.04.1, 4.13.0-43-generic
My hosting provider, if applicable, is: Digital Ocean
I can login to a root shell on my machine (yes or no, or I don’t know): yes
I’m using a control panel to manage my site (no, or provide the name and version of the control panel): no
It looks like you are using the webroot authenticator, which tries to put the response file in a specific location on your filesystem so the webserver will serve it in response to the challenge. Is it possible you've moved directories around since you first issued the certificate? As @rg305 mentioned, you should check in /etc/letsencrypt/renewal/scammell.co.uk.conf and make sure the the configured webroot directory matches the directory where you just put .well-known-acme-challenge/test.txt.
You may also want to try switching to the Nginx plugin, which can manage things for you more automatically, like figuring out where to put the file, and reloading your webserver once the new certificate is available.
@rg305's diagnosis seems exactly right to me. In IPv4 the test file returns a redirect to the HTTPS version of the site, which then returns "ugh". In IPv6, the test file returns a 404 error. So, the problem seems to be the discrepancy between the IPv4 and IPv6 versions of the site.
To add a little detail: If @rg305’s and @schoen’s theory is correct, the fix will be to add [::]:443 to the listen directives of each of your virtual hosts that serves SSL/TLS. Or to remove the AAAA (IPv6) record from your DNS.
Since the site returns an nginx default page in IPv6, it seems like the server is already fully IPv6-capable and the [::]:443 solution should work fine.
I’m bowled over by the speed and quality of these responses. I’m travelling for the next couple of hours, but I’ll try this change as soon as I can. Thank you to everyone!
Okay, I tried the above options and there's no change.
I've replaced the webroot plugin with the nginx plugin and I have a functioning Let's Encrypt certificate again! However, I run into the same error when trying to renew that (the permissions error).
Here are some more details that were requested in the thread so far.
version = 0.17.0
archive_dir = /etc/letsencrypt/archive/scammell.co.uk
cert = /etc/letsencrypt/live/scammell.co.uk/cert.pem
privkey = /etc/letsencrypt/live/scammell.co.uk/privkey.pem
chain = /etc/letsencrypt/live/scammell.co.uk/chain.pem
fullchain = /etc/letsencrypt/live/scammell.co.uk/fullchain.pem
# Options used in the renewal process
[renewalparams]
authenticator = webroot
installer = None
account = 20bfe4dd01f2ae0b164f302d3af6cbf3
renew_hook = /usr/sbin/service nginx reload
[[webroot_map]]
scammell.co.uk = /var/www/scammell.co.uk/htdocs
www.scammell.co.uk = /var/www/scammell.co.uk/htdocs
You know what, I think there are enough reasons to completely rebuild this server, so I'm not going to take this any further. Thank you for your help so far everyone, I'm going to do a clean Ubuntu 18.04 LTS build on a fresh DO server and follow some best practices.