[Solved] Webroot Renewal Fails

mitchel.nieto · October 31, 2017, 3:00am

Hello Everyone,
I am just lost, I know that there are a number of threads on this subject, but I just cannot seem to get this resolved. I have multiple domains on this server, all of which successfully renew except for the 1 domain (my mail domain) that I created using -Webroot.

sudo certbot certonly --rsa-key-size 4096 --renew-by-default --webroot -w /var/www/html/mail.nietostack.com/ -d mail.nietostack.com --renew-by-default

Now, when I run sudo certbot renew --dry-run it fails with the below output:

Attempting to renew cert from /etc/letsencrypt/renewal/mail.nietostack.com.conf produced an unexpected error: Failed authorization procedure. mail.nietostack.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://mail.nietostack.com/.well-known/acme-challenge/JU54bI9HuOP7Porh_XXsmNEOGpIOBBR8zjz6t6Gm67A: "
** DRY RUN: simulating ‘certbot renew’ close to cert expiry
** (The test certificates below have not been saved.)

The following certs were successfully renewed:
/etc/letsencrypt/live/www.mitchelnieto.com/fullchain.pem (success)
/etc/letsencrypt/live/nietostack.com/fullchain.pem (success)
/etc/letsencrypt/live/www.mitchelnieto.tech/fullchain.pem (success)
/etc/letsencrypt/live/nietofamily.com/fullchain.pem (success)

The following certs could not be renewed:
/etc/letsencrypt/live/mail.nietostack.com/fullchain.pem (failure)
** DRY RUN: simulating ‘certbot renew’ close to cert expiry
** (The test certificates above have not been saved.)
1 renew failure(s), 0 parse failure(s)

Domain: mail.nietostack.com
_ Type: unauthorized_
_ Detail: Invalid response from_
_ http://mail.nietostack.com/.well-known/acme-challenge/JU54bI9HuOP7Porh_XXsmNEOGpIOBBR8zjz6t6Gm67A:_
_ “_
_ _
_ 404 Not Found_
_ _
_

Not Found
_
_ <p”_
_ To fix these errors, please make sure that your domain name was_
_ entered correctly and the DNS A record(s) for that domain_
_ contain(s) the right IP address._

I have modified the permissions of .well-known/acme-challenge and added a test file (mail.nietostack.com/.well-known/acme-challenge/OK), but each time I get:

Not Found

The requested URL /.well-known/acme-challenge/OK was not found on this server.

Apache/2.4.18 (Ubuntu) Server at mail.nietostack.com Port 80

From the other threads, it looks like this is an redirection (port 80 > 443) error that prevents -Webroot certificates from renewing? Any help is greatly appreciated.

Requested Information:
My domain is: mail.nietostack.com

I ran this command: sudo certbot renew --dry-run

It produced this output:
IMPORTANT NOTES:

The following errors were reported by the server:

Domain: mail.nietostack.com
Type: unauthorized
Detail: Invalid response from
http://mail.nietostack.com/.well-known/acme-challenge/JU54bI9HuOP7Porh_XXsmNEOGpIOBBR8zjz6t6Gm67A:
"
404 Not Found
Not Found
<p"
To fix these errors, please make sure that your domain name was
entered correctly and the DNS A record(s) for that domain
contain(s) the right IP address.

My web server is (include version): Apache2.4.18

The operating system my web server runs on is (include version): Ubuntu 16.04

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): NO

rg305 · October 31, 2017, 3:13am

The first command string uses “–renew-by-default” twice - not sure if that matters…

mail.nietostack.com has IPv4 and IPv6 addresses.
NOTE: LE prefers IPv6 over IPv4
You must ensure the site is accessible via IPv6 or remove the IPv6 entries from DNS.

Based on the webroot path "–webroot -w /var/www/html/mail.nietostack.com/"
The OK file (mail.nietostack.com/.well-known/acme-challenge/OK) should be found at:
/var/www/html/mail.nietostack.com/.well-known/acme-challenge/OK
Is that where the OK file is located?

mitchel.nieto · October 31, 2017, 3:31am

rg305, thanks for the quick reply.

I believe that there are no issues with IPv6 connectivity.

Yes, the OK file is located at /var/www/html/mail.nietostack.com/.well-known/acme-challenge/OK

rg305 · October 31, 2017, 4:18am

As I suspected, the OK file can’t be reached via IPv6:

But that is no consolation, as the IPv4 site also shows 404:

The OK file is just not being served by your system.
Try the standard “test.txt” file name instead.

mitchel.nieto · October 31, 2017, 11:29am

Thanks again rg305.

I have uploaded test.txt

Osiris · October 31, 2017, 1:07pm

You’ve got directory listing enabled: http://mail.nietostack.com/.well-known/acme-challenge/

Looks like you called the file Rest.txt

I’m on IPv4 only currently. Do others with IPv6 also see the files in the directory listing?

rg305 · October 31, 2017, 2:59pm

I do see the “Rest.txt” file and contents “this is a test” via both IPv4 and IPv6.

Please retry the certbot renewal command.

Also: The challenge file has size zero

mitchel.nieto · October 31, 2017, 6:36pm

Thanks both rg305 and Osiris.

I have tried the renew again, still fails. I have uploaded Sample.Txt, which I see as well.

Attempting to renew cert from /etc/letsencrypt/renewal/mail.nietostack.com.conf produced an unexpected error: Failed authorization procedure. mail.nietostack.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://mail.nietostack.com/.well-known/acme-challenge/T5xE6bQK48a8x2K52VWSBiLlEUDlEdHseaI51NA0wQ0: "
404 Not Found
Not Found
<p". Skipping.

schoen · October 31, 2017, 7:12pm

You’re sure that you’re placing these test files in /var/www/html/mail.nietostack.com/?

Can you post the log from /var/log/letsencrypt associated with your failed Certbot runs?

mitchel.nieto · October 31, 2017, 8:39pm

letsencrypt.txt (155.2 KB)

Sure thing, here is the letsencrypt.log file.

Thanks Schoen.

Osiris · October 31, 2017, 8:56pm

Also, check your webservers access log. It should tell you where it looked for the file…

mitchel.nieto · October 31, 2017, 9:08pm

access.txt (36.2 KB)

Today’s log, if I am reading this correctly, it was http://mail.nietostack.com/.well-known/acme-challenge/. I am not sure how I even got this certificate in the first place with all the issues renewing, I have not made any changes since I got the cert.

Osiris · October 31, 2017, 9:22pm

Hm, sorry, the actual locations on the file system where Apache looked are in the error log, not the access log. My bad!

schoen · October 31, 2017, 10:23pm

Your log reflects a completely different error and failure mode based on using the a renewal command that selected the apache plugin with the TLS-SNI-01 challenge type, rather than the webroot plugin with the HTTP-01 challenge type.

There is something really confusing about this (including for trying to diagnose what’s going on). These different plugins try to authenticate in different ways and change different things on your web server. Are you sure that you really used only --webroot when originally obtaining the certificate? Could you post the output of certbot certificates and any files that are in /etc/letsencrypt/renewal on this system?

schoen · October 31, 2017, 10:25pm

In fact, the log you posted shows that you ran a command with --duplicate and --apache, which is again different from the commands you mentioned before (sudo certbot certonly [...] --webroot and sudo certbot renew --dry-run) and would probably fail for different, unrelated reasons.

mitchel.nieto · October 31, 2017, 11:21pm

Thanks Schoen, I really do appreciate the help!

Found the following certs:
Certificate Name: www.mitchelnieto.com
Domains: mitchelnieto.com www.mitchelnieto.com
Expiry Date: 2018-01-07 04:14:05+00:00 (VALID: 67 days)
Certificate Path: /etc/letsencrypt/live/www.mitchelnieto.com/fullchain.pem
Private Key Path: /etc/letsencrypt/live/www.mitchelnieto.com/privkey.pem
Certificate Name: nietostack.com
Domains: nietostack.com www.nietostack.com
Expiry Date: 2018-01-07 04:14:14+00:00 (VALID: 67 days)
Certificate Path: /etc/letsencrypt/live/nietostack.com/fullchain.pem
Private Key Path: /etc/letsencrypt/live/nietostack.com/privkey.pem
Certificate Name: www.mitchelnieto.tech
Domains: www.mitchelnieto.tech mitchelnieto.tech
Expiry Date: 2018-01-07 16:38:03+00:00 (VALID: 67 days)
Certificate Path: /etc/letsencrypt/live/www.mitchelnieto.tech/fullchain.pem
Private Key Path: /etc/letsencrypt/live/www.mitchelnieto.tech/privkey.pem
Certificate Name: mail.nietostack.com
Domains: mail.nietostack.com
Expiry Date: 2017-11-11 14:12:00+00:00 (VALID: 10 days)
Certificate Path: /etc/letsencrypt/live/mail.nietostack.com/fullchain.pem
Private Key Path: /etc/letsencrypt/live/mail.nietostack.com/privkey.pem
Certificate Name: nietofamily.com
Domains: nietofamily.com www.nietofamily.com
Expiry Date: 2018-01-09 04:49:07+00:00 (VALID: 69 days)
Certificate Path: /etc/letsencrypt/live/nietofamily.com/fullchain.pem
Private Key Path: /etc/letsencrypt/live/nietofamily.com/privkey.pem

Files in /etc/letsencrypt/renewal

mail.nietostack.com.conf nietofamily.com.conf nietostack.com.conf www.mitchelnieto.com.conf www.mitchelnieto.tech.conf

Do I need to update the -Webroot certificate differently than just: sudo certbot renew?

schoen · October 31, 2017, 11:25pm

Normally sudo certbot renew should renew everything that’s less than 30 days away from expiry. It will use the same authentication method that you originally used when you first obtained the certificate, whatever that was.

In this case, --apache and --webroot are using totally different authentication methods, and if they fail, they fail for very different reasons. Hopefully certbot renew can succeed using the original one that you used when you obtained the certificate; if not, it would be good to see a log file from the failed certbot renew command to understand more about why it failed.

mitchel.nieto · October 31, 2017, 11:31pm

Alright thanks for the clarification, this is my first renewal as I just stood up the webserver about 90 days ago… I ran the actual sudo certbot renew and it failed out again with the same error.

Processing /etc/letsencrypt/renewal/mail.nietostack.com.conf

Cert is due for renewal, auto-renewing…
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for mail.nietostack.com
Waiting for verification…
Cleaning up challenges
Unable to clean up challenge directory /var/www/html/mail.nietostack.com/.well-known/acme-challenge
Attempting to renew cert from /etc/letsencrypt/renewal/mail.nietostack.com.conf produced an unexpected error: Failed authorization procedure. mail.nietostack.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://mail.nietostack.com/.well-known/acme-challenge/DEiFARiMwgGeCfYtv_7MWqX-U6ikBZEkG0QizLxTVcE: "
404 Not Found
Not Found
<p". Skipping.

Processing /etc/letsencrypt/renewal/nietofamily.com.conf

Cert not yet due for renewal

The following certs are not due for renewal yet:
/etc/letsencrypt/live/www.mitchelnieto.com/fullchain.pem (skipped)
/etc/letsencrypt/live/nietostack.com/fullchain.pem (skipped)
/etc/letsencrypt/live/www.mitchelnieto.tech/fullchain.pem (skipped)
/etc/letsencrypt/live/nietofamily.com/fullchain.pem (skipped)
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/mail.nietostack.com/fullchain.pem (failure)
1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:

The following errors were reported by the server:

Domain: mail.nietostack.com
Type: unauthorized
Detail: Invalid response from
http://mail.nietostack.com/.well-known/acme-challenge/DEiFARiMwgGeCfYtv_7MWqX-U6ikBZEkG0QizLxTVcE:
"
404 Not Found
Not Found
<p"
To fix these errors, please make sure that your domain name was
entered correctly and the DNS A record(s) for that domain
contain(s) the right IP address.

schoen · October 31, 2017, 11:40pm

Could we see the log file from that attempt?

mitchel.nieto · October 31, 2017, 11:43pm

letsencrypt.txt (23.1 KB)

Sure thing.