Renewing fails after switch to http01-challenge - Getting 404


#1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:

mail.increaseyourskills.com

I ran this command:

certbot renew --dry-run

It produced this output:

The following certs could not be renewed:
/etc/letsencrypt/live/mail.increaseyourskills.com/fullchain.pem (failure)
** DRY RUN: simulating ‘certbot renew’ close to cert expiry
** (The test certificates above have not been saved.)


1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:

My web server is (include version):

nginx 1.14

The operating system my web server runs on is (include version):

Ubuntu 18.04.1 LTS (bionic)

My hosting provider, if applicable, is:

Hetzner (https://www.hetzner.de/)

I can login to a root shell on my machine (yes or no, or I don’t know):

Yes.

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):

No.

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot):

certbot 0.28.0

My additional thoughts:

I read through some similar souding threads and already tested a little bit of some advice.
For instance I created a file in the webroot under ./well-known/acme-challenge. I could access this file without any issues.

I also made sure to have the right file and group permissions on this folder.

Output of “dpkg -l | grep certbot”:
ii certbot 0.28.0-1+ubuntu18.04.1+certbot+4 all automatically configure HTTPS using Let’s Encrypt
ii python-augeas 0.5.0-1+ubuntu18.04.1+certbot+1 all Python bindings for Augeas
ii python-certbot-nginx 0.28.0-1+ubuntu18.04.1+certbot+3 all transitional dummy package
ii python3-acme 0.28.0-1+ubuntu18.04.1+certbot+3 all ACME protocol library for Python 3
ii python3-certbot 0.28.0-1+ubuntu18.04.1+certbot+4 all main library for certbot
ii python3-certbot-nginx 0.28.0-1+ubuntu18.04.1+certbot+3 all Nginx plugin for Certbot
ii python3-configobj 5.0.6-2+ubuntu18.04.1+certbot+1 all simple but powerful config file reader and writer for Python 3
ii python3-future 0.15.2-4+ubuntu18.04.1+certbot+3 all Clean single-source support for Python 3 and 2 - Python 3.x
ii python3-parsedatetime 2.4-3+ubuntu18.04.1+certbot+3 all Python 3 module to parse human-readable date/time expressions
ii python3-requests-toolbelt 0.8.0-1+ubuntu18.04.1+certbot+1 all Utility belt for advanced users of python3-requests
ii python3-zope.component 4.3.0-1+ubuntu18.04.1+certbot+3 all Zope Component Architecture
ii python3-zope.hookable 4.0.4-4+ubuntu18.04.1+certbot+1 amd64 Hookable object support
ii python3-zope.interface 4.3.2-1+ubuntu18.04.1+certbot+1 amd64 Interfaces for Python3

Content of Virtual Host file for mail.increaseyourskills.com:

server {
if ($host = mail.increaseyourskills.com) {
return 301 https://$host$request_uri;
} # managed by Certbot

listen 80;
listen [::]:80;
server_name mail.increaseyourskills.com;
return 301 https://$server_name$request_uri;

}

server {
# SSL configuration
#
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name mail.increaseyourskills.com;
client_max_body_size 2000m;

ssl_protocols TLSv1.2;
ssl_ciphers EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:EECDH+3DES:RSA+3DES:!MD5;
ssl_prefer_server_ciphers On;
ssl_trusted_certificate /etc/letsencrypt/live/mail.increaseyourskills.com/chain.pem;
ssl_certificate /etc/letsencrypt/live/increaseyourskills.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/increaseyourskills.com/privkey.pem; # managed by Certbot
ssl_session_cache shared:SSL:128m;
ssl_session_timeout 1h;
add_header Strict-Transport-Security "max-age=31557600; includeSubDomains";
add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-Content-Type-Options "nosniff" always;
add_header X-Xss-Protection "1";
#add_header Content-Security-Policy "default-src 'self'; script-src 'self' *.google-analytics.com";
ssl_stapling on;
ssl_stapling_verify on;
#
# Note: You should disable gzip for SSL traffic.
# See: https://bugs.debian.org/773332
#
# Read up on ssl_ciphers to ensure a secure configuration.
# See: https://bugs.debian.org/765782
#
# Self signed certs generated by the ssl-cert package
# Don't use them in a production server!
#
# include snippets/snakeoil.conf;

root /var/www/webmail;

# Add index.php to the list if you are using PHP
index index.php index.html index.htm index.nginx-debian.html;

location / {
	# First attempt to serve request as file, then
	# as directory, then fall back to displaying a 404.
	try_files $uri $uri/ /index.php?q=$uri&$args;
}
location ~ ^/(README.md|INSTALL|LICENSE|CHANGELOG|UPGRADING)$ {
	deny all;
}
location ~ ^/(config|temp|logs)/ {
	deny all;
}
#location ~ /\. {
#	deny all;
#	access_log off;
#	log_not_found off;
#}

    location ^~ /.well-known/acme-challenge/ {
        allow all;
            default_type "text/plain";
    }


# pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
#
location ~ [^/]\.php(/|$) {
	fastcgi_split_path_info ^(.+?\.php)(/.*)$;
	if (!-f $document_root$fastcgi_script_name) {
		return 404;
	}
	include fastcgi_params;

	client_max_body_size 2000m;
	include snippets/fastcgi-php.conf;

#	# With php7.0-cgi alone:
#	fastcgi_pass 127.0.0.1:9000;
	# With php7.0-fpm:
	fastcgi_pass unix:/run/php/php7.0-fpm.sock;
}

}

This domain is the only one which is not compatible for the renewing process for an unknown reason (the other domains are working like a charm)
I would appreciate your help.

Best regards

Christian


#2

What was the earlier part of Certbot’s output?


#3

Hi @digitalforensics

your basic configuration is ok, you have an ipv4 and an ipv6 address, both redirect to https, that’s good.

Your test file 1234 answers with a 200, so you have found your correct webroot.

So try

certbot run -a webroot -i nginx -w /var/www/webmail -d mail.increaseyourskills.com

But: Checking your domain ( https://check-your-website.server-daten.de/?q=mail.increaseyourskills.com ) there is a certificate with a lot of domain names:

CN=increaseyourskills.com
	23.01.2019
	23.04.2019
	ajenti.increaseyourskills.com, certificate.increaseyourskills.com, 
elearning.increaseyourskills.com, files.increaseyourskills.com, 
increaseyourskills.com, mail.increaseyourskills.com, 
order.increaseyourskills.com, schoeck.increaseyourskills.com, 
www.increaseyourskills.com - 9 entries

Do you want to change that - now one certificate with one domain name?


#4

Hello everyone,

@mnordhoff
The earlier part of Certbots output shows the successfull validation for the domains mentioned by @JuergenAuer.

I wonder if your command Juergen would create a new certificate? The current certificate is expiring on 23. April this year and I don’t want to destroy it or replace it with something not working.

I was updating certbot today from version 0.26 to 0.28.
I want to keep all other domain aliases as well. All other aliases are working fine with:
certbot renew --dry-run.

Can I try your suggested command or is it unsafe for my purpose?

Best regards

Christian


#5

Can you paste it here?


#6

Sure I can:

Saving debug log to /var/log/letsencrypt/letsencrypt.log


Processing /etc/letsencrypt/renewal/files.increaseyourskills.com.conf


Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator nginx, Installer nginx
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for files.increaseyourskills.com
Waiting for verification…
Cleaning up challenges


new certificate deployed with reload of nginx server; fullchain is
/etc/letsencrypt/live/files.increaseyourskills.com/fullchain.pem



Processing /etc/letsencrypt/renewal/elearning.increaseyourskills.com.conf


Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator nginx, Installer nginx
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for elearning.increaseyourskills.com
Waiting for verification…
Cleaning up challenges


new certificate deployed with reload of nginx server; fullchain is
/etc/letsencrypt/live/elearning.increaseyourskills.com/fullchain.pem



Processing /etc/letsencrypt/renewal/increaseyourskills.com.conf


Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator nginx, Installer nginx
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for ajenti.increaseyourskills.com
http-01 challenge for certificate.increaseyourskills.com
http-01 challenge for elearning.increaseyourskills.com
http-01 challenge for files.increaseyourskills.com
http-01 challenge for increaseyourskills.com
http-01 challenge for mail.increaseyourskills.com
http-01 challenge for order.increaseyourskills.com
http-01 challenge for schoeck.increaseyourskills.com
http-01 challenge for www.increaseyourskills.com
Waiting for verification…
Cleaning up challenges


new certificate deployed with reload of nginx server; fullchain is
/etc/letsencrypt/live/increaseyourskills.com/fullchain.pem



Processing /etc/letsencrypt/renewal/schoeck.increaseyourskills.com.conf


Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator nginx, Installer nginx
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for schoeck.increaseyourskills.com
Waiting for verification…
Cleaning up challenges


new certificate deployed with reload of nginx server; fullchain is
/etc/letsencrypt/live/schoeck.increaseyourskills.com/fullchain.pem



Processing /etc/letsencrypt/renewal/order.increaseyourskills.com.conf


Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator nginx, Installer nginx
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for order.increaseyourskills.com
Waiting for verification…
Cleaning up challenges


new certificate deployed with reload of nginx server; fullchain is
/etc/letsencrypt/live/order.increaseyourskills.com/fullchain.pem



Processing /etc/letsencrypt/renewal/mail.increaseyourskills.com.conf


Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for mail.increaseyourskills.com
Waiting for verification…
Cleaning up challenges
Attempting to renew cert (mail.increaseyourskills.com) from /etc/letsencrypt/renewal/mail.increaseyourskills.com.conf produced an unexpected error: Failed authorization procedure. mail.increaseyourskills.com (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://mail.increaseyourskills.com/.well-known/acme-challenge/w-Xmsi-sgHT9O9lEA2MXwobvqXQ2mBsknFuiMt4yUMc: “\r\n404 Not Found\r\n<body bgcolor=“white”>\r\n

404 Not Found

\r\n
”. Skipping.

Processing /etc/letsencrypt/renewal/certificate.increaseyourskills.com.conf


Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator nginx, Installer nginx
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for certificate.increaseyourskills.com
Waiting for verification…
Cleaning up challenges


new certificate deployed with reload of nginx server; fullchain is
/etc/letsencrypt/live/certificate.increaseyourskills.com/fullchain.pem



Processing /etc/letsencrypt/renewal/ajenti.increaseyourskills.com.conf


Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator nginx, Installer nginx
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for ajenti.increaseyourskills.com
Waiting for verification…
Cleaning up challenges


new certificate deployed with reload of nginx server; fullchain is
/etc/letsencrypt/live/ajenti.increaseyourskills.com/fullchain.pem


#7

If you want to create a new certificate, alle challenges must pass.

So if all other domains of this certificate are correct, what’s the difference with this domain?

Then I would do nothing and wait.

But this

means: The complete certificate works.

So you don’t need this certificate.

So delete it with certbot delete certificate-name.

PS: Looks like you have an old certificate with a not-longer-used configuration. But the certificate with 9 names works.


#8

@JuergenAuer I don’t want to create a new certificate. I just want to make sure that everything goes smoothly with the next automatic renewing.

If I understand you correctly, the main certificate with the 9 names is working as supposed, but the individual certificate for mail.increaseyourskills.com is not working and therefor can be deleted. That would mean there is one lets call it big certificate for all domains and then one for each domain individually?

The main difference between the other domains is the underlying service. The mail domain is pointing to a roundcube installation.

I would like to test the site first, if it’s working without the broken configuration. Could I just try to move the certificate to some other place?. What would be the location for the cert? I assume

/etc/letsencrypt/live/mail.increaseyourskills.com

In this folder there are all 9 other domain aliases with individual subfolders.

Best regards

Christian


#9

But you use already this “big certificate” with your Roundcube:

I have checked your configuration via https://check-your-website.server-daten.de/?q=mail.increaseyourskills.com - then I saw a Grade A (not much sites have Grade A),

then I saw the “big certificate”

CN=increaseyourskills.com
	23.01.2019
	23.04.2019
	ajenti.increaseyourskills.com, certificate.increaseyourskills.com, 
elearning.increaseyourskills.com, files.increaseyourskills.com, 
increaseyourskills.com, mail.increaseyourskills.com, 
order.increaseyourskills.com, schoeck.increaseyourskills.com, 
www.increaseyourskills.com - 9 entries

And checking your url via browser

https://mail.increaseyourskills.com/

there is your roundcube with this big certificate.

Your Roundcube with this domain name uses this big certificate.

So which instance should use the certificate with one domain name if this domain name has already a working certificate?


#10

Hello Juergen,

I think I fixed the issue now by myself.
I rechecked the configuration under /etc/letsencrypt/renewal/mail…conf
and thanks for you pointing out something was wrong with the config file.

It was set to “/var/www/webmail/public_html/” instead of “/var/www/webmail”.
The webroot of the virtual host is just /var/www/webmail so I adapted the file and re-run the dry-run and got the following result:


** DRY RUN: simulating ‘certbot renew’ close to cert expiry
** (The test certificates below have not been saved.)

Congratulations, all renewals succeeded. The following certs have been renewed:
/etc/letsencrypt/live/files.increaseyourskills.com/fullchain.pem (success)
/etc/letsencrypt/live/elearning.increaseyourskills.com/fullchain.pem (success)
/etc/letsencrypt/live/increaseyourskills.com/fullchain.pem (success)
/etc/letsencrypt/live/schoeck.increaseyourskills.com/fullchain.pem (success)
/etc/letsencrypt/live/order.increaseyourskills.com/fullchain.pem (success)
/etc/letsencrypt/live/mail.increaseyourskills.com/fullchain.pem (success)
/etc/letsencrypt/live/certificate.increaseyourskills.com/fullchain.pem (success)
/etc/letsencrypt/live/ajenti.increaseyourskills.com/fullchain.pem (success)
** DRY RUN: simulating ‘certbot renew’ close to cert expiry
** (The test certificates above have not been saved.)


So far so good :slight_smile: I thank you and mnordhoff for your patience and wisdom.

And I will mark one of your last comments as the accepted answer.

Have a nice evening and Cheers :slight_smile:

Christian

P.S.: I can’t really answer your question with the instances. I got asked from a friend of mine if I could help him with the issue. The server got setup by someone which is not reachable anymore. Using this forum was a great experience overall.