Cannot renew certs when redirecting http to https

1

Hi,

I have a hard time getting letsencrypt to work and slowly and surely its getting frustrating :frowning:
I use nginx as a proxy for a couple of lxc containers (20 or so) and I´m redirecting all http requests to https. Here is a example config for one container

## example.com ##
server {
    listen       15.19.247.xxx:80;
    server_name  example.com www.example.com;
    access_log  /var/log/nginx/log/example.access.log;
    error_log  /var/log/nginx/log/example.log;
    # This is for certification renewal
   include /etc/nginx/snippets/letsencrypt-acme-challenge.conf;

    ## send request back to container ##
    location / {
    return 301 https://$server_name$request_uri;
  }
}
server {
    listen   15.19.247.xxx:443;
    server_name  example.com www.example.com;
    ssl on;
    ssl_session_timeout 24h;
    ssl_ciphers AES256+EECDH:AES256+EDH:!aNULL;
    ssl_prefer_server_ciphers on;
    ssl_stapling on;
    ssl_stapling_verify on;
    resolver 8.8.4.4 8.8.8.8 valid=300s;
    resolver_timeout 10s;
    spdy_keepalive_timeout 300;
    spdy_headers_comp 9;
    add_header Strict-Transport-Security max-age=63072000;
    add_header X-Frame-Options DENY;
    add_header X-Content-Type-Options nosniff;
    ssl_trusted_certificate /var/lib/certs/ocsp.trust;
    ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
    ssl_session_cache shared:SSL:10m;
    ssl_protocols  TLSv1 TLSv1.1 TLSv1.2;  # don’t use SSLv3 ref: POODLE

    access_log  /var/log/nginx/log/ssl.example.com.access.log;
    error_log  /var/log/nginx/log/ssl.example.com.log;
    root   /usr/share/nginx/html;
    index  index.html index.htm;

    client_max_body_size 30M;

    ## send request back to container ##
    location / {
     proxy_pass  http://container10/;
     proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
     proxy_redirect off;
     proxy_buffering off;
     proxy_set_header        Host            $host;
     proxy_set_header        X-Real-IP       $remote_addr;
     proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
     proxy_set_header        X-Forwarded-Proto $scheme;
   }

 }
# End example.com ##

And here is the letsencrypt-acme-challenge.conf file found here in the forum:

cat /etc/nginx/snippets/letsencrypt-acme-challenge.conf 
#############################################################################
# Configuration file for Let's Encrypt ACME Challenge location
# This file is already included in listen_xxx.conf files.
# Do NOT include it separately!
#############################################################################
#
# This config enables to access /.well-known/acme-challenge/xxxxxxxxxxx
# on all our sites (HTTP), including all subdomains.
# This is required by ACME Challenge (webroot authentication).
# You can check that this location is working by placing ping.txt here:
# /var/www/letsencrypt/.well-known/acme-challenge/ping.txt
# And pointing your browser to:
# http://xxx.domain.tld/.well-known/acme-challenge/ping.txt
#
# Sources:
# https://community.letsencrypt.org/t/howto-easy-cert-generation-and-renewal-with-nginx/3491
#
#############################################################################

# Rule for legitimate ACME Challenge requests (like /.well-known/acme-challenge/xxxxxxxxx)
# We use ^~ here, so that we don't check other regexes (for speed-up). We actually MUST cancel
# other regex checks, because in our other config files have regex rule that denies access to files with dotted names.
location ^~ /.well-known/acme-challenge/ {

    # Set correct content type. According to this:
    # https://community.letsencrypt.org/t/using-the-webroot-domain-verification-method/1445/29
    # Current specification requires "text/plain" or no content header at all.
    # It seems that "text/plain" is a safe option.
    default_type "text/plain";

    # This directory must be the same as in /etc/letsencrypt/cli.ini
    # as "webroot-path" parameter. Also don't forget to set "authenticator" parameter
    # there to "webroot".
    # Do NOT use alias, use root! Target directory is located here:
    # /var/www/common/letsencrypt/.well-known/acme-challenge/
    root         /usr/share/nginx/html;
}

# Hide /acme-challenge subdirectory and return 404 on all requests.
# It is somewhat more secure than letting Nginx return 403.
# Ending slash is important!
location = /.well-known/acme-challenge/ {
    return 404;
}

The configuration file for that particular domain looks like this:

/etc/letsencrypt/renewal/example.com.conf

# renew_before_expiry = 30 days
cert = /etc/letsencrypt/live/example.com/cert.pem
privkey = /etc/letsencrypt/live/example.com/privkey.pem
chain = /etc/letsencrypt/live/example.com/chain.pem
fullchain = /etc/letsencrypt/live/example.com/fullchain.pem
version = 0.8.0

# Options and defaults used in the renewal process
[renewalparams]
installer = None
authenticator = webroot
webroot_path = /usr/share/nginx/html,
account = xxxxxxxxxxxxxxxxxxxx
[[webroot_map]]
www.example.com = /usr/share/nginx/html
example.com = /usr/share/nginx/html

And the when running certbot from cron with: /opt/letsencrypt/certbot-auto renew --pre-hook "service nginx stop" --post-hook "service nginx start"

I get following error:

2016-06-13 22:57:51,662:WARNING:certbot.renewal:Attempting to renew cert from /etc/letsencrypt/renewal/example.com.conf produced an unexpected error: Failed authorization procedure. example.com (http-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Could not connect to http://example.com/.well-known/acme-challenge/kHc3n5aDY-5-hEBd66l69UUBN3XGRBjKimtlpeB-hu0, www.example.com (http-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Could not connect to http://www.example.com/.well-known/acme-challenge/CELUpo-_pCmaSF0Yqa_n6FejIHz_34SMoffQx2xGrRo. Skipping.

I hope somebody can help me with this issue. Thank you.

2

Are you sure the configuration by itself is working? It’s just a bit odd that you seem to bind to different IPs there, unless it’s just a random set of digits to mask actual IPs :slight_smile:

3

My mistake with the IPs. There are the same of course :slight_smile:

4

Double-check from another box that you can actually access the resource (for example with wget -S http://server/) - just to make sure that you are able to connect, and the redirect is what you expect it to be.

5

Yes, inside the /.well-known/acme-challenge/ directory I have a index.html which can be accessed in the browser:

http://example.com/.well-known/acme-challenge/

still getting this error

Failed authorization procedure. www.example.com (http-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Could not connect to http://www.example.com/.well-known/acme-challenge/91-iGi8G53Z5jRfE1M6s4V6gf3PgF3BZxb88H7rzeKs, example.com (http-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Could not connect to http://example.com/.well-known/acme-challenge/mV_VU31p0XnJpcx_KWTyHL5ZipYwSJOgfw7GPMW1PBw

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: www.example.com
   Type:   connection
   Detail: Could not connect to
   http://www.example.com/.well-known/acme-challenge/91-iGi8G53Z5jRfE1M6s4V6gf3PgF3BZxb88H7rzeKs

   Domain: example.com
   Type:   connection
   Detail: Could not connect to
   http://example.com/.well-known/acme-challenge/mV_VU31p0XnJpcx_KWTyHL5ZipYwSJOgfw7GPMW1PBw

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.

The cli command was invoked with /opt/letsencrypt/certbot-auto certonly -c /etc/letsencrypt/configs/renew-webroot-example.ini --pre-hook "service nginx stop" --post-hook "service nginx start"

And here is the config for that domain:

cat /etc/letsencrypt/configs/renew-webroot-example.ini

email = no-reply@example.com
domains = example.com, www.example.com
text = True
authenticator = webroot
webroot-path = /usr/share/nginx/html

I´m out of ideas.

6

I had the same problem with a nginx proxy instance. What I did was avoided 301 redirecting the /.well-known directory by having a separate location block

  location /.well-known {
     alias /var/www/letsencrypt/.well-known; # have this as the webroot
 }
  location / {
        return         301 https://$server_name$request_uri;
 }
5 Likes
7

Thank you for your reply. Yes, as you can see I have a separate location block:

location ^~ /.well-known/acme-challenge/ {

    # Set correct content type. According to this:
    # https://community.letsencrypt.org/t/using-the-webroot-domain-verification-method/1445/29
    # Current specification requires "text/plain" or no content header at all.
    # It seems that "text/plain" is a safe option.
    default_type "text/plain";

    # This directory must be the same as in /etc/letsencrypt/cli.ini
    # as "webroot-path" parameter. Also don't forget to set "authenticator" parameter
    # there to "webroot".
    # Do NOT use alias, use root! Target directory is located here:
    # /var/www/common/letsencrypt/.well-known/acme-challenge/
    root         /usr/share/nginx/html;
}

Accessing .well-known/acme-challenge/ from a browser works so this URL doesn´t get redirected to https.

8

During the certificate renewal I watched the contents of the directory

watch -n1 ls

and i could see the files where created:

kwpmIklXr1K_g1yZdnI8ZBNzoEPGUle1XoFlbKi3zw4
test # my test file
vhB_-Xa7C1M6efMpvtR3sFrN_7kDZCgKIb7mGnXspAU

and after throwing the error this files (except my test file) are deleted.

9

You're stopping nginx before the renewal process starts and the validation is performed. That is not necessary with the webroot plugin, and it is in fact what's preventing the validation server from connecting to your site - there's simply no web server listening on port 80 during that time.

The renew example using that --pre-hook (which I assume you got from the certbot documentation) is using the standalone plugin, which would spawn a web server for you, but that's not the case with webroot.

Basically, remove the --pre-hook and replace the --post-hook command with something like service nginx reload and you should be good to go.

4 Likes
10

You're stopping nginx before the renewal process starts and the
validation is performed. That is not necessary with the webroot plugin,
and it is in fact what's preventing the validation server from
connecting to your site - there's simply no web server listening on port
80 during that time.

The renew example using that --pre-hook
(which I assume you got from the certbot documentation) is using the
standalone plugin, which would spawn a web server for you, but that's
not the case with webroot.

Basically, remove the --pre-hook and replace the --post-hook command with something like service nginx reload and you should be good to go.

OMG you´re so right. Thank you so much it works now.

3 Likes
11

If it makes you feel any better, I was writing a rather long reply about debugging connectivity problems from the validation server when I finally noticed that --pre-hook and made the connection. Sometimes the most obvious thing is the most unnoticed. :smile:

3 Likes
12

Damn right. I can´t tell you how much time I´ve spent getting this *** to work :slight_smile:

2 Likes
13

Happens to the best of us :slight_smile:

2 Likes
14

Brilliant, thanks … I was missing the location / { before the return 301 https://blahblah$request_uri; } … now have webroot authentication working again.

1 Like
15

I've tried many variations on the suggestions but I'm still stuck. I'm doing HTTP to HTTPS redirect via nginx and as well I'm redirect from non-www to www. This is using the certbot from the certbot PPA for ubuntu, ubuntu 16.04.

Relevant parts of (the latest version of) server blocks:

server {
server_name mydomain.com;
listen 80;
listen [::]:80;

location /.well-known/acme-challenge/ {}

location / {
    return 301 https://www.mydomain.com$request_uri;
}

}

server {
server_name www.mydomain.com
gzip on;
gzip_proxied any;
gzip_types text/plain text/xml text/css application/javascript;
gzip_vary on;
gzip_disable "MSIE [1-6].(?!.*SV1)";

# SSL configuration
    
listen 443 ssl http2;
listen [::]:443 ssl http2; 
include snippets/ssl-www.mydomain.com.conf;
include snippets/ssl-params.conf;

access_log  /var/log/nginx/mydomain.com.access.log; 
error_log /var/log/nginx/mydomain.com.error.log;

root /var/www/www.mydomain.com;
index index.php;

location ~* \.(?:js|css|png|jpe?g|gif|ico)$ {
    expires max;
    break;
}

location / {
    try_files $uri $uri/ /index.php?q=$request_uri;
}


# If directly accessing a PHP file in the public dir other than index.php
location ~* \.php$ {
    fastcgi_pass unix:/run/php/php7.0-fpm.sock;
    fastcgi_index index.php;
    include /etc/nginx/fastcgi_params;
    fastcgi_param SCRIPT_FILENAME $document_root/$fastcgi_script_name;
}

}

As others have done I did a watch on the "ls" and verified the files get created and removed after the 'certbot renew --dry-run' fails.

And here's my failing dry-run:

sudo certbot renew --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/www.mydomain.com.conf

Cert is due for renewal, auto-renewing...
Starting new HTTPS connection (1): acme-staging.api.letsencrypt.org
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for mydomain.com
http-01 challenge for www.mydomain.com
Waiting for verification...
Cleaning up challenges
Attempting to renew cert from /etc/letsencrypt/renewal/www.mydomain.com.conf produced an unexpected error: Failed authorization procedure. www.mydomain.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.mydomain.com/.well-known/acme-challenge/7c6_KkOErGi-Nk4rObk5uri5Nj2NB35D9nVLfsBiU-Q: "

404 Not Found

404 Not Found

". Skipping. ** DRY RUN: simulating 'certbot renew' close to cert expiry ** (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/www.mydomain.com/fullchain.pem (failure)
** DRY RUN: simulating 'certbot renew' close to cert expiry
** (The test certificates above have not been saved.)
1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:

16

@chrisco23, it doesn’t look like your configuration file defines any web root at all (via root or alias directives, I guess), unless you didn’t include that in your configuration excerpt. In that case, the web server would not know where to look for the files that Certbot creates.

17

Sorry, had abbreviated the config and having trouble with the formatting a lost the line that set “# …etc” … I have edited my post to include the rest of the nginx config.

Note that I’ve run this site for years and the lets encrypt for… forget now but maybe a year. I renewed it once before with some difficulty but I used a different method then, involving certbotrenew.sh.

18

Thanks!

If you create a file /var/www/www.mydomain.com/.well-known/acme-challenge/test.txt, can you then see it in a browser at http://www.myomain.com/.well-known/acme-challenge/test.txt?

19

Nope :frowning: Seems Chrome is still wanting to redirect to https, and failing. What’s worse is that Chrome won’t even let you through with the “Advanced” button (as I’m pretty sure it used to) so the site is completely inaccessible right now.

I guess that means there’s really something I’m missing about the nginx config then right?

20

now I’m seeing if I try the same test without the “www” (but with http rather than https) I get 404.