Certbot does not follow 308 redirects

Help
1

Certbot’s auto-renew recently failed and when looking into why, it seems it doesn’t follow 308 redirects (301 redirects work fine though). This seems like a bug, given that 301 works. In the meantime I’ve “fixed” it by setting the http -> https redirect to use 301 instead of 308, but this is suboptimal.

Details below:

My domain is: sharparam.com, www.sharparam.com

I ran this command: sudo certbot renew

It produced this output (--dry-run):

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/sharparam.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator webroot, Installer nginx
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for sharparam.com
http-01 challenge for www.sharparam.com
Waiting for verification...
Cleaning up challenges
Attempting to renew cert (sharparam.com) from /etc/letsencrypt/renewal/sharparam.com.conf produced an unexpected error: Failed authorization procedure. www.sharparam.com (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from https://www.sharparam.com/.well-known/acme-challenge/AKeTKrbC6CL3uSKwRKlfvsbn-zRTbuV6EikGmKA9EK0 [2606:4700:30::681c:1959]: 308. Skipping.

My web server is (include version): nginx/1.10.3 (Ubuntu)

The operating system my web server runs on is (include version): Ubuntu 16.04 LTS

My hosting provider, if applicable, is: DigitalOcean

I can login to a root shell on my machine (yes or no, or I don’t know): Yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot): certbot 0.31.0 (latest from the PPA)

2

It’s not certbot problem: it’s LE’s VA not supporting 308 redirect and considered invaild
possibley it goes up to golang bug…
@lestaff

3

Agreed.

Unclear: Investigate HTTP-01 challenges and 308 redirects · Issue #4256 · letsencrypt/boulder · GitHub

3 Likes
4

@Sharparam, could you provide the full Nginx config you were using? And ideally restore your Nginx server to use that same config? That way we can double check whether this is really a problem with Let’s Encrypt following redirects or it’s an Nginx config problem.

I ask because I briefly tried to configure Nginx to serve 308 redirects with

    if ($scheme != "https") {
        return 308 https://$host$request_uri;
    }

But I found that Nginx handles this incorrectly and does not set a Location header.

2 Likes
5

@jsha I’ve changed the nginx config back on HTTP sharparam.com to use 308 again.

This is the full nginx config I’m using for that site:

upstream app_server {
    server unix:/tmp/unicorn.sharparamcom.sock fail_timeout=0;
}

upstream matrix_server {
    server matrix.sharparam.com:443;
}

server {
    listen 80;
    listen [::]:80;
    server_name sharparam.com www.sharparam.com;
    return 308 https://sharparam.com$request_uri;
}

server {
    listen 443 ssl; # managed by Certbot
    listen [::]:443 ssl;

    server_name www.sharparam.com;

    include /etc/nginx/conf-available/hsts_preload.conf;

    ssl_certificate /etc/letsencrypt/live/sharparam.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/sharparam.com/privkey.pem; # managed by Certbot
    #include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    #ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

    return 308 $scheme://sharparam.com$request_uri;
}

server {
    listen 443 ssl; # managed by Certbot
    listen [::]:443 ssl;

    server_name sharparam.com;

    access_log /home/rails/apps/sharparam.com/shared/log/nginx.access.log;
    error_log /home/rails/apps/sharparam.com/shared/log/nginx.error.log;

    include /etc/nginx/conf-available/hsts_preload.conf;

    root /home/rails/apps/sharparam.com/current/public;

    location ^~ /_matrix {
        proxy_pass https://matrix_server;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host matrix.sharparam.com;
        proxy_ssl_server_name on;
    }

    location ^~ /.well-known/matrix {
        add_header Access-Control-Allow-Origin *;
        root /var/www/sharparam.com/public_html;
        default_type application/json;
    }

    location ^~ /assets/ {
      gzip_static on;
      expires max;
      add_header Cache-Control public;
    }

    try_files $uri/index.html $uri @app;

    location @app {
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header Host $http_host;
      proxy_redirect off;
      proxy_pass http://app_server;
    }

    error_page 500 502 503 504 /500.html;
    client_max_body_size 20M;
    keepalive_timeout 10;

    ssl_certificate /etc/letsencrypt/live/sharparam.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/sharparam.com/privkey.pem; # managed by Certbot
    #include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    #ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}

Looking at the responses, I notice that nginx is not adding a Location header on 308 like it does for 301. It’s not a requirement in the RFC for 308 but is still recommended. It does enclose the preferred URL in the response body as specified in the RFC though. This behaviour is different from 301 which could be what is causing issues?

Edit: Though on closer inspection, the RFC doesn’t say anything about “SHOULD” like this page (which I usually use for HTTP code references) does while referencing the RFC: https://httpstatuses.com/308

Edit yet again: The “SHOULD” is mentioned here: https://tools.ietf.org/html/rfc7538#section-3

Edit (third time’s the charm): nginx 1.10 (used on sharparam.com) returns the URL in the response body. nginx 1.14 (used on another of my servers) returns it in the Location header. I guess to be completely compatible with how the RFC is written a tool would need to check both places, prioritizing the header?

2 Likes
6

Sounds like you've found a good solution: Upgrade to Nginx 1.14.

Here's what RFC 7538 says:

Note that it doesn't say anything about User-Agents using the response body as a value for automatic redirection if the response body happens to parse as a URL. I'm pretty sure that would be incorrect behavior.

Testing with curl and Chrome suggests they agree with Go's net/http library: If there's no Location header they don't perform a redirect.

BTW, thanks for providing your Nginx config and setting the server up for testing again! I'm done testing now if you'd like to revert it.

3 Likes
7

Yeah, the server was stuck on 1.10 since Ubuntu 16.04 doesn’t have later versions in its repo (been holding off on upgrading to 18.04 because I’m lazy). But there’s a PPA that has 1.14 so it’s now running 1.14 like the 18.04 server.

It’s now returning a Location header so should work the next time the renewal runs!

2 Likes
8

FYI, this was fixed in Nginx 1.13.0.

https://nginx.org/en/docs/http/ngx_http_rewrite_module.html#return

The code 308 was not treated as a redirect until version 1.13.0.

3 Likes
9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.