Protocol error after renewal - nginx

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: coachmaster,co.uk, star.thecomapnycoachltd.com, thecoachmasternetwork.com

I ran this command: Automatic renewal

It produced this output: website fails to negotiate ssl on all browsers I have tested.

My web server is (include version): compile of nginx 1.21.6 with openSSL 1.1.1.n and other modules.

The operating system my web server runs on is (include version): Ubuntu 20.04

My hosting provider, if applicable, is: Bitfolk

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): 1.28.0

In addition, I am using SNI. The raw ip on port 443 attempts to respond with a self signed certificate.
various test sites report TLSv.13 only although I have enabled TLSv1.2. Perhaps I need to add some ciphers? uptime robot reports the site up. All http sites continue to work normally.

1 Like

Let's have a look at the output of:
nginx -T | grep -Ei 'ssl_ciphers|ssl_protocols'

6 Likes

Tried and got permission probs. Here is the result when run by sudo :wink:

ian@ianhobson:/etc/nginx$ sudo nginx -T | grep -Ei 'ssl_ciphers|ssl_protocols'
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
        ssl_protocols TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers EECDH+AESGCM:EDH+AESGCM;
    # ssl_protocols TLSv1.2;
    # ssl_ciphers "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384";
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384";
ian@ianhobson:/etc/nginx$

Went on one test site that reported only TLSv1.3 in use, but TLSv1.2 was being used so tried to add it back.

1 Like

I don't see much difference between the two:

yields:

ECDHE-ECDSA-AES128-GCM-SHA256
ECDHE-RSA-AES128-GCM-SHA256
ECDHE-ECDSA-AES256-GCM-SHA384
ECDHE-RSA-AES256-GCM-SHA384
+DHE-DSS-AES256-GCM-SHA384
+DHE-RSA-AES256-GCM-SHA384
DHE-DSS-AES128-GCM-SHA256
DHE-RSA-AES128-GCM-SHA256

yields:

ECDHE-ECDSA-AES128-GCM-SHA256
ECDHE-RSA-AES128-GCM-SHA256
ECDHE-ECDSA-AES256-GCM-SHA384
ECDHE-RSA-AES256-GCM-SHA384
+ECDHE-ECDSA-CHACHA20-POLY1305
+ECDHE-RSA-CHACHA20-POLY1305
DHE-RSA-AES128-GCM-SHA256
DHE-RSA-AES256-GCM-SHA384

[the second is slightly better IMO]

6 Likes

The site is also set up to serve the index.php page and not give a 404 for missing files.
I tested with https://check-your-website.server-daten.de/?q=coachmaster.co.uk and the result contains the comment..

Warning: Not existing ACME-file, but Server sends 200, not 404 or redirect. May be a problem creating a Letsencrypt certificate. Checking /.well-known/acme-challenge/random-filename - a http status 404 - Not Found - is expected. If your server sends content and a http status 200, the validation file (87 bytes, token, dot and the hash of the public part of the account key) may be invisible, so Letsencrypt can't validate your domain. If it is an application that sends this content, perhaps create an exception, so /.well-known/acme-challenge sends raw files. Or create a redirect to another domain and / or port 443, but your Letsencrypt client must support such a solution. Certbot: Use webroot as authenticator - User Guide β€” Certbot 1.28.0 documentation Trouble creating a certificate? Use https://community.letsencrypt.org/ to ask.

Is this relevant? All coachmaster.co.uk is a new install (one thing I tried) but the other two are refreshes, I think all have been approved in the past.

1 Like

I don't use that "test site", so I can't say if that result is relevant or not.

I would try a test run [use certbot with --dry-run] to see if that fails or not.

6 Likes

FWIW, for TLSv1.2+1.3 I use:
ssl_ciphers ECDHE+CHACHA20:ECDHE+AESGCM;
which yields:

ECDHE-ECDSA-CHACHA20-POLY1305
ECDHE-RSA-CHACHA20-POLY1305
ECDHE-ECDSA-AES256-GCM-SHA384
ECDHE-RSA-AES256-GCM-SHA384
ECDHE-ECDSA-AES128-GCM-SHA256
ECDHE-RSA-AES128-GCM-SHA256
5 Likes

certbot with --dry-run failed so switched to server 404 for missing files.
Now I get

ian@ianhobson:/etc/nginx/sites-enabled$ sudo certbot certonly -d coachmaster.co.uk --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log

How would you like to authenticate with the ACME CA?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: Nginx Web Server plugin (nginx)
2: Spin up a temporary webserver (standalone)
3: Place files in webroot directory (webroot)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-3] then [enter] (press 'c' to cancel): 3
Simulating a certificate request for coachmaster.co.uk
Input the webroot for coachmaster.co.uk: (Enter 'c' to cancel): /var/www/coachmaster.co.uk/htsecure

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
  Domain: coachmaster.co.uk
  Type:   tls
  Detail: 85.119.83.64: Fetching https://coachmaster.co.uk/.well-known/acme-challenge/to5ceYidPskHqXF4GfOpNzFzV_83DGAG_p2-JTui5jM: remote error: tls: illegal parameter

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
ian@ianhobson:/etc/nginx/sites-enabled$
1 Like

Well that's a first for me!

Can we see the HTTP vhost config file (for -d coachmaster.co.uk)?

6 Likes

Also, there was a cert issued for that FQDN recently:

7 Likes
ian@ianhobson:/etc/nginx/sites-enabled$ cat coachmaster.co.uk
# Statements for coachmaster.co.uk virtual host
#   modified for coachmaster 3 - 19/Sep/2013
#   removed insecure protocols - 14/Apr/2015
#   moved logging 18/Apr/2017
#   switched to letsencrypt cert  30/6/2019
#   reinstalled letsencrypt cert - 2/Feb/22

#  redirect from http at bottom of file
server {
    server_name coachmaster.co.uk  www.coachmaster.co.uk;
#    listen 443 ssl;
#    ssl_certificate /etc/letsencrypt/live/coachmaster.co.uk/fullchain.pem; # managed by Certbot
#    ssl_certificate_key /etc/letsencrypt/live/coachmaster.co.uk/privkey.pem; # managed by Certbot
#    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
#    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

    # ssl_protocols TLSv1.2;
    # ssl_ciphers "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384";
    # ssl_session_timeout 10m;
    #
    # Aditional Security Headers
    # ref: https://developer.mozilla.org/en-US/docs/Security/HTTP_Strict_Transport_Security
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains";

    # ref: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Frame-Options
    add_header X-Frame-Options DENY always;

    # ref: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Content-Type-Options
    add_header X-Content-Type-Options nosniff always;

    # ref: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-XSS-Protection
    add_header X-Xss-Protection "1; mode=block" always;

    # Enable OCSP stapling
    # ref. http://blog.mozilla.org/security/2013/07/29/ocsp-stapling-in-firefox

    ssl_stapling on;
    ssl_stapling_verify on;
    ssl_trusted_certificate /etc/letsencrypt/live/coachmaster.co.uk-0001/fullchain.pem;

    # rate limit 12 fast, 8 at 5/sec, and then overflow
    limit_req zone=ip burst=12 delay=8;

    location ^~ /Avatars {
       limit_req zone=fp burst=70 nodelay;
    }

    root /var/www/coachmaster.co.uk/htsecure;
    access_log /var/log/nginx/coachmaster.co.uk.access.log;
    # error_log  /var/log/nginx/error.log;
    index index.php;
    location = /Coachmaster.html {
        rewrite ^(.*)  http://thecoachmasternetwork.com/software/;
    }
    location = / {
        rewrite ^ /index.php last;
    }
    location /easyrtc {
        proxy_pass http://localhost:5006;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
    location /socket.io {
        proxy_pass http://localhost:5006;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
    # serve php files via fastcgi if the file exists
    location ~ \.php$ {
        try_files $uri =404;
        fastcgi_split_path_info ^(.+\.php)(/.+)$;
        fastcgi_param  SCRIPT_FILENAME    $document_root$fastcgi_script_name;
        fastcgi_param  CENTRAL_ROOT       $document_root;
        fastcgi_param  RESELLER_ROOT      $document_root;
        fastcgi_param  ENVIRONMENT        production;
        fastcgi_param  HTTPS ON;
        include /etc/nginx/fastcgi.conf;
        fastcgi_pass 127.0.0.1:9000;
    }
    # serve static files
    try_files $uri $uri/ =404;
    expires 30m;
    # now to configure the long polling
    location /publish {
        nchan_publisher;
        nchan_channel_id $arg_id;
        nchan_message_buffer_length 10;
        nchan_message_timeout 90s;
    }
    # public long-polling endpoint
    location /activity {
        nchan_subscriber;
        nchan_channel_id $arg_id;
    }


    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/coachmaster.co.uk-0001/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/coachmaster.co.uk-0001/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}

server {
    if ($host = www.coachmaster.co.uk) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


    if ($host = coachmaster.co.uk) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


    server_name coachmaster.co.uk  www.coachmaster.co.uk;

    listen 80;
    return 404; # managed by Certbot

}
ian@ianhobson:/etc/nginx/sites-enabled$

There is also an options file

ian@ianhobson:/etc/letsencrypt$ cat options-ssl-nginx.conf
# This file contains important security parameters. If you modify this file
# manually, Certbot will be unable to automatically provide future security
# updates. Instead, Certbot will print and log an error message with a path to
# the up-to-date file that you will need to refer to when manually updating
# this file. Contents are based on https://ssl-config.mozilla.org

ssl_session_cache shared:le_nginx_SSL:10m;
ssl_session_timeout 1440m;
ssl_session_tickets off;

ssl_protocols TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers off;

ssl_ciphers "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384";
ian@ianhobson:/etc/letsencrypt$

That was probably me attempting to fix the problem, by switching back to http, getting latest certbot and starting afresh. Had fun getting rid of letsencrypt stuff only for coachmaster.co.uk :slight_smile:

I'd update this section to handle the challenge requests:

You can use a unique root / --webroot [only for these requests - not same root as in HTTPS]

Try something like:

  location ^/(?!\.well-known) {            # skip challenge requests
    return 301 https://$host$request_uri;  # send all requests to HTTPS
  }# location
  root /new/dedicated/challenge/path;      # path for challenge requests
7 Likes

That section is now

server {
    listen 80;
    server_name coachmaster.co.uk  www.coachmaster.co.uk;
    location ^/(?!\.well-known) {             # skip challenge requests
       return 301 https://$host$request_uri;  # send all requests to HTTPS
    }
    # writable root for challenges
    root /var/www/coachmaster.co.uk/challenges;       # path for challenge requests
    try_files $uri =404;
}

Hope I understood you correctly.

Unfortunately there is still the protocol error. :frowning:

1 Like

Tried the --dry-run and...

ian@ianhobson:/etc/nginx/sites-enabled$ sudo certbot certonly -d coachmaster.co.uk --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log

How would you like to authenticate with the ACME CA?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: Nginx Web Server plugin (nginx)
2: Spin up a temporary webserver (standalone)
3: Place files in webroot directory (webroot)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-3] then [enter] (press 'c' to cancel): 3
Simulating a certificate request for coachmaster.co.uk
Input the webroot for coachmaster.co.uk: (Enter 'c' to cancel): /var/www/coachmaster.co.uk/challenges
The dry run was successful.
ian@ianhobson:/etc/nginx/sites-enabled$
1 Like

Renewed the certificates for coachmaster.co.uk, and that was successful. Restarted nginx. Attempted to load coachmaster.co.uk into chrome, and got...

This site can’t provide a secure connection
coachmaster.co.uk sent an invalid response.
Try running Windows Network Diagnostics.
ERR_SSL_PROTOCOL_ERROR

Windows Network diagnostics fails to start.

1 Like

Very interesting. I also see a Protocol Error with Chrome and Edge from Android.

But, SSL Labs gives your site an A+

And, from an RHEL system I can curl to your https site just fine (wget too)

But, whynopadlock also reports a protocol failure when it does Mixed Content test. I don't even see any mixed content in your home page so I assume it has a problem with one of your other hrefs (but I don't have such problem with curl)

Odd mix of symptoms. It might be worth restarting your server. It wouldn't be the first time a weird combination of symptoms was resolved that way.

7 Likes

I have shutdown and restarted the server (a VM). Problem not sorted.

As for Mixed Content, the site does request stuff from google (analytics and a font IIRC) but is otherwise complete. It has been working fine until recently. I'm 95% sure coachmaster.co.uk and thecoachmasternetwork.com failed when their certificates were renewed a week or so back. star.thecompanycoachltd.com fails a few days later when it renewed.

However I released a new compilation of nginx about this time, so I can't be 100% sure. Can only test ssl on a server on the internet, of course.

1 Like

I'm not sure how to proceed with that at the moment. I do see your HTTP server is not redirecting to HTTPS. If you still have this:

server {
    listen 80;
    server_name coachmaster.co.uk  www.coachmaster.co.uk;
    location ^/(?!\.well-known) {             # skip challenge requests
       return 301 https://$host$request_uri;  # send all requests to HTTPS
    }
    # writable root for challenges
    root /var/www/coachmaster.co.uk/challenges;       # path for challenge requests
    try_files $uri =404;
}

You need to change it to something like below. Or, put the root and try_files in a location block above

server {
    listen 80;
    server_name coachmaster.co.uk  www.coachmaster.co.uk;
    # writable root for challenges
    location /.well-known/acme-challenge {
        root /var/www/coachmaster.co.uk/challenges;       # path for challenge requests
        try_files $uri =404;   # you dont actually need this, will happen anyway
    }
    # redirect all others
    location / {    
       return 301 https://$host$request_uri;
    }
}
7 Likes

Thanks. I have some ideas.

  1. Buy a cheap domain and create an https website with that, and see if that has the same problem. If so, it means Letsencrypt is the problem, not nginx, its compile or configuration Goto step 3.

  2. Swap my compile of nginx out and the standard one in and see if that will work on the new site.
    If it does, then its my compile that is faulty.

  3. Strip the config of coachmaster.com back to simplest possible, and try that. If it works, add complications back until it breaks. If it can't work, then try step 2.

  4. Recompile nginx with different version of the modules used. Currently it is nginx 1.21.6, openSSL 1.1.1.n with nchan 1.2.12.

Try nginx-1.23.0 , openSSL-1.1.1.p, and nchan-1.3.0 - in various combinations

  1. Try above with openSSL1.3.0 (which won't compile with nchan 1.2.12 on Ubuntu

  2. Could also try restoring an old backup and reverting to that compile of nginx that worked before the latest update.

  3. Update to Ubuntu 22.04. and try the compiles again

  4. Swap Ubuntu for freeBSD where I am assured a combination does work.

  5. If its Letsencrypt, strip out Letsencrypt, buy new certificates from somewhere else. Reconfigure each site.

  6. Install wireshark and see what is going up/down the line. Find it in openSSL's source and correct it. This needs skills I had a weeks course in, 35 years ago and never used! Ohh Joy!

  7. Install a vanilla nginx to handle SSL and proxy to a compile with nchan, and the chat room.

1 Like