Problems shipping certificates to another webserver

Help
1

I have a website that is only accessible to out company LAN. Certificates are managed via the DNS-01 challenge with certbot. This works fine most of the time, but users are having several situations in which the site can't be reached (using the guest-WiFi instead of the employee-WiFi; or forgetting to enable the VPN when working from home; or even some not-understood behaviour of Google Chrome that can't reach the internal website, even when Safari or curl can).

So I thought to implement a public internet facing counterpart of the internal website, which will display some helpfull text, explaining the user the situation. Both have the domain name aiwa.archipunt.nl, but our internal DNS resolves to a private IP address (10.x.y.z), and the public DNS system resolves to the public IP address.

For the public website to work, I shipped the certificates with scp to the internet webserver (not sure this is the correct way of doing things). I put the certificates in ~/etc_shadow to avoid misunderstandsings. I shipped all files in the archive folder. For the live folder, I recreated the symbolic links by determining the files in archive via readlink.

When enabling the website, everything seemd fine. When at home, I got the internet facing website, and at work I got the internal one. As expected.

However, for some reason, another website on the same VM stopped working properly. The domain for this website is apawa.archipunt.nl. The error message I got from Firefox is:

Secure Connection Failed

An error occurred during a connection to apawa.archipunt.nl. SSL peer rejected a handshake message for unacceptable content.

Error code: SSL_ERROR_ILLEGAL_PARAMETER_ALERT

    The page you are trying to view cannot be shown because the authenticity of the received data could not be verified.
    Please contact the web site owners to inform them of this problem.

And this happened with curl:

$ curl -v https://apawa.archipunt.nl/
* Host apawa.archipunt.nl:443 was resolved.
* IPv6: (none)
* IPv4: 93.184.96.199
*   Trying 93.184.96.199:443...
* Connected to apawa.archipunt.nl (93.184.96.199) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (OUT), TLS handshake, Client hello (1):
* LibreSSL/3.3.6: error:1404B417:SSL routines:ST_CONNECT:sslv3 alert illegal parameter
* Closing connection
curl: (35) LibreSSL/3.3.6: error:1404B417:SSL routines:ST_CONNECT:sslv3 alert illegal parameter

As soon as I disabled the public facing aiwa.archipunt.nl site (and reloaded nginx), the apawa.archipunt.nl-site worked again.

This is the situation now. So when you open aiwa.archipunt.nl, you will actually open the apawa.archipunt.nl website, because that is nginx's first website to match the same ip/port, so that is to be expected.

For both public and private VM:

My web server is (include version): nginx/1.22.1

The operating system my web server runs on is (include version): Debian GNU/Linux 12 (bookworm)

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 2.1.0

Questions:

  • how can enabling one website make the other one faulty?
  • is shipping shipping certificates like this the right way?
  • what is the "correct" way to have certificates for both a private and a public website?
1 Like
2

What you describe is odd.

Were there any TLS/SSL settings in the new "website" that were not enclosed within its server block?

Were there any warnings in the nginx error log? Or with nginx -t

Do both of these server blocks (sites) have the same IP:port definition? Do you actually need to specify the IP rather than just rely on SNI? Sounds like you might I am just wondering.

Is there some default server block that could be activated for the failing domain when you activate the new one? Perhaps that default can't resolve TLS connection properly. This is the idea behind asking about the IP:port listens.

UPDATE: Oh, you could try putting a unique error_log filename in each of these two server blocks. Be sure it is different than the overall error_log. Then see which, if any, show the error about the TLS connection.

Sure, what you did is fine. Although, I personally would just copy the two files pointed to by the ../live/.. symlink. Which are the latest fullchain.pem and privkey.pem itself. I wouldn't bother preserving the symlink to ../archive/.. which then avoids needing to clone ../archive/...

Certbot has a --deploy-hook which you could use for this copy.

I'll state the obligatory caution about ensuring the privkey.pem stays especially secure via permissions or what-not :slight_smile:

3 Likes
3

Two common patterns for integrating this hook:

1- Copy the files from your machine to the server; have nginx restart via cron nightly to pick up the new certificate.

2- Use a framework like https://fabfile.org to copy the file and automatically ssh into the machine to restart nginx

4 Likes
4

If the public site can be reached via HTTP, then it can get it's own cert via HTTP [using certbot].
There is no need to use the exact same cert on both servers.

6 Likes
5

I didn't get errors with nginx -t. And I don't have any IP-addresses in my nginx config; only server names. I removed nginx's default server, and I don't have a default server at the moment.

I do have different error logs for all sites:
In the error log of nginx I found the following error (this is when both internet facing websites were enabled) on apawa.archipunt.nl:

SSL_do_handshake() failed (SSL: error:0A0000BA:SSL routines::bad cipher) while SSL handshaking

So I checked if there were differences between ssl-config, and there were. So I made them both use the same config, and now the error is fixed.

Thank you all for the input, it is greatly appreciated.

But for me to understand the error, I checked the differences between included ssl directives. The config for the apawa.archipunt.nl (failing) website:

resolver 127.0.0.1 8.8.8.8 8.8.4.4 valid=300s;
resolver_timeout 5s;
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
ssl_dhparam /etc/ssl/certs/dhparam_ffdhe2048.pem;
ssl_prefer_server_ciphers off;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_session_cache shared:SSL:10m;
ssl_session_tickets off;
ssl_session_timeout 1d;

And the config for the other website:

resolver 8.8.8.8 8.8.4.4 valid=300s;
resolver_timeout 5s;
ssl_ciphers "ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384";
ssl_dhparam /etc/ssl/certs/dhparam.pem;
ssl_ecdh_curve secp384r1;
ssl_prefer_server_ciphers on;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_session_cache shared:SSL:10m;
ssl_session_tickets off;

I've sorted the config for easy comparing them.

The used SSL config needs review. I will do that at a later stage. I did have two SSL configurations, because the apawa.archipunt.nl-site needed to support older clients as well. This is probably not necessary anymore.

But the questions is, why is enabling one wite causing an error in the other one. Something with ssl_session_cache and different ssl-ciphers?

1 Like
6

Are those SSL settings inside each server block? If they are just in the respective config files but not in the server block they become part of the http scope and could affect others.

If they are inside the server block they shouldn't be affecting other server blocks. Long ago I did have an nginx SSL setting that carried over from one server block to the next server block that was missing the same setting. That was an nginx bug and I don't recall the exact details. Just giving an example of what you might be facing here.

Maybe some other volunteer will see something. I'd only suggest isolating the problem line(s) by changing one line at a time, restarting nginx and check the error. This isn't a Let's Encrypt related problem but unusual things like this often interest various volunteers.

Personally, I'd focus on the ssl_ciphers, ssl_dhparam, and ssl_ecdh_curve

For your later research about updating your ssl settings see: https://ssl-config.mozilla.org/

You might also try ServerFault or even one of the nginx community resources (link here). Although, if you can isolate to specific line(s) you'll likely get better response.

2 Likes
7

Thanks for all the input. I'm not going to put effort into checking what parameter actually caused the problem: I have to do this after hours, and it doesn't have any priority.

Thanks for all the input. I'm surely going to use the Mozilla SSL Config Generator, but am going to do that after the summer, since our company is rebranding, and we;re getting a new domain name.

1 Like