SSL error after cert renew

Help
1

Hello guys!

Today I had to renew the certificates of a numerous domains on my server (one IP, multiple domains). All domains with renewed certificates are no longer accessible via HTTPS with Chrome, Firefox, Wget. Work fine with Safari (web and mobile), Curl, however.

The error messages from the different browsers are:
Chrome: ERR_SSL_PROTOCOL_ERROR
Firefox: SSL_ERROR_RX_UNEXPECTED_NEW_SESSION_TICKET
Wget: OpenSSL: error:1408E0F4:SSL routines:ssl3_get_message:unexpected message

I do understand the issue is with establishing a TLS connection, but I cannot figure out what it is.
openssl s_client -connect adm.clearware.eu:443
establishes a connection and reports correct values.

I proved, that HTTPS is served on port 443.

===
My domain is:
adm.clearware.eu

I ran this command:
certbot-auto renew

It produced this output:
All certificates that had to be renewed were renewed successfully (I lost the exact output, as I’m trying to resolve the issue for many hours already)

My web server is (include version):
Server version: Apache/2.4.25 (Debian)
Server built: 2017-01-25T22:59:26

The operating system my web server runs on is (include version):
PRETTY_NAME=“Debian GNU/Linux 9 (stretch)”
NAME=“Debian GNU/Linux”
VERSION_ID=“9”
VERSION=“9 (stretch)”

My hosting provider, if applicable, is:
Hetzner root server

I can login to a root shell on my machine (yes or no, or I don’t know):
Yes. I do have root access

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):
No control panel

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot):
certbot 0.37.0

Thanks for your help!

2

Hi @tedyme

another user had the same problem.

The solution:

If in /etc/letsencrypt/options-ssl-apache.conf I set SSLSessionTickets to on
everything seems to work properly.

4 Likes
3

I can confirm this - the problem is solved! You made my day, thanks @JuergenAuer!

4 Likes
4

Bigger question is why did this occur. I had about 30 sites go down today because of this and even completely removing every trace of letsencrypt and reinstalling/setting up this occurred.

Thanks for the fix btw.

1 Like
5

Had the same thing as mtruitt happen today as well.

What’s the best course to prevent this from happening on our other servers?

Thanks for the fix

1 Like
6

I think this was a change in Certbot 0.37.0:

This is pure speculation, but if I remember correctly, things can go horribly wrong in Nginx if you disable session caching in one virtual host while leaving it enabled in another virtual host. I wonder if that applies to Apache and session tickets.

Speculating:

  • Hold off on upgrading to Certbot 0.37.0 for now.

  • Adjust or revert the Certbot Apache configuration file. It's named something like /etc/letsencrypt/options-ssl-apache.conf.

    If I understand it correctly, Certbot will automatically upgrade the file if it hasn't been modified. So you could do something like add a comment saying "# XXX upgrade me later`" or something and Certbot would leave the file alone.

3 Likes
7

Indeed, here's a helpful post on the topic from 2016: Errors from browsers with ssl_session_tickets off (nginx) - #5 by Overlord

3 Likes
8

I haven’t been able to reproduce the problem here, however, we’re currently working on a bug fix release to resolve this issue until we can look into it more.

If anyone can provide more info on how to configure Apache to cause this error, I’d love to have it!

3 Likes
9

FYI, Certbot 0.37.1 has been released, which reverts the change that was causing this: https://lists.eff.org/pipermail/certbot-dev/2019-August/000089.html, https://github.com/certbot/certbot/pull/7316.

If you ran 0.37.0 and started getting this error, I believe you will need to edit your Apache or Nginx configs by hand to re-enable session tickets (or make sure they are disabled consistently across all virtualhosts).

5 Likes
10

Thanks for following up here jsha.

For affected Apache users who haven’t already manually edited their configuration to workaround the problem, after upgrading to Certbot 0.37.1 I recommend running:

sudo certbot plugins --prepare
sudo apachectl graceful

This first command will cause Certbot to automatically revert the changes made to your Apache configuration causing the problem here and the second command has Apache reload its configuration with no downtime.

For Nginx users, our 0.37.1 release didn’t change anything in our Nginx plugin which disabled TLS session tickets in our release a month ago. As far as we know, no nginx users were negatively affected by that change.

6 Likes
11

Hey @tedyme! Erica from Certbot here. We’re trying to figure out what caused this issue so we can get the feature back in without the errors. It would really help us out if you could email me at erica@eff.org with the following information:

  • the results of the command grep OpenSSL /var/log/apache2/error.log (or elsewhere, if you’ve moved your apache error log location)
  • the contents of your config directory (probably located at /etc/apache2/), redacted as you see fit
  • any changes you’ve made to the config directory in the time since encountering this issue

Thanks!

1 Like
12

@mtruitt, @MadMottScott, also wanted to make the offer to you – would you be up for helping out our investigation into this issue? It would really help us out if you could email me at erica@eff.org with the following information:

  • the results of the command grep OpenSSL /var/log/apache2/error.log (or elsewhere, if you’ve moved your apache error log location)
  • the contents of your config directory (probably located at /etc/apache2/ ), redacted as you see fit
  • any changes you’ve made to the config directory in the time since encountering this issue
13

Pinging again @tedyme @mtruitt @MadMottScott – we still haven’t been able to reproduce the issue, and even the Apache developers we’ve talked to don’t understand what’s going on here. Regardless of us making this change again, we’d really like to get to the bottom of this so we can avoid problems like this in the future and can make sure the problem is fixed in whatever software causing a TLS error in this configuration.

If you don’t feel comfortable sending the info I previously mentioned, some other info that might help us is:

  • All mod_ssl directives in the Apache config
  • A high level description of your setup (e.g. are you using PHP/phpmyadmin?)
  • Apache modules loaded

Thanks again!

14

Thanks everybody for your help here! We’ve figured this one out, and just put out a release (0.37.2) fixing it in Nginx as well. If you’re interested in the story, you can follow our discussion on GitHub at https://github.com/certbot/certbot/issues/7322.

15

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.