How to change cert to enable TLSv1.3

My domain is: flag.daemon.contact

I ran this command: firefox

It produced this output:
JavaScript error: chrome://browser/content/browser-siteIdentity.js, line 643: TypeError: issuerCert is undefined

My web server is (include version): apache 2.4.48

The operating system my web server runs on is (include version): FreeBSD 12.2-RELEASE-p10

My hosting provider, if applicable, is: n/a

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 1.16.0

.-.-.-.-.-.-.-.-.-.-.-.-.-.

Question: what do I need to change in the let's encrypt configuration in order to obtain certifcates that are compatible with TLSv1.3?

Currently my certificates work fine with TLSv1.2. But when I enable TLSv1.3 in the webserver, then Firefox will by default connect with TLSv1.3.
Consequentially Firefox (Rel. 73 to 90 on Linux and Freebsd) will delete all certificates upon subsequent requests, display an empty certificate chain with zero certificates, nevertheless enable and sustain the secure connection, and report that connection being perfectly secure after having verified the entire certificate chain of zero certificates.

I do not think this is acceptable. To the contrary, I think it is probably dangerous.

How can I obtain certificates that are compatible with TLSv1.3?

1 Like

X.509 certificates issued by Let's Encrypt have no relationship with the TLS version used, and can be used with any protocol version (even SSLv2 if you want your keys leaked).

I'm not sure how a browser can delete certificates installed on a server. Could you explain your setup more? Are you using Let's Encrypt certificates for client authentication?

I can not reproduce this.

4 Likes

X.509 certificates issued by Let's Encrypt have no relationship with the TLS version used, and can be used with any protocol version (even SSLv2 if you want your keys leaked).

This is what I have learned, also. But I am currently shown that there is a serious difference in how certificates are handled with TLSv1.2 versus TLSv1.3, at least with Firefox (which holds a big market share of browsers). And I am fighting for a week now to isolate this issue and only today figured that it (most likely) comes from TLSv1.3

I'm not sure how a browser can delete certificates installed on a server. Could you explain your setup more? Are you using Let's Encrypt certificates for client authentication?

No, none of this. Let me explain:
The browser holds a copy of the certificate chain as configured on the web server. It does this for every open tab and for every server it talks to. If you have a firefox running, you can observe this in such way:
click on the lock icon in the address bar. A dropdown should open. Click on connection details. Then click on "more information". A window should open with the encryption details, There, click on "View certificate". Then you should see the certificate chain just as it is configured on the respective server.

This is the information the browser needs to verify the authenticity of the server.
And this is the data I am talking about - with certain subsequent requests in a TLS session the browser does delete this information; the certificate chain will then be shown containing zero certificates, the lock icon does not even open a dropdown, and I do not want to know how such connection would then be verified.

And the only lead on the issue that I currently have, is what the cited error message points to, that there is some problem with the issuer of the root certificate.

I can not reproduce this.

I can send you screenshots.

1 Like

It looks to me like your site is running on TLSv1.3 fine, but isn't willing to speak TLSv1.2. I don't know if that's related to whatever you're seeing in your copy of Firefox, but it is rather unusual for a publicly-facing web site.

2 Likes

This is actually connection-based and since Firefox does connection reusing quite heavily this often means the same connection is used for different servers, if their IP's and certificates are valid for both origins.

It's possible that this is a bug in the new certificate viewer (from Firefox). It was already pretty broken in past versions, but I haven't seen much issues lately. What version of Firefox are you using?

Firefox uses NSS and I'm not aware of any difference.

Can you do an external test, e.g enable TLSv1.3 on your service and then connect with a bunch of tools, like SSLabs, openssl (command line, s_client), curl, wget or whatever and see what they show you? This might give a clearer view of the situation.

Edit: I just did a quick openssl s_client -connect daemon.contact:443 and TLSv1.3 negotiates just fine, with all certificates present.

3 Likes

It looks to me like your site is running on TLSv1.3 fine, but isn't willing to speak TLSv1.2. I don't know if that's related to whatever you're seeing in your copy of Firefox, but it is rather unusual for a publicly-facing web site.

Then You were late. 20 minutes ago it would only accept TLSv1.2 laugh

I am working on it, trying to isolate the issue. I should have cut off public access, but then, why bother? After a week of searching this is a chaosheap now, and will be restored from backup when this finally comes to a solution, anyway.

2 Likes

This is actually connection-based and since Firefox does connection reusing quite heavily this often means the same connection is used for different servers, if their IP's and certificates are valid for both origins.

Well, yes, it SHOULD be that way. But, is it, striktly? I am currently still far away from understanding all that logic (one week seems not to suffice for that), but i think somewhere in that area is something that is not fully consistent.

It's possible that this is a bug in the new certificate viewer (from Firefox). It was already pretty broken in past versions, but I haven't seen much issues lately. What version of Firefox are you using?

Believe me, it is not. I tracked that javascript variable back from the page-info-viewer to the main browser. It comes already from there with a zero-length array. But there the trail leads into cpp code - and that gets a bit more difficult to trace.

Version is 78.12.0esr (my usual), 90.0.1 for crosscheck, and some 72 or 73.x.x that came with a peppermint linux I downloaded for another crosscheck. All show the same behaviour.

Edit: I just did a quick openssl s_client -connect daemon.contact:443 and TLSv1.3 negotiates just fine, with all certificates present.

Thank You, good to know. Nevertheless, the error continues to appear in firefox. And in a week I didn't find a way to properly reproduce it. I can make two fresh profiles, one shows the flaw 90% of the time, the other only 10%. Here is a bit more of the story:
https://support.mozilla.org/questions/1350861?utm_campaign=questions-reply&utm_source=notification&utm_medium=email#answer-1444949

1 Like

Which FQDN is having the problem?

Maybe you could also show the vhost config(s) for those that are having this trouble.

1 Like

Which FQDN is having the problem?

All of them. Only some others created with my own CA do not have it.

Maybe you could also show the vhost config(s) for those that are having this trouble.

SSLRandomSeed startup file:/dev/random  512
SSLRandomSeed connect file:/dev/random  512

Listen 443

SSLOpenSSLConfCmd DHParameters "/usr/local/etc/apache24/dhparams.pem"

SSLCipherSuite ECDHE-RSA-AES256-GCM-SHA384:etc.etc.
SSLProxyCipherSuite ECDHE-RSA-AES256-GCM-SHA384:etc.etc.

SSLHonorCipherOrder on 

#SSLProtocol all -SSLv2 -SSLv3 -TLSv1
SSLProtocol TLSv1.3
SSLProxyProtocol all -SSLv2 -SSLv3 -TLSv1

SSLPassPhraseDialog  builtin

#SSLSessionCache         "dbm:/var/run/ssl_scache"
SSLSessionCache        "shmcb:/var/run/ssl_scache(512000)"
#SSLSessionTickets      off
SSLSessionTickets       on
SSLSessionCacheTimeout  300

#SSLUseStapling On

SSLStaplingCache "shmcb:/var/run/ssl_stapling(32768)"

SSLStaplingResponderTimeout 90

SSLStaplingStandardCacheTimeout 3600

SSLStaplingErrorCacheTimeout 600

SSLOptions +StrictRequire
<VirtualHost oper.dinoex.de:443>
ServerName oper.dinoex.de:443
SSLEngine on
SSLUseStapling On
SSLCertificateFile "/usr/local/etc/apache24/oper.crt"
SSLCertificateKeyFile "/usr/local/etc/apache24/oper.key"
SSLCertificateChainFile "/usr/local/etc/apache24/oper.crt"
</VirtualHost>                                  

<VirtualHost 51.158.21.23:443>
ServerName 51.158.21.23:443
SSLEngine on
SSLUseStapling On
SSLCertificateFile "/usr/local/etc/apache24/daemon.crt"
SSLCertificateKeyFile "/usr/local/etc/apache24/daemon.key"
SSLCertificateChainFile "/usr/local/etc/apache24/daemon.crt"
</VirtualHost>                                  

<VirtualHost daemon.contact:443>
ServerName daemon.contact:443
SSLEngine on
SSLUseStapling On
SSLCertificateFile "/usr/local/etc/apache24/daemon.crt"
SSLCertificateKeyFile "/usr/local/etc/apache24/daemon.key"
SSLCertificateChainFile "/usr/local/etc/apache24/daemon.crt"
</VirtualHost>                                  

<VirtualHost daemon.contact:443>
ServerName flag.daemon.contact:443
SSLEngine on
SSLUseStapling On
SSLCertificateFile "/usr/local/etc/apache24/flag.crt"
SSLCertificateKeyFile "/usr/local/etc/apache24/flag.key"
SSLCertificateChainFile "/usr/local/etc/apache24/flag.crt"
</VirtualHost>
1 Like

Please remove all the :443 from the end of the ServerName lines.

What prompted this change?:

The outcome of which is:

1 Like

Please remove all the :443 from the end of the ServerName lines.

I can try, but - why? See Apache doc:

ServerName Directive

Syntax: ServerName [scheme://]domain-name|ip-address[:port]

What prompted this change?:

The outcome of which is:

Beg You pardon, what do You mean??

The line with the hash mark is my normal configuration. After I commentet this out and put "TLSv1.2" into the other line, I did no longer perceive the misbehaviour. So I figured out that the misbehaviour is related to TLSv1.3.
Since I am not really interested in anything else except TLSv1.2 and TLSv1.3, and since TLSv1.2 seems to work okay, and since we are currently looking for a problem appearing only with TLSv1.3, and since people from here might connect to the system (like @Nummer378 above) and then should see the problematic situation, I changed that to TLSv1.3.

Why do You ask?

1 Like

The ServerName directive is already inside of a vhost config that limits the IP:port.
Redundancy is unnecessary and may cause problems in the future - if ever a change is required, you would have to make that change in more than one place.

I mean that the first line already allowed TLSv1.2 and TLSv1.3 (plus any future ones).
While the second line ONLY allows TLSv1.3.

[adding TLSv1.2 to the second line does make them "equal" - but only for today; when any other TLS version comes out (like TLSv2.0), they will no longer be equal]

1 Like

Just for full completeness here, this is what I verified (needed this for some academic work, so might as well share it here):

  • Running TLSv1.2 in Anon-DH mode
    • This is the official way of sending no certificates/doing no auth in TLS (without resumption or PSK). This mode does not exist in TLSv1.3, any attempt to negotiate it in TLSv1.3 fails (OpenSSL says wrong cipher returned, NSS just a generic SSL_ERROR_NO_CYPHER_OVERLAP).
    • I had trouble getting an OpenSSL 1.1.1 client or server to operate in this mode. Apparently the required cipher suites were disabled in my builds. I managed to get it working using an older 1.0.2 build. From the tests, my default openssl s_client did not allow this mode by default, instead failing with no shared cipher. On old builds I managed to get it working by enabling unauthenticated modes -cipher aNULL (note that the server needed this too, otherwise it was like running a server with zero enabled ciphersuites). Firefox had these modes disabled in any case, failing with SSL_ERROR_NO_CYPHER_OVERLAP.
  • Sending zero-byte certificates (or certificate messages) in TLSv1.3
    • This results in a tls_process_server_certificate:length mismatch for OpenSSL, and more helpful SSL_ERROR_RX_MALFORMED_CERTIFICATE (zero-byte certificate) or SSL_ERROR_BAD_SERVER* (zero-byte certificate message) messages in NSS.
  • It is possible to perform a TLSv1.3 handshake without certificates by using the short handshake, which means either using resumption or PSK.
    • My test 1.3 implementation did not support PSK, so my testing here was limited. However, I did not find an option to configure Firefox to do PSK handshakes, I believe this is unsupported.
    • Resumption worked as expected in all implementations tested by me.

*Love this error message by the way, want to see that in more implementations now.

3 Likes

I found the bug. It's in the certificate.

I've read all the Firefox sources. I go to sleep now.

1 Like

I'm sorry, I swapped two lines yesterday. The error is actually not in the certificate, it is something with the OSCP response.

The errorcode is ERROR_REQUIRED_TLS_FEATURE_MISSING
and the debugmessage
D/pipnss HandshakeCallback: couldn't rebuild verified certificate info

When I remove the verification from the code, the error goes away.

Are you using OCSP stapling or OCSP Must Staple? This error message is expected, if you're sending a certificate with must staple, but are omitting the required OCSP response.

2 Likes

Ah, great, thank You. I still have to learn how this stapling thing does work in detail. (Currently I'm only sure I found the root cause why things to break subsequently.)

It is must staple. > cli.ini:
must-staple = true

Shouldn't I do this?

And, how do I not omit the required OSCP response?
(In Apache24, given that it works in the first request, and then fails in subsequent requests)

3 Likes

Must-staple is an option for certificates. Certificates can be issued either with or without must-staple (subscriber decides).

For certificates with must-staple, OCSP stapling must be operative on the server.

OCSP is a protocol to obtain the current status (revoked/not-revoked) of a certificate. OCSP status updates "responses" are generated by the CA (Let's Encrypt) periodically.

The default workflow of obtaining OCSP responses works like this:

  1. The client establishes a TLS connection to a server
  2. Server sends its certificates
  3. The client contacts the CA and asks for the current OCSP status for the certificate(s) just received.
  4. CA responds with OCSP status
  5. TLS connection continues (some clients don't wait for the OCSP response in this configuration and just continue straight away).

This workflow has a few disadvantages:

  • The client needs to contact the CA. This adds additionally latency to connections and also leaks some information about the client (OCSP runs over plain HTTP, so any listener or the CA can see what server the client is connecting to).
  • If some connection error happens, the client doesn't know what to do. Usually it is just ignored.
  • The CA server gets hit with a lot of OCSP queries, because every single client of any subscriber server fetches the OCSP response.

There is however a much better workflow, called OCSP stapling. The workflow for stapled OCSP is like this:

  1. Prior to the connection, the server obtains one or more OCSP responses for its certificates.
  2. The client connects to the server, indicating that it would like an OCSP response.
  3. The server responds with the certificates and the cached OCSP response(s) for the certificates.
  4. Client validates certificates and OCSP responses together.

This has the obvious advantage, that the CA is never contacted by the clients - only by the server, which usually owns the certificate anyway. This improves privacy and latency of the connection and reduces overall load on infrastructure.

The disadvantage is that the server is now responsible for obtaining and serving OCSP responses. OCSP servers naturally have hiccups and return weird data. Servers need to handle this correctly, and ensure they always have up-to-date OCSP responses.

When a certificate contains the must-staple attribute, conforming clients will reject the handshake, if no OCSP response was send by the server.

This usually happens, either because

  • the server was not (correctly) configured to send up-to-date OCSP responses
  • the server did not have any (valid) OCSP response available at the time the client connected

Some server implementations (for example nginx) do lazy-loading of OCSP responses. They do not obtain OCSP responses at startup, but only after the first client requested one. This doesn't play well together with must-staple, as the OCSP response must always be available, even at the very first connection.

Generally it is very difficult to get OCSP must-staple to work well, as many server implementations aren't that good at handling OCSP. Often it is required to employ external solutions that handle OCSP.

The easiest way to get around this is by simply disabling OCSP must-staple. You can still send stapled OCSP responses even without must-staple, the difference is that this is no longer required and clients will not fail, if OCSP responses are (temporarily) omitted. The downside is marginally decreased security, because an active attacker could temporarily inject a compromised certificate and suppress the OCSP response for that certificate. It is a remote attack scenario though.

PS: Sorry for the text wall, didn't realize how long this got.

11 Likes

Oh wow - this is now a lot more info than I would need - but then, we have other readers also. So thank You!!

So much as I know, Stapling is an improvement and is preferrable. I remeber this key 1.3.6.1.5.5.7.1.24 in the cert, but now it seems to look like this (probably both is the same):

            TLS Feature: 
                status_request

So this seems to be correct so far, and we should use stapling. It probably slipped my attention that there are two levels of engagement with it, an optional and a mandatory.

In any case, a failure to obtain a required OCSP response is something that could technically happen. The consequence in firefox, to still handle all webpage interactions correctly, but render parts of the UI nonfunctional, is certainly not the correct thing to do. So we already have a firefox bug (all versions 73 to 93.0b8).

Then, in the apache I can see this kind of messages:
AH01975: sending request to OCSP responder (and a lot more to it).
These would need to be analyzed further if something there matches to the misbehaviour.

Then, as most people on the web are driven by time-to-market and just-make-it-work, they go the easy way and run the feature as optional. This might explain the one big question: why does the crap hit only me.

The other big question: is stapling a feature for only TLS 1.3?
The comments in the client code point to rfc 6066, that one points to rfc 5246 for TLSv1.2, and one would probably need to understand the protocols in full to get a clue from all that.

2 Likes

OCSP stapling is not a key value held in the cert nor it is restricted from any version of TLS.