DST Root CA X3 based cert no longer accepted by Java 8u144

Help
1

My domain is: slaytherelics.xyz

I have a server running on this domain and all of a sudden Java 8u144 (and also 8u241) stopped trusting the certificate.
sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

I also have a website running on exactly the same certificate chain (different port) and browsers see it as a valid certificate.

I run express server that's initialized as follows:
https.createServer({ key: fs.readFileSync(privateKeyPath), cert: fs.readFileSync(publicKeyPath), ca: [fs.readFileSync('ssl/ca/dst_root_ca_x3.pem'), fs.readFileSync('ssl/ca/lets_encrypt_authority_x3.pem')] }, app).listen(port)

Also I ran my domain through analyzer at ssllabs, showing no issues.

image
image1058×545 25.2 KB

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): 92.0.10

Please, any help is most welcome because honestly I have no idea how to fix this, since the certificate appears to be valid everywhere else.

1 Like
2

The DST Root X3 was added in 8u101 and hasn't been removed since.

Did you by any chance build some kind of custom chain checking and hardcoded the previous Let's Encrypt intermediate? Because the only chance currently visible is the X3 to R3 intermediate switchover.

Looking further at your code (didn't read it at first, sorry about that), it indeed looks you're hardcoding the previous intermediate:

You should never ever hardcode these intermediates, as it can change without any notice!

Edit: Wait a minute, that's the server code.. Which is weird, as your server is serving the correct R3 intermediate, while your code says the incorrect X3.. I'm very much confused.

4 Likes
3

The ca option to https.createServer is for mTLS. It's not for configuring your certificate chain and is unnecessary if you just want to secure your Node.js server with TLS.

To create an HTTPS webserver using a Let's Encrypt certificate, it's just:

https.createServer({
  cert: fs.readFileSync('/etc/letsencrypt/live/example.org/fullchain.pem'),
  key: fs.readFileSync('/etc/letsencrypt/live/example.org/privkey.pem')
}, app).listen(port);

A Node.js server which is configured, as above, will be trusted by Java 8u144.

4 Likes
4

@Osiris yes, when researching this error I have also noticed the R3/X3 discrepancy, and was equally confused.

@_az okay, this has to be it. When setting this up a year ago, I was quite clueless about how to do it properly, so that's why it's not correct. Indeed I have a 'fullchain.pem' file available that changed recently - indicating change in issuer certificates (I guess?). I did a bunch of experimenting and found out: for browsers, just the certificate public key (without issuer public key) seems to be enough, that's why my website on different port works no problem (because I'm not actually even loading the overriding certs properly, so it ignores them).

But as for the server used by Java, I still can't figure it out. When I peek into fullchain.pem, I see 2 certificates, none of which is my public key (perhaps 1st is issuer, 2nd is root?). Out of despair I deleted my old certificate and installed a new one (I do it in cPanel interface, no CLI options available afaik), this generated privkey.key and pubkey.crt, both of which contain just 1 certificate upon inspection, fullchain.pem didn't change. When I run the server providing just the pubkey and privkey (which works for the website on different port), Java still gives the same error (I assume because it needs Let's Encrypt pubkey as well in the chain). What would be the correct way to properly assemble the public key chain? In https.createServer it seems you can provide multiple entries in cert, so maybe load multiple keys here? Or perhaps construct the correct fullchain.pem myself?

1 Like
5

Okay, I solved the issue.

I was indeed originally using the retired X3 intermediate certificate and that's why Java failed. The Let's Encrypt R3 intermediate is apparently in cert store for browsers (that's why I reported website working), while it is not in the default cert store for Java. So I downloaded the correct R3 certificate, and provided it in ca and voila, server is again working correctly.

So this is the current working setup:

https.createServer({
  cert: fs.readFileSync('/ssl/pubkey.crt'),
  key: fs.readFileSync('/ssl/privkey.key'),
  ca: fs.readFileSync('/ssl/lets-encrypt-r3-cross-signed.pem')
}, app).listen(port);

I totally understand that downloading the intermediate certificate by hand is highly suboptimal, unfortunately, I don't see another easy way with my hosting provider (i tried using their fullchain.pem, but it didn't work). Thank you very much for all the help! And if there is a more automated way for me to handle this in the future, please share any suggestions! (Since I don't have access to certbot, every few months I have to renew the cert and change the paths in my env variables for the server... Better way would be awesome too)

6

That's not how it works. The roots should be in the clients cert store and the intermediates are provided by the webserver. Webservers may cache intermediates from other sites, so they can be used to build a chain up to a root certificate, but that may fail suddenly if there is no cached intermediate available.

Also, by hardcoding the R3 certificate, you're again hardcoding an intermediate, which should not be done, as the R3 certificate might change in the future without any notice (for example, when there is an incident and R3 is revoked and R4 is being used to sign certificates).

4 Likes
7

Okay, I understand, so the fact that my certificate worked with a website even without providing intermediate, is more of a coincidence and the result of the R3 intermediate being used on other sites too, so that it might have been cached. Okay, I will properly include it in the second server application as well.

I completely understand that it should not be done but do you see any other options in my situation? As I said, my hosting provider neither gives me access to the CLI interface nor do they provide the intermediate certificate themselves.

1 Like
8

If your Java client really is connecting to https://slaytherelics.xyz, then there is no issue server wise, as slaytherelics.xyz is providing the correct intermediate certificate without any issue.

The issue is with your Java client.

4 Likes
9

I don't understand, are you now talking about the current state? Because yes, it is working now, by hardcoding the R3 intermediate on the server.

Are you suggesting I should be somehow including the R3 intermediate in the client application instead of the server?

1 Like
10

The current and previous state of slaytherelics.xyz yes. If there were a missing intermediate when you opened this thread (when I checked too), the SSLLabs would have noticed it and would have flagged it as incorrect.

Also, the hardcoding done was at your Java client. I can't test your client obviously. Only your server and you didn't modify your server. It already served the correct intermediate when your Java client failed and still does at the moment.

I'm wondering/doubting if your Java client actually is connecting to https://slaytherelics.xyz and if that's where you made the change what actually made your Java client work. As your own initial SSLLabs test shows: there is no incorrect intermediate served @ slaytherelics.xyz, not now, not then.

Also, including an intermediate in the client application is just as wrong as hardcoding it at the server.

4 Likes
11

Ooops, I now totally realized where the misunderstanding stems from...

The report from SSL labs is completely misleading and I didn't realize it. Neither of my server applications run on the default HTTP or HTTPS ports. And my hosting provider provides a static html hosting service by default - using their certificate. So the report is almost certainly referencing their certificate and not the Let's encrypt one... :man_facepalming::man_facepalming::man_facepalming: I aplogoize for the confusion.

That's why you didn't trust me, that the changes on the server solved the issue - rightfully so - but in the light of the new information, let me assure you that they in fact did. I'm staring at the server log right now, and as soon as I provided the correct R3 intermediate in the ca parameter on the server and fired it up, I imediatelly started receiving requests from the Java client application of my users, which were not coming in previously.

So what would you suggest as the optimal way to get the intermediate certificate on the server in my situation?

2 Likes
12

Ah, that explains a lot.

The Let's Encrypt servers actually provide the intermediate certificate used for signing the certificate when a cert is issued. Normally, the ACME client used to get the certificate, would provide this intermediate too.

4 Likes
13

Ah, that explains a lot .

Yeah :sweat_smile:

The Let's Encrypt servers actually provide the intermediate certificate used for signing the certificate when a cert is issued. Normally, the ACME client used to get the certificate, would provide this intermediate too.

Yeah, that's sort of what I figured. I'll try contacting my hosting support, if they can help me get the intermediate somewhere automatically, or perhaps get fullchain.pem with intermediate included.

Thank you very much for your patience and for not letting me come to wrong conclusions! I appreciate it, I feel now that I have definitely a better grasp of what's going on.

3 Likes
14

Having the actual full chain delivered with each of your new certificates will make most sense and you should pursue this.

However, the leaf certificate you do get contains a structure called Authority Information Access which you could use to find a certificate for the CA which signed it, in the current case that's R3 but of course it would be different in any future certificate you got signed by a different intermediate.

Decoding your X.509 certificate, pulling the AIA out of it, then fetching the URL shown, and converting the results (which will be binary Distinguished Encoding of the certificate not the PEM files you're likely used to) is a bunch of work, but it is certainly possible, if the sensible "Just get the fullchain file" option doesn't work out.

To return to an earlier topic. Web browsers all have relatively sophisticated but varying ways to try to figure out any reason they could trust your certificate. For example, Microsoft's Internet Explorer automatically does those steps I talked about for AIA above, internally to figure out if your certificate was signed by somebody who in turn was signed by somebody it trusts. This is bad for user privacy but it's fairly effective. On the other hand Mozilla's Firefox carries a complete list (painstakingly assembled) of intermediate CAs known at time of shipping, which makes the browser software bigger and is extra effort for Mozilla's team. You should not rely on this, most other software is much less sophisticated and is expecting your server to do more by e.g. sending over some intermediate CA certificates.

As an earlier poster suggested, the 'ca' option passed to your Express setup is incorrect. This option controls which CAs you want Express to trust, and the defaults are going to be fine for you. In fact chances are you don't care what Express trusts, as this only makes sense for Mutual authenticated TLS where both servers and clients have certificates to identify them. You ought to be passing your entire chain to the 'cert' option as one string. If you have two strings (representing the leaf certificate and then one or more intermediates) you should be able to concatenate them to produce a valid chain.

4 Likes
15

Thank you for your insights!

Decoding your X.509 certificate, pulling the AIA out of it, then fetching the URL shown, and converting the results (which will be binary Distinguished Encoding of the certificate not the PEM files you're likely used to) is a bunch of work, but it is certainly possible, if the sensible "Just get the fullchain file" option doesn't work out.

This sounds like a lot of fun :smiley: unfortunately I'll not be able to justify doing this, since 1. it's probably best to first ask support if they can change the system or tell me how to get correct fullchain.pem from them and 2. now that I understand the underlying issues and the solutions, it won't be hard for me to solve a similar issue in the future. This server is not business critical, it supports my hobby project - a twitch extension, so occasional downtime is not a deal-breaker. Still, I'm glad to learn more about the internal workings

As an earlier poster suggested, the 'ca' option passed to your Express setup is incorrect. This option controls which CAs you want Express to trust, and the defaults are going to be fine for you. In fact chances are you don't care what Express trusts, as this only makes sense for Mutual authenticated TLS where both servers and clients have certificates to identify them. You ought to be passing your entire chain to the 'cert' option as one string. If you have two strings (representing the leaf certificate and then one or more intermediates) you should be able to concatenate them to produce a valid chain.

So essentially you're saying that the fact that the setup works with the intermediate in ca parameter is probably a side effect of Express's or node's internal workings - e.g. if it knows the intermediate certificate which was used to sign the leaf cert in certs but which is not provided there, it fills in the chain automatically, i.e. behaving as if I provided the proper certificate chain myself. Or something along those lines.

In any case, what you suggested works!

https.createServer({
  key: fs.readFileSync(privateKeyPath),
  cert: fs.readFileSync(publicKeyPath) + '\n' + fs.readFileSync(intermediateKeyPath),
}, app).listen(port)

I remember a year ago, when I was setting the cert up for the first time, it felt very daunting and disorienting since it was hard for me to find relevant resources (if my hosting provided the fullchain.pem right away, probably none of this would have been an issue - also most likely the reason why I struggled in the first place to make it work). Still, I'm curious to learn more about this topic, would you recommend some resources to me please? A book/youtube series/online wiki, where this topic is covered at an introductory/intermediate level?

2 Likes
16

I don't have a great recommendation for you concerning introductory material about either X.509 and PKIX (the standards for these certificates and how they can be used on the Internet, respectively) or about the Web PKI (the Public Key Infrastructure actually used to secure services running TLS on the public Internet today). Perhaps somebody else has some suggestions.

I'm afraid I acquired the knowledge I have about this stuff in a very piecemeal fashion, and there's a lot of misinformation out there which is unfortunate. There are some really funny talks I've seen over the years but I hesitate to recommend them to a newcomer because I think it's a bit like showing someone learning English as a Second Language the famous "Four Candles" sketch. The Two Ronnies - Four Candles 480p - YouTube

2 Likes
17

I understand, but please don't be afraid to recommend, I'm not as big of a beginner as it might seem. It's just this topic that is new to me. Otherwise I'm familiar with a lot of other sides of information technologies (including encryption), mainly as a programmer but now I'm learning other aspects too (cloud infrastructure, full stack dev, ...). It's my job too, so I'm definitely not afraid of deep dives or researching some topics that I hear for the first time about. :slight_smile: So, please, I'm quite interested in the funny talks you mentioned! :smiley:

1 Like
18

Thanks a lot. I just spent an hour watching a lot of those old skits. :laughing:

1 Like
19

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.