Issues with Electron and expired root

Our Electron (https://www.electronjs.org/) application is failing to connect to websites using LE certificates. This seems like a major problem that may not have been reported yet.

I am not fluent on the technical details here, but it seems that Electron is internally using an OpenSSL fork which may not trust the new root cert.

This is a major business impact for us, so any advice on potential resolution paths would be much appreciated.

1 Like

To the best of my (limited) knowledge, Electron doesn't ship its own trust store, it uses the operating system's built-in trust store. Is your electron app failing on all platforms, or just certain platforms?

Electron doesn't use OpenSSL at all -- because it is Chromium under the hood, it uses BoringSSL, which does good chain-building. So I suspect that the issue is the trust store on the devices your app is running on, rather than the app itself.

Unless your app is based on an incredibly old version of Electron?

2 Likes

Our app is using a modern version of electron, this issue is observed running Electron version 14.0.1 using Chromium 93.0.4577.63. We have users reporting failure across multiple platforms (windows, mac, and linux).

The only relevant logging messages generated by the electron process are these:

[19596:0930/111427.693:ERROR:ssl_client_socket_impl.cc(981)] handshake failed; returned -1, SSL error code 1, net_error -202
[19596:0930/111427.695:ERROR:ssl_client_socket_impl.cc(981)] handshake failed; returned -1, SSL error code 1, net_error -202
[19596:0930/111427.732:ERROR:ssl_client_socket_impl.cc(981)] handshake failed; returned -1, SSL error code 1, net_error -202
[19596:0930/111427.735:ERROR:ssl_client_socket_impl.cc(981)] handshake failed; returned -1, SSL error code 1, net_error -202
[19596:0930/111427.736:ERROR:ssl_client_socket_impl.cc(981)] handshake failed; returned -1, SSL error code 1, net_error -202
[19596:0930/111427.737:ERROR:ssl_client_socket_impl.cc(981)] handshake failed; returned -1, SSL error code 1, net_error -202
[19596:0930/111427.740:ERROR:ssl_client_socket_impl.cc(981)] handshake failed; returned -1, SSL error code 1, net_error -202
[19596:0930/111427.741:ERROR:ssl_client_socket_impl.cc(981)] handshake failed; returned -1, SSL error code 1, net_error -202
[19596:0930/111427.742:ERROR:ssl_client_socket_impl.cc(981)] handshake failed; returned -1, SSL error code 1, net_error -202
[19596:0930/111427.745:ERROR:ssl_client_socket_impl.cc(981)] handshake failed; returned -1, SSL error code 1, net_error -202
[19596:0930/111428.118:ERROR:ssl_client_socket_impl.cc(981)] handshake failed; returned -1, SSL error code 1, net_error -202

Aha, interesting, thanks! Then I'd suggest updating the chain file used by your API server. I'm guessing (could provide more precise help if I knew the domain these electron clients are contacting) that your API server is currently serving

---
Certificate chain
 0 s:/CN=<your domain here>
   i:/C=US/O=Let's Encrypt/CN=R3
 1 s:/C=US/O=Let's Encrypt/CN=R3
   i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
 2 s:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3
---

If you update that to

---
Certificate chain
 0 s:/CN=<your domain here>
   i:/C=US/O=Let's Encrypt/CN=R3
 1 s:/C=US/O=Let's Encrypt/CN=R3
   i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
---

(i.e. just remove the cross-signed ISRG Root X1 from the chain), I suspect that your electron clients will begin to trust your server again.

You can make that change manually by editing the chain file used by your webserver. You can make the change permanently by editing the configuration of your ACME client to request the alternate chain.

1 Like

Thanks for your advice - the application is connecting to https://foundryvtt.com, the certificate running there was provisioned via certbot. I'm currently trying to understand how to regenerate a certificate via certbot using the --preferred-chain fallback method to specifiy the shorter chain for extended compatibility, but if you have a different suggestion that would be grand. I'm not an expert in this field so unfortunately some of your advice is not meaningful to me, although I will keep researching.

The certificate generation command I am currently testing is:

sudo ./certbot certonly --nginx -d foundryvtt.com --dry-run --preferred-chain=R3

This produces a certificate with the chain of ISRG Root X1 -> R3 -> Client Domain, I don't seem to have the right approach for generating the shorter backwards-compatible chain.

I am testing using openssl s_client -connect foundryvtt.com:443 -servername foundryvtt.com (borrowed from parrisrmcmasterca)

Certificate chain
 0 s:/CN=foundryvtt.com
   i:/C=US/O=Let's Encrypt/CN=R3
 1 s:/C=US/O=Let's Encrypt/CN=R3
   i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
 2 s:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3
2 Likes

Aha, foundryvtt! Now this is something I'm excited about getting fixed :smiley: You want to pass --preffered-chain="ISRG Root X1", like

sudo ./certbot certonly --nginx -d foundryvtt.com --dry-run --preferred-chain="ISRG Root X1"

That should result in the chain which is rooted at ISRG Root X1, and therefore doesn't include ISRG Root X1 in the fullchain.pem itself (chains always stop one cert before the root of trust).

1 Like

Thanks for the suggestion. I tried this (not as a dry-run) on a different domain (atropos.foundryvtt.com), regenerating the cert using:

sudo ./certbot certonly --nginx -d atropos.foundryvtt.com --preferred-chain="ISRG Root X1"

I examined the generated certificate using:

openssl s_client -connect atropos.foundryvtt.com:443 -servername atropos.foundryvtt.com

It has produced the following chain (which appears to be the same as before):

Certificate chain
 0 s:/CN=atropos.foundryvtt.com
   i:/C=US/O=Let's Encrypt/CN=R3
 1 s:/C=US/O=Let's Encrypt/CN=R3
   i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
 2 s:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3

Perhaps this is an issue at the certbot level, or I am invoking the --preferred-chain option incorrectly, but it does not produce the shortened legacy chain that you suggested would be expected.

I sincerely appreciate the help, it's always nice to meet someone familiar with foundryvtt in the wild. Wish it was under better circumstances though, this is frustrating.

EDIT: I am using certbot version 1.19.0 if that is useful.

Can you give one example?

What version of certbot are you using? The behavior of --preferred-chain was updated in 1.12.0; versions prior to that can't distinguish between the two chains offered by Let's Encrypt.

I am using certbot version 1.19.0.

(Busted this out to its own thread since we're going back and forth a lot here.)

Hmm. Then I'm not immediately sure why it isn't getting the chain correctly. I'm probably just missing something obvious as well.

I went ahead and deployed a regenerated cert to the foundryvtt.com website. The certificate was generated as:

sudo ./certbot certonly --nginx -d foundryvtt.com --preferred-chain "ISRG Root X1"

The resulting chain shows as:

Certificate chain
 0 s:/CN=foundryvtt.com
   i:/C=US/O=Let's Encrypt/CN=R3
 1 s:/C=US/O=Let's Encrypt/CN=R3
   i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
 2 s:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3

I had hoped that perhaps regnerating the cert in this way might solve the connectivity problems from Electron, but unfortunately they still occur. Any ideas of what I could try next?

  • restart nginx ?
  • is nginx behind any load-balancer or proxy?

I hesitate to suggest this, but you could edit the fullchain.pem file and just remove the bottom-most block in the file. That won't solve the problem long-term, but it might solve the issue right now and give us 60 days to figure out what's actually going wrong here.

Howdy, I work with Andrew and am also helping to diagnose this.

Removing NGINX from the equation, I'm testing with a domain I own via sudo certbot certonly -d tsde.ironmoose.us --preferred-chain="ISRG Root X1"

Analyzing the output of that via $ sudo openssl crl2pkcs7 -nocrl -certfile /etc/letsencrypt/live/tsde.ironmoose.us/fullchain.pem | openssl pkcs7 -print_certs -text -noout, we end up with two parent certs:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            03:81:bc:a8:5f:d7:9e:ca:98:35:66:d6:57:76:13:d2:5b:84
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=Let's Encrypt, CN=R3
        Validity
            Not Before: Sep 30 15:11:13 2021 GMT
            Not After : Dec 29 15:11:12 2021 GMT
        Subject: CN=tsde.ironmoose.us
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            91:2b:08:4a:cf:0c:18:a7:53:f6:d6:2e:25:a7:5f:5a
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=Internet Security Research Group, CN=ISRG Root X1
        Validity
            Not Before: Sep  4 00:00:00 2020 GMT
            Not After : Sep 15 16:00:00 2025 GMT
        Subject: C=US, O=Let's Encrypt, CN=R3

Based on the description of the short chain R3 (signed by ISRG Root X1) -> Subscriber certificate, I'm not sure if I should expect the second parent cert. I think so, but it's been a few years since my last cert deep dive

1 Like

Related issue on the Electron tracker: [Bug]: Let's Encrypt root CA isn't working properly · Issue #31212 · electron/electron · GitHub

That set of two certs (one for tsde.ironmoose.us, signed by R3; and the other for R3, signed by ISRG Root X1) is exactly what we expect to get when using --preferred-chain "ISRG Root X1", and is what I hope is the solution the Foundry's problem here.

So yeah maybe the issue is that nginx isn't reloading the new fullchain.pem put in place by certbot?

Okay, good news. After generating the certificates I had attempted to reload them into nginx using sudo service nginx reload - which normally (AFAIK) will pick up on new certificates without needing to fully restart the nginx server. It appears, however, that it was not picking up the new certificate in this case so I needed to do a full sudo service nginx restart - as a consequence foundryvtt.com is now serving the correct shortened certificate and our Electron client is successfully connecting to the website!

For others who encounter this thread, it looks like the full solution is as follows:

  1. Generate a new certificate using the --preferred-chain custom certbot option, for example:
sudo certbot certonly --nginx -d <domain> --preferred-chain "ISRG Root X1"
  1. Full restart (not reload) of nginx
sudo service nginx restart
1 Like

I'm not that familiar with nginx, but with Apache a "reload" actually means a "gracefull restart". Which entails that every child process gets the signal to shutdown, but is allowed to finish current connections. Maybe nginx works in a sort of similar way and maybe when you did the reload, there were some older processes left. Although one would think those processes shouldn't be getting new connections. Maybe connections were being reused? Who knows..

But in doubt, restart the service indeed :wink: (Although usually a reload should pick up the new cert and chain..)

Awesome, I'm glad we were able to get this sorted!

Thanks for bringing the fact that Electron's chain-building behavior is different from Chromium's chain-building behavior to our attention.

1 Like