Subset of Windows Users See Expired Let's Encrypt Certificates with `curl`

Hi petercooperjr,

  1. You're right about the alias in PowerShell. In Windows 10, there's indeed a separate curl located at C:\Windows\System32\curl.exe. When using the Windows Command Prompt, this particular curl is executed. However, I feel the issue might not be related to the curl version. The reason being, our game is written in C#, and on the machines of these affected users, our game also fails to access partyanimals.cn due to a certificate error.

  2. We will ask the affected users to test the mentioned URLs. Once we have the results, we will update this thread.

  3. We will be comparing the curl --version outputs between the systems that are affected and those that are not. As for the Windows version, we tried replicating the issue on a local machine with the same Windows 10 version as the users reported, but we couldn't reproduce the problem.

Thank you very much for your insights, and we'll keep you posted.

2 Likes

If that's the problem you're actually trying to solve, you might have better luck capturing more information from C# than from trying to run curl. In particular, if you can capture the certificate chain that the client is seeing, or maybe even a packet capture of the problem, you might be able to see exactly what might be expired.

4 Likes

Hi rg305,

  1. Our game, which is written in C#, encounters the same certificate issue when accessing PartyAnimals.cn on these users' systems. So, I believe it's unlikely to be an issue specific to the curl version.

  2. We had one user attempt to update Windows, but it did not resolve their issue. We plan to suggest more affected users to try updating Windows as a possible solution.

  3. I'm not very familiar with certificate-related technologies, so I'm unsure about the next steps for testing. I'd greatly appreciate any recommendations or guidance you can provide.

  4. When you mention a "proxy", are you referring to a network proxy? We've ruled out this possibility. Some of our users have two computers at home, and under the exact same network conditions, one machine works fine while the other encounters the issue.

Thank you for your insights.

1 Like

Hi schoen,

We have downloaded the certificate from the affected users' computers, both using the browser and the openssl command-line tool. In both cases, the certificates they received are identical and indeed correct.

What's peculiar is that for this small subset of users, while their browsers indicate that the certificate for our website is trustworthy, both the curl command-line and our game (written in C#) throw certificate-related errors when accessing our site.

Thank you for your insights.

1 Like

Hi webprofusion:
We recently launched our game and just discovered this issue. Additionally, when I used Cert Bot to apply for a certificate, the full certificate chain I received is:

-----BEGIN CERTIFICATE-----
MIIFFjCCAv6gAwIBAgIRAJErCErPDBinU/bWLiWnX1owDQYJKoZIhvcNAQELBQAw
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMjAwOTA0MDAwMDAw
WhcNMjUwOTE1MTYwMDAwWjAyMQswCQYDVQQGEwJVUzEWMBQGA1UEChMNTGV0J3Mg
RW5jcnlwdDELMAkGA1UEAxMCUjMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQC7AhUozPaglNMPEuyNVZLD+ILxmaZ6QoinXSaqtSu5xUyxr45r+XXIo9cP
R5QUVTVXjJ6oojkZ9YI8QqlObvU7wy7bjcCwXPNZOOftz2nwWgsbvsCUJCWH+jdx
sxPnHKzhm+/b5DtFUkWWqcFTzjTIUu61ru2P3mBw4qVUq7ZtDpelQDRrK9O8Zutm
NHz6a4uPVymZ+DAXXbpyb/uBxa3Shlg9F8fnCbvxK/eG3MHacV3URuPMrSXBiLxg
Z3Vms/EY96Jc5lP/Ooi2R6X/ExjqmAl3P51T+c8B5fWmcBcUr2Ok/5mzk53cU6cG
/kiFHaFpriV1uxPMUgP17VGhi9sVAgMBAAGjggEIMIIBBDAOBgNVHQ8BAf8EBAMC
AYYwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMBMBIGA1UdEwEB/wQIMAYB
Af8CAQAwHQYDVR0OBBYEFBQusxe3WFbLrlAJQOYfr52LFMLGMB8GA1UdIwQYMBaA
FHm0WeZ7tuXkAXOACIjIGlj26ZtuMDIGCCsGAQUFBwEBBCYwJDAiBggrBgEFBQcw
AoYWaHR0cDovL3gxLmkubGVuY3Iub3JnLzAnBgNVHR8EIDAeMBygGqAYhhZodHRw
Oi8veDEuYy5sZW5jci5vcmcvMCIGA1UdIAQbMBkwCAYGZ4EMAQIBMA0GCysGAQQB
gt8TAQEBMA0GCSqGSIb3DQEBCwUAA4ICAQCFyk5HPqP3hUSFvNVneLKYY611TR6W
PTNlclQtgaDqw+34IL9fzLdwALduO/ZelN7kIJ+m74uyA+eitRY8kc607TkC53wl
ikfmZW4/RvTZ8M6UK+5UzhK8jCdLuMGYL6KvzXGRSgi3yLgjewQtCPkIVz6D2QQz
CkcheAmCJ8MqyJu5zlzyZMjAvnnAT45tRAxekrsu94sQ4egdRCnbWSDtY7kh+BIm
lJNXoB1lBMEKIq4QDUOXoRgffuDghje1WrG9ML+Hbisq/yFOGwXD9RiX8F6sw6W4
avAuvDszue5L3sz85K+EC4Y/wFVDNvZo4TYXao6Z0f+lQKc0t8DQYzk1OXVu8rp2
yJMC6alLbBfODALZvYH7n7do1AZls4I9d1P4jnkDrQoxB3UqQ9hVl3LEKQ73xF1O
yK5GhDDX8oVfGKF5u+decIsH4YaTw7mP3GFxJSqv3+0lUFJoi5Lc5da149p90Ids
hCExroL1+7mryIkXPeFM5TgO9r0rvZaBFOvV2z0gp35Z0+L4WPlbuEjN/lxPFin+
HlUjr8gRsI3qfJOQFy/9rKIJR0Y/8Omwt/8oTWgy1mdeHmmjk7j1nYsvC9JSQ6Zv
MldlTTKB3zhThV1+XWYp6rjd5JW1zbVWEkLNxE7GJThEUG3szgBVGP7pSWTUTsqX
nLRbwHOoq7hHwg==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIFYDCCBEigAwIBAgIQQAF3ITfU6UK47naqPGQKtzANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTIxMDEyMDE5MTQwM1oXDTI0MDkzMDE4MTQwM1ow
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwggIiMA0GCSqGSIb3DQEB
AQUAA4ICDwAwggIKAoICAQCt6CRz9BQ385ueK1coHIe+3LffOJCMbjzmV6B493XC
ov71am72AE8o295ohmxEk7axY/0UEmu/H9LqMZshftEzPLpI9d1537O4/xLxIZpL
wYqGcWlKZmZsj348cL+tKSIG8+TA5oCu4kuPt5l+lAOf00eXfJlII1PoOK5PCm+D
LtFJV4yAdLbaL9A4jXsDcCEbdfIwPPqPrt3aY6vrFk/CjhFLfs8L6P+1dy70sntK
4EwSJQxwjQMpoOFTJOwT2e4ZvxCzSow/iaNhUd6shweU9GNx7C7ib1uYgeGJXDR5
bHbvO5BieebbpJovJsXQEOEO3tkQjhb7t/eo98flAgeYjzYIlefiN5YNNnWe+w5y
sR2bvAP5SQXYgd0FtCrWQemsAXaVCg/Y39W9Eh81LygXbNKYwagJZHduRze6zqxZ
Xmidf3LWicUGQSk+WT7dJvUkyRGnWqNMQB9GoZm1pzpRboY7nn1ypxIFeFntPlF4
FQsDj43QLwWyPntKHEtzBRL8xurgUBN8Q5N0s8p0544fAQjQMNRbcTa0B7rBMDBc
SLeCO5imfWCKoqMpgsy6vYMEG6KDA0Gh1gXxG8K28Kh8hjtGqEgqiNx2mna/H2ql
PRmP6zjzZN7IKw0KKP/32+IVQtQi0Cdd4Xn+GOdwiK1O5tmLOsbdJ1Fu/7xk9TND
TwIDAQABo4IBRjCCAUIwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYw
SwYIKwYBBQUHAQEEPzA9MDsGCCsGAQUFBzAChi9odHRwOi8vYXBwcy5pZGVudHJ1
c3QuY29tL3Jvb3RzL2RzdHJvb3RjYXgzLnA3YzAfBgNVHSMEGDAWgBTEp7Gkeyxx
+tvhS5B1/8QVYIWJEDBUBgNVHSAETTBLMAgGBmeBDAECATA/BgsrBgEEAYLfEwEB
ATAwMC4GCCsGAQUFBwIBFiJodHRwOi8vY3BzLnJvb3QteDEubGV0c2VuY3J5cHQu
b3JnMDwGA1UdHwQ1MDMwMaAvoC2GK2h0dHA6Ly9jcmwuaWRlbnRydXN0LmNvbS9E
U1RST09UQ0FYM0NSTC5jcmwwHQYDVR0OBBYEFHm0WeZ7tuXkAXOACIjIGlj26Ztu
MA0GCSqGSIb3DQEBCwUAA4IBAQAKcwBslm7/DlLQrt2M51oGrS+o44+/yQoDFVDC
5WxCu2+b9LRPwkSICHXM6webFGJueN7sJ7o5XPWioW5WlHAQU7G75K/QosMrAdSW
9MUgNTP52GE24HGNtLi1qoJFlcDyqSMo59ahy2cI2qBDLKobkx/J3vWraV0T9VuG
WCLKTVXkcGdtwlfFRjlBz4pYg1htmf5X6DYO8A4jqv2Il9DjXA6USbW1FzXSLr9O
he8Y4IWS6wY7bCkjCWDcRQJMEhg76fsO3txE+FiYruq9RUWhiF1myv4Q6W+CyBFC
Dfvp7OOGAN6dEOM4+qR9sdjoSYKEBpsr6GtPAQw4dy753ec5
-----END CERTIFICATE-----

However, when I access PartyAnimals.cn using the Edge browser and export the root certificate, I get:

-----BEGIN CERTIFICATE-----
MIIFazCCA1OgAwIBAgIRAIIQz7DSQONZRGPgu2OCiwAwDQYJKoZIhvcNAQELBQAw
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMTUwNjA0MTEwNDM4
WhcNMzUwNjA0MTEwNDM4WjBPMQswCQYDVQQGEwJVUzEpMCcGA1UEChMgSW50ZXJu
ZXQgU2VjdXJpdHkgUmVzZWFyY2ggR3JvdXAxFTATBgNVBAMTDElTUkcgUm9vdCBY
MTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAK3oJHP0FDfzm54rVygc
h77ct984kIxuPOZXoHj3dcKi/vVqbvYATyjb3miGbESTtrFj/RQSa78f0uoxmyF+
0TM8ukj13Xnfs7j/EvEhmkvBioZxaUpmZmyPfjxwv60pIgbz5MDmgK7iS4+3mX6U
A5/TR5d8mUgjU+g4rk8Kb4Mu0UlXjIB0ttov0DiNewNwIRt18jA8+o+u3dpjq+sW
T8KOEUt+zwvo/7V3LvSye0rgTBIlDHCNAymg4VMk7BPZ7hm/ELNKjD+Jo2FR3qyH
B5T0Y3HsLuJvW5iB4YlcNHlsdu87kGJ55tukmi8mxdAQ4Q7e2RCOFvu396j3x+UC
B5iPNgiV5+I3lg02dZ77DnKxHZu8A/lJBdiB3QW0KtZB6awBdpUKD9jf1b0SHzUv
KBds0pjBqAlkd25HN7rOrFleaJ1/ctaJxQZBKT5ZPt0m9STJEadao0xAH0ahmbWn
OlFuhjuefXKnEgV4We0+UXgVCwOPjdAvBbI+e0ocS3MFEvzG6uBQE3xDk3SzynTn
jh8BCNAw1FtxNrQHusEwMFxIt4I7mKZ9YIqioymCzLq9gwQbooMDQaHWBfEbwrbw
qHyGO0aoSCqI3Haadr8faqU9GY/rOPNk3sgrDQoo//fb4hVC1CLQJ13hef4Y53CI
rU7m2Ys6xt0nUW7/vGT1M0NPAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNV
HRMBAf8EBTADAQH/MB0GA1UdDgQWBBR5tFnme7bl5AFzgAiIyBpY9umbbjANBgkq
hkiG9w0BAQsFAAOCAgEAVR9YqbyyqFDQDLHYGmkgJykIrGF1XIpu+ILlaS/V9lZL
ubhzEFnTIZd+50xx+7LSYK05qAvqFyFWhfFQDlnrzuBZ6brJFe+GnY+EgPbk6ZGQ
3BebYhtF8GaV0nxvwuo77x/Py9auJ/GpsMiu/X1+mvoiBOv/2X/qkSsisRcOj/KK
NFtY2PwByVS5uCbMiogziUwthDyC3+6WVwW6LLv3xLfHTjuCvjHIInNzktHCgKQ5
ORAzI4JMPJ+GslWYHb4phowim57iaztXOoJwTdwJx4nLCgdNbOhdjsnvzqvHu7Ur
TkXWStAmzOVyyghqpZXjFaH3pO3JLF+l+/+sKAIuvtd7u+Nxe5AW0wdeRlN8NwdC
jNPElpzVmbUq4JUagEiuTDkHzsxHpFKVK7q4+63SM1N95R1NbdWhscdCb+ZAJzVc
oyi3B43njTOQ5yOf+1CceWxG1bQVs5ZufpsMljq4Ui0/1lvh+wjChP4kqKOJ2qxq
4RgqsahDYVvTH9w7jXbyLeiNdd8XM2w9U/t7y0Ff/9yi0GE44Za4rF2LN9d11TPA
mRGunUHBcnWEvgJBQl9nJEiU0Zsnvgc/ubhPgXRR4Xq37Z0j4r7g1SgEEzwxA57d
emyPxgcYxn/eR44/KJ4EBs+lVDR3veyJm+kXQ99b21/+jh5Xos1AnX5iItreGCc=
-----END CERTIFICATE-----

Is this normal? It seems that the root certificate displayed in the Edge browser is different from the one I obtained from Cert Bot.
Could this be the reason why some users think our certificate has expired?

1 Like

What you see in your browser and what your server actually serves can be different things. On Windows (Edge, IE) the client (browser) will resolve the best path it can find, which is generally always Leaf > R3 > ISRG Root X1 if windows knows that root certificate (which it usually does because by default the OS dynamically keeps the intermediate and root trust store certificates up to date). However on Firefox for instance, it will resolve differently because it doesn't use the windows native certificate path resolution or certificate store.

2 Likes

So the most important question is, why is curl on windows relevant for your users? Normally only their browser behaviour is important.

2 Likes

Just to jump on these, yes: There can be proxies configured in one browser, but not in another client, on the same system. Even if they're on the same network, the computers might be configured to use different proxies. Often these are intentional on the system administrator's part (routing traffic through some sort of monitoring firewall, or through a VPN that gives access to corporate resources), but sometimes the user isn't aware of how their administrator configured things. (And sometimes there's malware that configured proxies in a way that really isn't what the user intended.) There are a lot of possibilities, so it's really hard for us random people on the Internet to tell you what your users are seeing.

4 Likes

If the app doesn't like the cross-signed root cert...
Then,

  • have you tried changing the chain file?
  • have you tried using a cert from another FREE CA?
2 Likes

This entire issue to me sounds like this is yet another issue with Windows not loading ISRG Root X1 (we have seen this numerious times in the past years).

Microsoft Windows lazily loads most root certificates: Except from a certain set of "preloaded" roots, all root certificates are loaded on demand, as computers establish TLS connections. There have been reports that this doesn't always work: Configuration, Firewall, Policy or other issues can prevent Windows from updating its platform trust store.

The platform trust store is used by Microsoft's TLS implementation "schannel". Historically, browsers such as Chrome or Edge also used this platform trust store. However, starting with Chrome ~108 and Edge 112, both browsers no longer use the Windows platform trust store. Instead, they use their own certificate verifiers. Therefore, these browsers can no longer be used to troubleshoot any certificate-related problems. However, Powershell, .NET/C# runtime and various other Windows components still use schannel and hence the platform trust store.

On the affected machines, I would suggest to check the certificate store (via "certmgr.msc") to ensure that ISRG Root X1 is listed under trusted root certificates. I suspect that it is not, which would confirm the issue.

8 Likes

Thanks for the correction @Nummer378

3 Likes

Thank you for your suggestion regarding capturing more information from C#. I appreciate the insight.

The primary reason I've been using curl is to eliminate the possibility of any bugs within our own code interfering with the analysis of this issue.

By choosing a common and standard networking tool like curl, it allows for a more straightforward discussion. If curl doesn't work on a subset of Windows users, at least we can rule out issues inherent to curl's own implementation.

1 Like

I appreciate your point regarding the relevance of browser behavior for most users. The reason I've chosen to use curl as a test is because our game encountered the same certificate issues on these users' computers. I asked these users to try using curl to see if the problem persisted outside of our game environment.

The rationale behind choosing curl is that it's a standard and widely-used networking tool. If curl exhibits the same issues, it provides a solid indication that the problem might not lie within our game's code. Moreover, using a tool like curl allows everyone to discuss the issue on the same baseline, making it easier to discuss this issue.

1 Like

What language, framework, library etc does your game code use to make the https/tls connection? curl is a special case as it does not originate as a windows native tool but it has been made to work for windows, it's behavior may be different to some other windows clients.

Here is an example of the correct root certificate installed in the local machine certificate store:

If this is not present there would be problems validating the certificate path for some windows clients. W are assuming that the users machines have the correct date/time set.

3 Likes

would visiting le site with ie mode solve this? it should still use windows ssl backed right?

4 Likes

Clever! Yes it's worth a try to attempt to force the certs to update in the store: Internet Explorer mode in Microsoft Edge - Microsoft Support - my assumption is that this would already have happened [at some point] if the OS feature was enabled.

3 Likes

We have identified that the player encountering the issue indeed lacks the necessary root certificate in their machine's certificate store. Additionally, their group policy does not prohibit root certificate updates, and they have also upgraded from Windows 10 to Windows 11. Despite these conditions, the required root certificate is still missing.

To address this gap, we are considering guiding them to try accessing the Let's Encrypt official website using IE mode to see if this resolves the issue.

Thank you for your assistance and guidance on this matter.

2 Likes

I'd suggest the following:

Browsing to https://valid-isrgrootx1.letsencrypt.org/ using IE (or IE mode) will prompt Windows to include ISRG Root X1 in its trust store automatically.

Where the trust store is not automatically updating they can manually install ISRG Root X1:

  • Browse to http://x1.i.lencr.org/ in order to download the .der file for ISRG Root X1 (your browser may warn about the file type and you may need to click "Keep" to save the file).
  • Open the file, click "Install Certificate..", Choose default option "automatically select..", Next, Finish
  • Reboot
3 Likes

It is impossible for the server end to fix such client deficiencies.
[you can't include the root cert for them to trust]
The closest you can come is to bend to their will.
By that I mean: Provide a cert that is trusted by as many of those "ancient" clients as possible.
And by combining that with a cert for all the newer clients you might closer approach 100%.
To be clear, what I'm proposing is that you serve more than one cert.
[to save on costs, only use certs from free CAs]

  • get an RSA 2048 cert that chains to a very old root [not sure about which one to use]
  • get an ECDSA 256 cert that chains to a very new root [Let's Encrypt]

Note: If for any reason you can't serve two certs from your current web service [IIS?], it would be very easy to put those requests through a web proxy that can serve two certs.

3 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.