Subset of Windows Users See Expired Let's Encrypt Certificates with `curl`

Hello everyone,

We've encountered a peculiar issue. We use a Let's Encrypt certificate for our site: https://partyanimals.cn.

Some Windows(both win11 and win10) users are seeing a "certificate expired" error when using curl -I on our website and on https://letsencrypt.org/. Yet, even for these affected users, the certificate appears normal when accessed via browsers like Edge.

Step to reproduce:
On affected Windows machine, run curl -I https://partyanimals.cn or curl -I https://letsencrypt.org.
The exact error message they receive is:

curl: (35) schannel: next InitializeSecurityContext failed: SEC_E_CERT_EXPIRED(0x80090328)

From our end, our site's certificate appears valid, and of course, we would expect the certificate for letsencrypt.org to be without issues. Yet, this problem doesn't seem to impact all Windows users, only a subset.

Would greatly appreciate any insights or advice on this sporadic behavior with curl and Let's Encrypt certificates.

Thanks in advance.

1 Like

I haven't seen this, but I haven't done much running of curl from Windows.

I have some random thoughts, though:

  1. Running curl within Powershell is actually an alias for Invoke-WebRequest. And there's also a "real" curl available on some systems (maybe all systems with current updates?). And I haven't played with Windows Subsystem for Linux, but it wouldn't shock me if that had its own curl as well. Can you be really specific on which curl people are seeing problems with, and if there might be a difference between users that see it working and users that don't? (It's really easy to run "Terminal" and not realize one is actually running Powershell.)
  2. Can you test the following URLs, to see if there's a difference? They use different certificate chains, and I'm wondering if the expired DST Root CA X3 (which modern systems should just be ignoring) is causing something somewhere to be unhappy.
    • https://valid-isrgrootx1.letsencrypt.org/
    • https://valid-isrgrootx2.letsencrypt.org/
    • https://helloworld.letsencrypt.org/
  3. And of course, if you could narrow down which systems have a problem, that would be helpful. Can you compare curl --version, winver, etc. between systems that work and systems that don't? Can you test it on a fresh clean install (maybe in a new VM)? Are there any "proxy" or "security" firewalls that might be intercepting the connection?
5 Likes

It sounds like the underlying SChannel [or OpenSSL] is outdated.
[which may be specific to Windows curl (and not system-wide)]
Have you tried "Windows Updates"?

Definitely more tests need to be run to better understand things more clearly.

OR

There is a proxy involved that is part of the problem...

3 Likes

Hi @youyounanfeng,

Could you get some of those users to save a copy of the certificate that they see and send it to you? Maybe they're getting network interference that is actually a MITM attack or something.

They could save it from the browser, or run something like

openssl s_client -connect partyanimals.cn:443 -servername partyanimals.cn

Your site has very good results at https://ssllabs.com/ (indicating that most clients should negotiate a connection successfully), so it doesn't seem like this is likely a problem with your own configuration.

2 Likes

It's most likely (?) the expired root DST Root CA X3 being flagged up by a specific version of curl or specific configuration of machine trust store.

If curl is using Schannel (the native windows alternative to openssl) then I would expect it should also be using the windows certificate store for known roots and intermediates and so when it sees the intermediate R3 leads to ISRG Root X1 (self signed) it should resolve that instead of the ISRG Root X1 > DST Root CA X3 path. That relies on the assumption that curl is following the normal windows certificate path validation and that ISRG Root X1 (self signed) is present in the machines Local Machine > Trusted Root Certification Authorities certificate store. It normally will be present unless group policy/registry has been set to disallow automatic root ca cert updates.

[However this problem is not new and you would have been seeing it since Sept 2021]

1 Like

Hi petercooperjr,

  1. You're right about the alias in PowerShell. In Windows 10, there's indeed a separate curl located at C:\Windows\System32\curl.exe. When using the Windows Command Prompt, this particular curl is executed. However, I feel the issue might not be related to the curl version. The reason being, our game is written in C#, and on the machines of these affected users, our game also fails to access partyanimals.cn due to a certificate error.

  2. We will ask the affected users to test the mentioned URLs. Once we have the results, we will update this thread.

  3. We will be comparing the curl --version outputs between the systems that are affected and those that are not. As for the Windows version, we tried replicating the issue on a local machine with the same Windows 10 version as the users reported, but we couldn't reproduce the problem.

Thank you very much for your insights, and we'll keep you posted.

2 Likes

If that's the problem you're actually trying to solve, you might have better luck capturing more information from C# than from trying to run curl. In particular, if you can capture the certificate chain that the client is seeing, or maybe even a packet capture of the problem, you might be able to see exactly what might be expired.

4 Likes

Hi rg305,

  1. Our game, which is written in C#, encounters the same certificate issue when accessing PartyAnimals.cn on these users' systems. So, I believe it's unlikely to be an issue specific to the curl version.

  2. We had one user attempt to update Windows, but it did not resolve their issue. We plan to suggest more affected users to try updating Windows as a possible solution.

  3. I'm not very familiar with certificate-related technologies, so I'm unsure about the next steps for testing. I'd greatly appreciate any recommendations or guidance you can provide.

  4. When you mention a "proxy", are you referring to a network proxy? We've ruled out this possibility. Some of our users have two computers at home, and under the exact same network conditions, one machine works fine while the other encounters the issue.

Thank you for your insights.

1 Like

Hi schoen,

We have downloaded the certificate from the affected users' computers, both using the browser and the openssl command-line tool. In both cases, the certificates they received are identical and indeed correct.

What's peculiar is that for this small subset of users, while their browsers indicate that the certificate for our website is trustworthy, both the curl command-line and our game (written in C#) throw certificate-related errors when accessing our site.

Thank you for your insights.

1 Like

Hi webprofusion:
We recently launched our game and just discovered this issue. Additionally, when I used Cert Bot to apply for a certificate, the full certificate chain I received is:

-----BEGIN CERTIFICATE-----
MIIFFjCCAv6gAwIBAgIRAJErCErPDBinU/bWLiWnX1owDQYJKoZIhvcNAQELBQAw
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMjAwOTA0MDAwMDAw
WhcNMjUwOTE1MTYwMDAwWjAyMQswCQYDVQQGEwJVUzEWMBQGA1UEChMNTGV0J3Mg
RW5jcnlwdDELMAkGA1UEAxMCUjMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQC7AhUozPaglNMPEuyNVZLD+ILxmaZ6QoinXSaqtSu5xUyxr45r+XXIo9cP
R5QUVTVXjJ6oojkZ9YI8QqlObvU7wy7bjcCwXPNZOOftz2nwWgsbvsCUJCWH+jdx
sxPnHKzhm+/b5DtFUkWWqcFTzjTIUu61ru2P3mBw4qVUq7ZtDpelQDRrK9O8Zutm
NHz6a4uPVymZ+DAXXbpyb/uBxa3Shlg9F8fnCbvxK/eG3MHacV3URuPMrSXBiLxg
Z3Vms/EY96Jc5lP/Ooi2R6X/ExjqmAl3P51T+c8B5fWmcBcUr2Ok/5mzk53cU6cG
/kiFHaFpriV1uxPMUgP17VGhi9sVAgMBAAGjggEIMIIBBDAOBgNVHQ8BAf8EBAMC
AYYwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMBMBIGA1UdEwEB/wQIMAYB
Af8CAQAwHQYDVR0OBBYEFBQusxe3WFbLrlAJQOYfr52LFMLGMB8GA1UdIwQYMBaA
FHm0WeZ7tuXkAXOACIjIGlj26ZtuMDIGCCsGAQUFBwEBBCYwJDAiBggrBgEFBQcw
AoYWaHR0cDovL3gxLmkubGVuY3Iub3JnLzAnBgNVHR8EIDAeMBygGqAYhhZodHRw
Oi8veDEuYy5sZW5jci5vcmcvMCIGA1UdIAQbMBkwCAYGZ4EMAQIBMA0GCysGAQQB
gt8TAQEBMA0GCSqGSIb3DQEBCwUAA4ICAQCFyk5HPqP3hUSFvNVneLKYY611TR6W
PTNlclQtgaDqw+34IL9fzLdwALduO/ZelN7kIJ+m74uyA+eitRY8kc607TkC53wl
ikfmZW4/RvTZ8M6UK+5UzhK8jCdLuMGYL6KvzXGRSgi3yLgjewQtCPkIVz6D2QQz
CkcheAmCJ8MqyJu5zlzyZMjAvnnAT45tRAxekrsu94sQ4egdRCnbWSDtY7kh+BIm
lJNXoB1lBMEKIq4QDUOXoRgffuDghje1WrG9ML+Hbisq/yFOGwXD9RiX8F6sw6W4
avAuvDszue5L3sz85K+EC4Y/wFVDNvZo4TYXao6Z0f+lQKc0t8DQYzk1OXVu8rp2
yJMC6alLbBfODALZvYH7n7do1AZls4I9d1P4jnkDrQoxB3UqQ9hVl3LEKQ73xF1O
yK5GhDDX8oVfGKF5u+decIsH4YaTw7mP3GFxJSqv3+0lUFJoi5Lc5da149p90Ids
hCExroL1+7mryIkXPeFM5TgO9r0rvZaBFOvV2z0gp35Z0+L4WPlbuEjN/lxPFin+
HlUjr8gRsI3qfJOQFy/9rKIJR0Y/8Omwt/8oTWgy1mdeHmmjk7j1nYsvC9JSQ6Zv
MldlTTKB3zhThV1+XWYp6rjd5JW1zbVWEkLNxE7GJThEUG3szgBVGP7pSWTUTsqX
nLRbwHOoq7hHwg==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIFYDCCBEigAwIBAgIQQAF3ITfU6UK47naqPGQKtzANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTIxMDEyMDE5MTQwM1oXDTI0MDkzMDE4MTQwM1ow
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwggIiMA0GCSqGSIb3DQEB
AQUAA4ICDwAwggIKAoICAQCt6CRz9BQ385ueK1coHIe+3LffOJCMbjzmV6B493XC
ov71am72AE8o295ohmxEk7axY/0UEmu/H9LqMZshftEzPLpI9d1537O4/xLxIZpL
wYqGcWlKZmZsj348cL+tKSIG8+TA5oCu4kuPt5l+lAOf00eXfJlII1PoOK5PCm+D
LtFJV4yAdLbaL9A4jXsDcCEbdfIwPPqPrt3aY6vrFk/CjhFLfs8L6P+1dy70sntK
4EwSJQxwjQMpoOFTJOwT2e4ZvxCzSow/iaNhUd6shweU9GNx7C7ib1uYgeGJXDR5
bHbvO5BieebbpJovJsXQEOEO3tkQjhb7t/eo98flAgeYjzYIlefiN5YNNnWe+w5y
sR2bvAP5SQXYgd0FtCrWQemsAXaVCg/Y39W9Eh81LygXbNKYwagJZHduRze6zqxZ
Xmidf3LWicUGQSk+WT7dJvUkyRGnWqNMQB9GoZm1pzpRboY7nn1ypxIFeFntPlF4
FQsDj43QLwWyPntKHEtzBRL8xurgUBN8Q5N0s8p0544fAQjQMNRbcTa0B7rBMDBc
SLeCO5imfWCKoqMpgsy6vYMEG6KDA0Gh1gXxG8K28Kh8hjtGqEgqiNx2mna/H2ql
PRmP6zjzZN7IKw0KKP/32+IVQtQi0Cdd4Xn+GOdwiK1O5tmLOsbdJ1Fu/7xk9TND
TwIDAQABo4IBRjCCAUIwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYw
SwYIKwYBBQUHAQEEPzA9MDsGCCsGAQUFBzAChi9odHRwOi8vYXBwcy5pZGVudHJ1
c3QuY29tL3Jvb3RzL2RzdHJvb3RjYXgzLnA3YzAfBgNVHSMEGDAWgBTEp7Gkeyxx
+tvhS5B1/8QVYIWJEDBUBgNVHSAETTBLMAgGBmeBDAECATA/BgsrBgEEAYLfEwEB
ATAwMC4GCCsGAQUFBwIBFiJodHRwOi8vY3BzLnJvb3QteDEubGV0c2VuY3J5cHQu
b3JnMDwGA1UdHwQ1MDMwMaAvoC2GK2h0dHA6Ly9jcmwuaWRlbnRydXN0LmNvbS9E
U1RST09UQ0FYM0NSTC5jcmwwHQYDVR0OBBYEFHm0WeZ7tuXkAXOACIjIGlj26Ztu
MA0GCSqGSIb3DQEBCwUAA4IBAQAKcwBslm7/DlLQrt2M51oGrS+o44+/yQoDFVDC
5WxCu2+b9LRPwkSICHXM6webFGJueN7sJ7o5XPWioW5WlHAQU7G75K/QosMrAdSW
9MUgNTP52GE24HGNtLi1qoJFlcDyqSMo59ahy2cI2qBDLKobkx/J3vWraV0T9VuG
WCLKTVXkcGdtwlfFRjlBz4pYg1htmf5X6DYO8A4jqv2Il9DjXA6USbW1FzXSLr9O
he8Y4IWS6wY7bCkjCWDcRQJMEhg76fsO3txE+FiYruq9RUWhiF1myv4Q6W+CyBFC
Dfvp7OOGAN6dEOM4+qR9sdjoSYKEBpsr6GtPAQw4dy753ec5
-----END CERTIFICATE-----

However, when I access PartyAnimals.cn using the Edge browser and export the root certificate, I get:

-----BEGIN CERTIFICATE-----
MIIFazCCA1OgAwIBAgIRAIIQz7DSQONZRGPgu2OCiwAwDQYJKoZIhvcNAQELBQAw
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMTUwNjA0MTEwNDM4
WhcNMzUwNjA0MTEwNDM4WjBPMQswCQYDVQQGEwJVUzEpMCcGA1UEChMgSW50ZXJu
ZXQgU2VjdXJpdHkgUmVzZWFyY2ggR3JvdXAxFTATBgNVBAMTDElTUkcgUm9vdCBY
MTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAK3oJHP0FDfzm54rVygc
h77ct984kIxuPOZXoHj3dcKi/vVqbvYATyjb3miGbESTtrFj/RQSa78f0uoxmyF+
0TM8ukj13Xnfs7j/EvEhmkvBioZxaUpmZmyPfjxwv60pIgbz5MDmgK7iS4+3mX6U
A5/TR5d8mUgjU+g4rk8Kb4Mu0UlXjIB0ttov0DiNewNwIRt18jA8+o+u3dpjq+sW
T8KOEUt+zwvo/7V3LvSye0rgTBIlDHCNAymg4VMk7BPZ7hm/ELNKjD+Jo2FR3qyH
B5T0Y3HsLuJvW5iB4YlcNHlsdu87kGJ55tukmi8mxdAQ4Q7e2RCOFvu396j3x+UC
B5iPNgiV5+I3lg02dZ77DnKxHZu8A/lJBdiB3QW0KtZB6awBdpUKD9jf1b0SHzUv
KBds0pjBqAlkd25HN7rOrFleaJ1/ctaJxQZBKT5ZPt0m9STJEadao0xAH0ahmbWn
OlFuhjuefXKnEgV4We0+UXgVCwOPjdAvBbI+e0ocS3MFEvzG6uBQE3xDk3SzynTn
jh8BCNAw1FtxNrQHusEwMFxIt4I7mKZ9YIqioymCzLq9gwQbooMDQaHWBfEbwrbw
qHyGO0aoSCqI3Haadr8faqU9GY/rOPNk3sgrDQoo//fb4hVC1CLQJ13hef4Y53CI
rU7m2Ys6xt0nUW7/vGT1M0NPAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNV
HRMBAf8EBTADAQH/MB0GA1UdDgQWBBR5tFnme7bl5AFzgAiIyBpY9umbbjANBgkq
hkiG9w0BAQsFAAOCAgEAVR9YqbyyqFDQDLHYGmkgJykIrGF1XIpu+ILlaS/V9lZL
ubhzEFnTIZd+50xx+7LSYK05qAvqFyFWhfFQDlnrzuBZ6brJFe+GnY+EgPbk6ZGQ
3BebYhtF8GaV0nxvwuo77x/Py9auJ/GpsMiu/X1+mvoiBOv/2X/qkSsisRcOj/KK
NFtY2PwByVS5uCbMiogziUwthDyC3+6WVwW6LLv3xLfHTjuCvjHIInNzktHCgKQ5
ORAzI4JMPJ+GslWYHb4phowim57iaztXOoJwTdwJx4nLCgdNbOhdjsnvzqvHu7Ur
TkXWStAmzOVyyghqpZXjFaH3pO3JLF+l+/+sKAIuvtd7u+Nxe5AW0wdeRlN8NwdC
jNPElpzVmbUq4JUagEiuTDkHzsxHpFKVK7q4+63SM1N95R1NbdWhscdCb+ZAJzVc
oyi3B43njTOQ5yOf+1CceWxG1bQVs5ZufpsMljq4Ui0/1lvh+wjChP4kqKOJ2qxq
4RgqsahDYVvTH9w7jXbyLeiNdd8XM2w9U/t7y0Ff/9yi0GE44Za4rF2LN9d11TPA
mRGunUHBcnWEvgJBQl9nJEiU0Zsnvgc/ubhPgXRR4Xq37Z0j4r7g1SgEEzwxA57d
emyPxgcYxn/eR44/KJ4EBs+lVDR3veyJm+kXQ99b21/+jh5Xos1AnX5iItreGCc=
-----END CERTIFICATE-----

Is this normal? It seems that the root certificate displayed in the Edge browser is different from the one I obtained from Cert Bot.
Could this be the reason why some users think our certificate has expired?

1 Like

What you see in your browser and what your server actually serves can be different things. On Windows (Edge, IE) the client (browser) will resolve the best path it can find, which is generally always Leaf > R3 > ISRG Root X1 if windows knows that root certificate (which it usually does because by default the OS dynamically keeps the intermediate and root trust store certificates up to date). However on Firefox for instance, it will resolve differently because it doesn't use the windows native certificate path resolution or certificate store.

2 Likes

So the most important question is, why is curl on windows relevant for your users? Normally only their browser behaviour is important.

2 Likes

Just to jump on these, yes: There can be proxies configured in one browser, but not in another client, on the same system. Even if they're on the same network, the computers might be configured to use different proxies. Often these are intentional on the system administrator's part (routing traffic through some sort of monitoring firewall, or through a VPN that gives access to corporate resources), but sometimes the user isn't aware of how their administrator configured things. (And sometimes there's malware that configured proxies in a way that really isn't what the user intended.) There are a lot of possibilities, so it's really hard for us random people on the Internet to tell you what your users are seeing.

4 Likes

If the app doesn't like the cross-signed root cert...
Then,

  • have you tried changing the chain file?
  • have you tried using a cert from another FREE CA?
2 Likes

This entire issue to me sounds like this is yet another issue with Windows not loading ISRG Root X1 (we have seen this numerious times in the past years).

Microsoft Windows lazily loads most root certificates: Except from a certain set of "preloaded" roots, all root certificates are loaded on demand, as computers establish TLS connections. There have been reports that this doesn't always work: Configuration, Firewall, Policy or other issues can prevent Windows from updating its platform trust store.

The platform trust store is used by Microsoft's TLS implementation "schannel". Historically, browsers such as Chrome or Edge also used this platform trust store. However, starting with Chrome ~108 and Edge 112, both browsers no longer use the Windows platform trust store. Instead, they use their own certificate verifiers. Therefore, these browsers can no longer be used to troubleshoot any certificate-related problems. However, Powershell, .NET/C# runtime and various other Windows components still use schannel and hence the platform trust store.

On the affected machines, I would suggest to check the certificate store (via "certmgr.msc") to ensure that ISRG Root X1 is listed under trusted root certificates. I suspect that it is not, which would confirm the issue.

8 Likes

Thanks for the correction @Nummer378

3 Likes

Thank you for your suggestion regarding capturing more information from C#. I appreciate the insight.

The primary reason I've been using curl is to eliminate the possibility of any bugs within our own code interfering with the analysis of this issue.

By choosing a common and standard networking tool like curl, it allows for a more straightforward discussion. If curl doesn't work on a subset of Windows users, at least we can rule out issues inherent to curl's own implementation.

1 Like

I appreciate your point regarding the relevance of browser behavior for most users. The reason I've chosen to use curl as a test is because our game encountered the same certificate issues on these users' computers. I asked these users to try using curl to see if the problem persisted outside of our game environment.

The rationale behind choosing curl is that it's a standard and widely-used networking tool. If curl exhibits the same issues, it provides a solid indication that the problem might not lie within our game's code. Moreover, using a tool like curl allows everyone to discuss the issue on the same baseline, making it easier to discuss this issue.

1 Like

What language, framework, library etc does your game code use to make the https/tls connection? curl is a special case as it does not originate as a windows native tool but it has been made to work for windows, it's behavior may be different to some other windows clients.

Here is an example of the correct root certificate installed in the local machine certificate store:

If this is not present there would be problems validating the certificate path for some windows clients. W are assuming that the users machines have the correct date/time set.

3 Likes

would visiting le site with ie mode solve this? it should still use windows ssl backed right?

4 Likes