We've encountered a peculiar issue. We use a Let's Encrypt certificate for our site: https://partyanimals.cn.
Some Windows(both win11 and win10) users are seeing a "certificate expired" error when using curl -I on our website and on https://letsencrypt.org/. Yet, even for these affected users, the certificate appears normal when accessed via browsers like Edge.
Step to reproduce:
On affected Windows machine, run curl -I https://partyanimals.cn or curl -I https://letsencrypt.org.
The exact error message they receive is:
curl: (35) schannel: next InitializeSecurityContext failed: SEC_E_CERT_EXPIRED(0x80090328)
From our end, our site's certificate appears valid, and of course, we would expect the certificate for letsencrypt.org to be without issues. Yet, this problem doesn't seem to impact all Windows users, only a subset.
Would greatly appreciate any insights or advice on this sporadic behavior with curl and Let's Encrypt certificates.
I haven't seen this, but I haven't done much running of curl from Windows.
I have some random thoughts, though:
Running curl within Powershell is actually an alias for Invoke-WebRequest. And there's also a "real" curl available on some systems (maybe all systems with current updates?). And I haven't played with Windows Subsystem for Linux, but it wouldn't shock me if that had its own curl as well. Can you be really specific on which curl people are seeing problems with, and if there might be a difference between users that see it working and users that don't? (It's really easy to run "Terminal" and not realize one is actually running Powershell.)
Can you test the following URLs, to see if there's a difference? They use different certificate chains, and I'm wondering if the expired DST Root CA X3 (which modern systems should just be ignoring) is causing something somewhere to be unhappy.
https://valid-isrgrootx1.letsencrypt.org/
https://valid-isrgrootx2.letsencrypt.org/
https://helloworld.letsencrypt.org/
And of course, if you could narrow down which systems have a problem, that would be helpful. Can you compare curl --version, winver, etc. between systems that work and systems that don't? Can you test it on a fresh clean install (maybe in a new VM)? Are there any "proxy" or "security" firewalls that might be intercepting the connection?
It sounds like the underlying SChannel [or OpenSSL] is outdated.
[which may be specific to Windows curl (and not system-wide)]
Have you tried "Windows Updates"?
Definitely more tests need to be run to better understand things more clearly.
OR
There is a proxy involved that is part of the problem...
Could you get some of those users to save a copy of the certificate that they see and send it to you? Maybe they're getting network interference that is actually a MITM attack or something.
They could save it from the browser, or run something like
Your site has very good results at https://ssllabs.com/ (indicating that most clients should negotiate a connection successfully), so it doesn't seem like this is likely a problem with your own configuration.
It's most likely (?) the expired root DST Root CA X3 being flagged up by a specific version of curl or specific configuration of machine trust store.
If curl is using Schannel (the native windows alternative to openssl) then I would expect it should also be using the windows certificate store for known roots and intermediates and so when it sees the intermediate R3 leads to ISRG Root X1 (self signed) it should resolve that instead of the ISRG Root X1 > DST Root CA X3 path. That relies on the assumption that curl is following the normal windows certificate path validation and that ISRG Root X1 (self signed) is present in the machines Local Machine > Trusted Root Certification Authorities certificate store. It normally will be present unless group policy/registry has been set to disallow automatic root ca cert updates.
[However this problem is not new and you would have been seeing it since Sept 2021]
You're right about the alias in PowerShell. In Windows 10, there's indeed a separate curl located at C:\Windows\System32\curl.exe. When using the Windows Command Prompt, this particular curl is executed. However, I feel the issue might not be related to the curl version. The reason being, our game is written in C#, and on the machines of these affected users, our game also fails to access partyanimals.cn due to a certificate error.
We will ask the affected users to test the mentioned URLs. Once we have the results, we will update this thread.
We will be comparing the curl --version outputs between the systems that are affected and those that are not. As for the Windows version, we tried replicating the issue on a local machine with the same Windows 10 version as the users reported, but we couldn't reproduce the problem.
Thank you very much for your insights, and we'll keep you posted.
If that's the problem you're actually trying to solve, you might have better luck capturing more information from C# than from trying to run curl. In particular, if you can capture the certificate chain that the client is seeing, or maybe even a packet capture of the problem, you might be able to see exactly what might be expired.
Our game, which is written in C#, encounters the same certificate issue when accessing PartyAnimals.cn on these users' systems. So, I believe it's unlikely to be an issue specific to the curl version.
We had one user attempt to update Windows, but it did not resolve their issue. We plan to suggest more affected users to try updating Windows as a possible solution.
I'm not very familiar with certificate-related technologies, so I'm unsure about the next steps for testing. I'd greatly appreciate any recommendations or guidance you can provide.
When you mention a "proxy", are you referring to a network proxy? We've ruled out this possibility. Some of our users have two computers at home, and under the exact same network conditions, one machine works fine while the other encounters the issue.
We have downloaded the certificate from the affected users' computers, both using the browser and the openssl command-line tool. In both cases, the certificates they received are identical and indeed correct.
What's peculiar is that for this small subset of users, while their browsers indicate that the certificate for our website is trustworthy, both the curl command-line and our game (written in C#) throw certificate-related errors when accessing our site.
Hi webprofusion:
We recently launched our game and just discovered this issue. Additionally, when I used Cert Bot to apply for a certificate, the full certificate chain I received is:
Is this normal? It seems that the root certificate displayed in the Edge browser is different from the one I obtained from Cert Bot.
Could this be the reason why some users think our certificate has expired?
What you see in your browser and what your server actually serves can be different things. On Windows (Edge, IE) the client (browser) will resolve the best path it can find, which is generally always Leaf > R3 > ISRG Root X1 if windows knows that root certificate (which it usually does because by default the OS dynamically keeps the intermediate and root trust store certificates up to date). However on Firefox for instance, it will resolve differently because it doesn't use the windows native certificate path resolution or certificate store.
Just to jump on these, yes: There can be proxies configured in one browser, but not in another client, on the same system. Even if they're on the same network, the computers might be configured to use different proxies. Often these are intentional on the system administrator's part (routing traffic through some sort of monitoring firewall, or through a VPN that gives access to corporate resources), but sometimes the user isn't aware of how their administrator configured things. (And sometimes there's malware that configured proxies in a way that really isn't what the user intended.) There are a lot of possibilities, so it's really hard for us random people on the Internet to tell you what your users are seeing.
This entire issue to me sounds like this is yet another issue with Windows not loading ISRG Root X1 (we have seen this numerious times in the past years).
Microsoft Windows lazily loads most root certificates: Except from a certain set of "preloaded" roots, all root certificates are loaded on demand, as computers establish TLS connections. There have been reports that this doesn't always work: Configuration, Firewall, Policy or other issues can prevent Windows from updating its platform trust store.
The platform trust store is used by Microsoft's TLS implementation "schannel". Historically, browsers such as Chrome or Edge also used this platform trust store. However, starting with Chrome ~108 and Edge 112, both browsers no longer use the Windows platform trust store. Instead, they use their own certificate verifiers. Therefore, these browsers can no longer be used to troubleshoot any certificate-related problems. However, Powershell, .NET/C# runtime and various other Windows components still use schannel and hence the platform trust store.
On the affected machines, I would suggest to check the certificate store (via "certmgr.msc") to ensure that ISRG Root X1 is listed under trusted root certificates. I suspect that it is not, which would confirm the issue.
Thank you for your suggestion regarding capturing more information from C#. I appreciate the insight.
The primary reason I've been using curl is to eliminate the possibility of any bugs within our own code interfering with the analysis of this issue.
By choosing a common and standard networking tool like curl, it allows for a more straightforward discussion. If curl doesn't work on a subset of Windows users, at least we can rule out issues inherent to curl's own implementation.
I appreciate your point regarding the relevance of browser behavior for most users. The reason I've chosen to use curl as a test is because our game encountered the same certificate issues on these users' computers. I asked these users to try using curl to see if the problem persisted outside of our game environment.
The rationale behind choosing curl is that it's a standard and widely-used networking tool. If curl exhibits the same issues, it provides a solid indication that the problem might not lie within our game's code. Moreover, using a tool like curl allows everyone to discuss the issue on the same baseline, making it easier to discuss this issue.
What language, framework, library etc does your game code use to make the https/tls connection? curl is a special case as it does not originate as a windows native tool but it has been made to work for windows, it's behavior may be different to some other windows clients.
Here is an example of the correct root certificate installed in the local machine certificate store:
If this is not present there would be problems validating the certificate path for some windows clients. W are assuming that the users machines have the correct date/time set.
