Since May 28th, we have been unable to issue certificates using Windows Server 2016. The program we developed returns a schannel error.
We tested it in our production and test environments, which are in different datacenters. The same program runs without problems on Windows Server 2019 and later.
Have you recently made any changes to the cyphers and TLS supported by Let's Encrypt?
Yes, the API servers now use an ECDSA certificate rather than RSA.
Please see this thread: Certify the Web Cannot create secure channel error - #10 by mcpherrinm
It has a discussion of this and how to modify your system to support that.
can your server 2016 can handle P384 certificates?
From what I understood, this week Let's Encrypt changed the supported ciphers without prior notice or a testing period on the staging servers. Is that right?
In this case, I kindly ask that you return the previous ciphers in production and change them only in staging, so that we can calmly test and make the necessary changes in our environment.
Yes.
@mcpherrinm Just wanted to make sure you saw this.
I don't know if they will or not. Another workaround you might want to consider is using another Certificate Authority; it looks like Buypass Go is using an RSA cert on their endpoint for now, though they're also planning changes soon. And I get that it's also a production config change, though Buypass does have a staging environment as well. Just another option, I don't know if it's less scary than changing your production enabled outgoing cipher suites.
If you have an urgent requirement to renew or create a new certificate with Let's Encrypt then you should apply the fix to enable TLS1.2 (if not already enabled for client schannel communication) and ECDSA cipher suites for at least:
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
and
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
Thank you Christopher, Mike, Peter and Orangepizza!
Including the ciphers TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 and TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 in the key [HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Cryptography\Configuration\SSL\00010002] solved the problem.
I also had to restart our Windows Server 2016 webservers.
I wasted two days trying to solve the problem here, without knowing that they made these changes without warning, which is not usual in Let's Encrypt.
To Let's Encrypt staff:
The staging environment is one of the best features of Let's Encrypt. Please don't forget to use it and let us know whenever you make changes that may affect us.
We're working on an announcement now. This change wasn't intentional, but a result of a default change in certbot, so new instances came with new certs.
This change was made in the staging environment first, but we didn't receive any complaints at that time.
Our metrics show the number of windows 2016 users is very small, as it's been out of standard Microsoft support for 3 years now.
I'm sorry for any disruption that this caused.
Apologies accepted, Matthew.
I usually only use the staging servers when Let's Encrypt announces changes.
To be even more safe, I do this monthly as well, even when there are no announcements.
BTW, Microsoft's official, free support for security updates for Windows Server 2016 will end on January 12, 2027.
[Windows Server 2016 - Microsoft Lifecycle | Microsoft Learn](https://Windows Server 2016 Lifecycle Policy)
Ah, I thought the "Mainstream End Date" was the important one. We don't have any Windows folks here, so that's a misunderstanding on my part.
As I draft this, I suppose this is as good a place to ask as any:
Are there supported windows versions before 2016?
Are there later versions that might be affected?
The issue isn't really specific to Windows Server 2016; its default configuration would have worked fine. The cases we've seen so far are when the administrator in the past had specifically disabled some cipher suites to only the ones that they wanted to use. And it seems that it generally uses the same list for both incoming connections and outgoing connections, so people may have limited it to only RSA cipher suites since they only used RSA certificates themselves.
I'm sure someone could have done the same on other platforms, but Windows may make it easier since it has a central TLS library that just about everything uses, as opposed to the Linux world where there are multiple possible libraries, and also less administrators doing things like setting openssl defaults across the server.
It'd be good for the announcement to be clear on the dates that the change was user-facing. From the CT logs, it looks like staging started getting ECDSA certs on May 21, and production on May 22. But people only started seeing problems a couple days ago, so presumably these new certificates weren't public-facing for some time?
Just a tidbit on your previous post ...
@webprofusion said below in another thread so others may be affected that did not specifically disable them.
Server 2016 is currently the oldest supported release. Microsoft has a list of the end of support dates here. Windows Server release information | Microsoft Learn
It is fairly common to run windows server releases up until the end of extended support, as they are still secure, just no longer receiving new features.
Edit: I see the lifecycle link was already posted, in that case, disregard.
People who pay Microsoft for extended support can have Server 2012 security updates to Oct 2026 currently but really whether a version of windows is supported by Microsoft or not doesn't really change how many are in use or how many support requests we'll see.
It's probably not worth being too specific about Windows versions because you can encounter this on newer versions of Windows Server if you have previously upgraded a much older version. I haven't tested that exhaustively, so it's a maybe.
Based on the last few days of sampling I'd expect this to affect less than 1% of Windows Servers using Let's Encrypt because many will already have the enabled these cipher suites so they could talk to other things, but that's still a few thousand users who will come looking for support eventually.
Affected operating systems would be any versions of Windows Server upgraded in-place from an older version (e.g. Server 2016 upgraded from Server 2008)
Affected clients could include Certify The Web, Posh-ACME, win-acme (wacs), or simple-acme or any others native to Windows that use built in Windows Schannel functionality for communication (it's possible to not be affected if your client uses it's own stack like openssl, but that's not generally relevant).
The key thing is to enable TLS 1.2 (if not already enabled), enable at least some minimal ECDSA cipher suite coverage using whichever means, and to reboot (ouch!). It's only urgent if you have certs approaching renewal, so it can generally wait until a maintenance window.
Certify The Web users will get an early notification if there certs are failing to renew, via our status reporting API (by default, knew that RSA cert would come in handy), other clients may be less fortunate and won't notice until their cert expires.
I don't think it would need to be an emailed announcement, an official post on here is probably enough to point people to, and it can be edited accordingly as new information surfaces.
Thanks, that’s helpful.
Note we already require TLS1.2, as of Rejecting TLS 1.0 / 1.1 for inbound ACME connections