We have been using Let's Encrypt for getting certificates for our application/server. Since last few days, We are not able to get a certificate for our domains.
We are getting errors.
Error when use this domain :- letstest.sandbox.facefirst.dev
validate domain, domain: letstest.sandbox.facefirst.dev, exception:
Can not find issuer 'C=US,O=Internet Security Research Group,CN=ISRG Root X1' for certificate 'C=US,O=Let's Encrypt,CN=R3'.
Error when use
Exception encountered while attempting to validate domain, domain: dny800158-1.sandbox.facefirst.dev,
exception: Fail to load resource from 'https://acme-v02.api.letsencrypt.org/acme/new-order'.
urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many certificates (5) already issued for this exact set of domains in the last 168 hours:
dny800158-1.sandbox.facefirst.dev,
retry after 2024-02-14T07:26:10Z:
I do see that certificate gets created when i search for facefirst.dev but for whatever reason it is not sending us reply back in time and somehow app is getting timeout or error?
Do I have to increase the timeout from our application when it calls for getting the response for creating a new certificate?
Please let me know.
I've moved your post to the Help category. If you fill this form out (which the Help category has be default), it will be much easier for people to help you.
Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.
My domain is:
I ran this command:
It produced this output:
My web server is (include version):
The operating system my web server runs on is (include version):
My hosting provider, if applicable, is:
I can login to a root shell on my machine (yes or no, or I don't know):
I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
I'll see what I can do to try to point you in the right direction though.
There was a change made last week, to no longer present the expired DST Root CA X3 certificate which had been included in certificate chains by default to improve compatibility with old Android versions. Nothing should be needing it anymore unless they need compatibility with those clients. You're going to need to give more details on what system and process is showing you this error. It sounds like it doesn't know about the ISRG Root X1 certificate, which would be unusual for a "normal" OS root store.
This error is saying that you did already get 5 certificates just fine, and it has rate-limited you to prevent further abuse of Let's Encrypt's resources. Whatever problem you're having, it isn't with getting the certificate, so getting even more certificates isn't going to help you.
It is getting the rate limit is because when we are calling the LE for getting a new certificate, we don't get the response back in time and our process throws error waiting for it. So User ends up trying again and again. That is one reason i can think of for getting the rate limit error. We have not changed our process to get new certificate but since last few days, it has stopped working and we are seeing duplicate certificates generates for domain like facefirst.dev.
At which point during the issuance process does this happen? When calling the "finalize" endpoint to get the certificate issued? When accessing the "certificate" URL from the finalized Order object to download the certificate itself?
Again, the only reason you're seeing duplicate certificates generated is because the certificate issuance process is working. If your client is failing to download the certificate after issuance has succeeded, that needs a fix on the client side.
Through the use of my psychic powers you are directly or indirectly using the Certes library with Kestrel as your webserver so I'm guessing you either have a custom certificate order process or you're using a kestrel middleware to fetch the certs (like LettuceEncrypt).
Certes can't handle building the PFX you need if the machine certificate trust store doesn't know the required root certificates. If you enable normal windows updates and remove any group policy restricting CA trusts store updates windows will normally be able to keep this up to date itself. Alternatively install ISRG Root X1 (self signed) into Trusted Root Certification Authorities.
The reason this was previously working is Let's Encrypt were issuing using the DST Root CA X3 chain (expired root), which Windows would have in it's trust store already so that was enough to build the PFX.
The reason you're hitting the rate limit for duplicate certs is the certificate download completes OK from Let's Encrypt, but the actual PFX build (the file you really use) fails.
I am experiencing the same issue as well and this is what I see in my log files:
Certificate request process failed: Certes.AcmeException: Can not find issuer 'C=US,O=Internet Security Research Group,CN=ISRG Root X1' for certificate 'C=US,O=Let's Encrypt,CN=R3'
We are trying to use the HTTP-01 renewal method for our certificate and we managed to reach the rate limit as well.
I saw you mentioned "The reason this was previously working is Let's Encrypt were issuing using the DST Root CA X3 chain (expired root), which Windows would have in it's trust store already so that was enough to build the PFX." and you also mentioned "Alternatively install ISRG Root X1 (self signed) into Trusted Root Certification Authorities."
I did check my Trusted Root Certification Authorities store and confirmed the ISRG Root X1 certificate is there with the expiration date of 6/4/2035.
I can confirm the same. I have that certificate installed under Trusted Root Certificate Authorities store.
We are still getting this error.
Can we please get some help?
Can you fill out the template I originally posted, and be really specific on all the software you're using, how you're running it, and what you're doing when you see what errors? @webprofusion made some decent educated guesses, but we just don't have any understanding of what it is that you're actually doing, and from the error you're posting it looks like you're actually getting certificates just fine (which is why you're getting rate limited) so whatever issue you're getting is in whatever you're running that's trying to use them (which you haven't described to us).
I ran this command:- We have a C# code from LetsEncrypt Github that we have downloaded and made some changes. But that is what we use to create Order for getting new certificate and then
We are also listening on port 80 on the same machine to receive the challenge resposne from LetsEncrypt. Once we receive the challange and read it, we make build the certificate.
It is a basic HTTP challange.
It produced this output:- Certes.AcmeException: 'Can not find issuer 'C=US,O=Internet Security Research Group,CN=ISRG Root X1' for certificate 'C=US,O=Let's Encrypt,CN=R3'.'
My web server is (include version): Kestrel
The operating system my web server runs on is (include version): Windows Server 2022 Data Center
My hosting provider, if applicable, is: Azure
I can login to a root shell on my machine (yes or no, or I don't know): Yes
I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No
The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): We are not using certbot
So maybe there's a bug in whatever code this is? Is it using the current version of the Certes library? I'm not familiar with that library, but it sounds like it's not configured with a correct trust store. Can you share more details about the code and how it's configured?
Same code has been working for years and started having issues since FEB 8. Whatever changes made has broken this integration. You can also see other users having same issue.
That may be, but it's looking like it needed to be updated at some point in the past few years to no longer depend on the DST Root CA X3 cross-sign which was a hack for old Android compatibility, and to use the current actual Let's Encrypt roots.
Here is some snippet of the code that is requesting a new certificate.
//Createing Order
var order = await acme.NewOrder(new { host });
var authzs = (await order.Authorizations());
var authz = authzs.First();
var httpChallenge = await authz.Http();
var orderInfo = new OrderInfo
{
Order = order,
Challenge = httpChallenge,
HostName = host,
FailFunc = failFunc,
SuccessFunc = successFunc,
CancellationTokenSource = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken, new CancellationTokenSource(timeoutMs).Token)
};
// Even if the challenges never come, attempt to finish creating the certificate after 30 seconds.
// We do this because, if you request a duplicate certificate within a certain time period, there will be no
// new challenges issued by Let's Encrypt.
await Task.Run(async () =>
{
// Give some time to Let´s Encrypt to send challenges and process our responses.
await Task.Delay(60 * 1000, orderInfo.CancellationTokenSource.Token);
if (!orderInfo.CancellationTokenSource.Token.IsCancellationRequested)
{
await BuildCertificate(orderInfo, orderInfo.HostName);
}
else
{
await orderInfo.FailedAsync($"Domain validation cancelled for {orderInfo.HostName}", orderInfo.HostName, null);
}
});
I am seeing this in your help documents about the shortening chain of trust changes you guys implemented.
If you are an ACME client author , please make sure that your client correctly downloads and installs the certificate chain provided by our API during every certificate issuance, including renewals. Failure modes we have seen in the past include a) never downloading the chain at all and only serving the end-entity certificate; b) never downloading the chain and instead serving a hard-coded chain; and c) only downloading the chain at first issuance and not re-downloading during renewals. Please ensure that your client does not fall into any of these buckets.
We do have an ACME client, but we have some questions.
Do you have an example of such changes implemented in any language or client implementations? We prefer C# but we can use any other language as an example.
Is there a manual step to add Shorten Chain of Trust to Windows OS certificate store?
Figure out where this exception is coming from. This is the root problem, and I don't know enough about Windows and C# to understand why it's happening.
Many ACME Clients are open-sourced on github. The suggestions you referenced are not specific to this recent change but have always been the proper way to handle chains. It was just pointing out that if you were not doing it right you should be sure to start.
Some examples on github off the top of my head
Certbot
acme.sh
lego
dehydrated
Also see below as I am sure there are other open-source examples.