Cert issued but is not being returned. Keeps trying to make a new cert resulting in hitting (5) cert domain limit

Help
1

Hi there, we use let's encrypt for custom domains on numerous websites that we host.

Many of our domains failed to renew this morning.

Our system is on Kubernetes and we use Cert Manager. We use the lua-resty-auto-ssl package.

My question is: How can we use the already issued 5 certs? I have heard this suggestion numberous times on these forums.

You'll see for our domain booking.keithreillyphotography.com, 5 certs were issued, OCSP is good. But the cert seems to not be returning because it keeps trying to issue
crt.sh | 9069310812

Here is a log: [lua] ssl_certificate.lua:291: auto-ssl: could not get certificate for booking.keithreillyphotography.com - using fallback - failed to get or issue certificate

Then we see these logs:

  1. Handling authorization for booking.keithreillyphotography.com
  2. Found valid authorization for booking.keithreillyphotography.com
  3. ssl_certificate.lua:291: auto-ssl: could not get certificate
  4. lets_encrypt.lua:40: issue_cert(): auto-ssl: dehydrated failed

This continues until the 5 cert limit per domain is hit

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:

Not working examples:
booking.penntographystudios.com
booking.omarlopezphotos.com
booking.keithreillyphotography.com

Working examples:
booking.nicoleashley.ca

I ran this command:

It produced this output:

My web server is (include version):

The operating system my web server runs on is (include version):
Openresty

My hosting provider, if applicable, is:
Google Kubernetes

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): Certmanager

2

Hello @bookfocal, welcome to the Let's Encrypt community. :slightly_smiling_face:

There are Rate Limits - Let's Encrypt and Duplicate Certificate Limit - Let's Encrypt

2 Likes
3

That sounds like a problem using cert-manager and not so much a problem with Let's Encrypt. You should see this post

4 Likes
4

Thanks Bruce, we are aware that there are rate limits. What happened was a part of our infrastructure was deprecated and caused the certs to fail repeatedly before we could fix it.

It is fixed now but we are trapped by the rate limit

I have read that the solution in this case is to use the existing issued certs. Like this one: crt.sh | 9069310812

The issue is that for some reason this cert is not working. We have the private key for it etc. Does this cert look all good?

5

All 3 have hit rate limits.

  1. https://letsdebug.net/booking.penntographystudios.com/1435475
  2. https://letsdebug.net/booking.omarlopezphotos.com/1435474
  3. https://letsdebug.net/booking.keithreillyphotography.com/1435470
1 Like
6

Not sure how much control over the system you have ...
But if you could switch to ZeroSSL, you could obtain a cert that way.
OR
If you can combine names, you could obtain one single cert (with all names on it) that way.

4 Likes
7

Hi Bruce, yes I agree they have hit the rate limit.

My question is if the (5) certs that were issued for booking.keithreillyphotography.com for example are still usable?

Provided we can fix the issue with cert manager!

8

First, if you have the private key for a cert that should work. But, you need to configure your service to use it. The one for your example is using a self-signed cert instead. See:

The docs for the Let's Encrypt work-around for this rate limit is below.

4 Likes
9

Here is a list of certificates crt.sh | booking.keithreillyphotography.com, any one of the 5 issued on 2023-04-05 should be fine, provided you can find the matching Private Key.

Taking this one crt.sh | 9069310812, as an example,
You can download the Certificate by clicking on PEM
image

2 Likes
10

Thank you Bruce and Mike for your replies. I really appreciate the help.

Bruce - from what you are saying the last 5 certs were correctly issued and signed by let's encrypt correct?

Mike - I am referring to this Cert for booking.keithreillyphotography.com -> crt.sh | 9069310812
This one I believe is signed by let's encrypt.

The issue I believe we are having is with cert manager not actually using the issued let's encrypt certs. Which is very weird because half of our domains are fine -> like this one https://decoder.link/sslchecker/booking.lindsaycoulterphoto.com/443

Again thanks for both your help. I was mainly trying to narrow down that the issue was that:

  1. the let's encrypt cert for booking.keithreillyphotography.com -> crt.sh | 9069310812 - although at the rate limit could still be used.
  2. That we need to figure out why cert manager is not using the valid certs.

Let me know if I seem to be on the right track here! Thank you all so much!

2 Likes
11

Yes, and this might help if you have to build the certificate chain by hand: Long (default) and Short (alternate) Certificate Chains Explained
Also see Chain of Trust - Let's Encrypt

4 Likes
12

Yes, that was what I was talking about too. All certs in crt.sh from Let's Encrypt were validly signed by them. You also need your private key though to make use of them. And, as I noted, you are using a self-signed cert for that domain not a Let's Encrypt cert

Visit this link and you'll see what I mean

3 Likes
13

Thanks - yes I understand now Mike. The issue is that is using the fallback self-signed cert instead of the proper let's encrypt issued certs.

Need to figure out how to get it using the let's encrypt certs. Something is funky with Cert Manager as some of our sites are fine but some aren't :frowning:
Thanks again both Mike and Bruce for your patient explanations!

4 Likes
14

Cheers.

Now, revisit my link in post #3 :slight_smile:

4 Likes
15

Thanks mike, reviewing #3 now!

4 Likes
16

Also note that further discussion of why this failed (specifically with OpenResty and the lua-resty-auto-ssl package) is in this thread: OpenResty with resty.auto-ssl failing with curl error 3 - #12 by jcjones

3 Likes
17

I think that this problem is possibly caused by the API change.

This means that the acme client you use (cert manager) might not be compatiable with the asynchronous order finalize API

The brownout period was Apr 5: 17:07~22:16 UTC.
Your cert is issued at Apr 5 17:16 UTC, which is exactly in that period.

3 Likes
18

It's actually very difficult for me to tell whether or not Cert Manager supports async finalization. I believe it does.

But the original post says:

That last bit is important. We know that lua-resty-auto-ssl does not support async finalization, which is why I linked to the other thread discussing that client specifically.

4 Likes
19

Just for future reference, I've confirmed that Cert Manager should, in isolation, support asynchronous finalization. Its finalizeOrder method uses CreateOrderCert to do the bulk of its work. That method is provided by golang.org/x/crypto, and has always supported polling the order after finalization. So it appears that in this case, it's just the use of lua-resty-auto-ssl that is causing the issue, not the use of Cert Manager.

6 Likes
20

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.