I work at the University of Washington with domain uw.edu
Our k8s cert-manager has been unable to issue new certs all week, providing messages like the following:
The certificate request has failed to complete and will be retried: Failed to wait for order resource "grouper-debug-app-managed-tls-x565n-3451792455" to become ready: order is in "errored" state: Failed to create Order: 429 urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many certificates already issued for "uw.edu". Retry after 2023-02-17T08:00:00Z: see Rate Limits - Let's Encrypt
The UW is large and decentralized and I'm content to believe we have in fact exceeded this limit. Before requesting any sort of rate limit increase it seemed prudent to track down whomever had requested new certs and coordinate. However I haven't figured out how to find those certs.
Checking https://crt.sh/?q=uw.edu&exclude=expired which appears to sort most recent first I see only two certificates issued this year - one on 2023-02-09 and one on 2023-01-15.
Using ./lectl uw.edu --sans -m1000 (GitHub - sahsanu/lectl: Script to check issued certificates by Let's Encrypt on CTL (Certificate Transparency Log) using https://crt.sh) I get output
lectl 0.21 (2020-August-09)
2023/February/17 12:26:20 - Checking all certs for uw.edu
I have found 600 non expired certificates (290 final certs and 310 pre certs) (max number of certs searched: 1000) for domain uw.edu and its subdomains *.uw.edu
...
You have issued 0 certificates in last 7 days so you could issue 50 more certificates now.
Can anyone help me understand how to find the recently issued certs?
Thanks,
-mike
My domain is: uw.edu
I ran this command: kubectl describe certificate jdev-web-managed-tls
It produced this output:
Truncated output - let me know if you want the whole thing
Status:
Conditions:
Last Transition Time: 2023-02-15T22:34:51Z
Message: Issuing certificate as Secret does not exist
Observed Generation: 1
Reason: DoesNotExist
Status: False
Type: Ready
Last Transition Time: 2023-02-17T05:35:01Z
Message: The certificate request has failed to complete and will be retried: Failed to wait for order resource "grouper-debug-app-managed-tls-x565n-3451792455" to become ready: order is in "errored" state: Failed to create Order: 429 urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many certificates already issued for "uw.edu". Retry after 2023-02-17T08:00:00Z: see Rate Limits - Let's Encrypt
Observed Generation: 1
Reason: Failed
Status: False
Type: Issuing
Failed Issuance Attempts: 6
Last Failure Time: 2023-02-17T05:35:01Z
Events:
My web server is (include version): nginx 1.19.9
The operating system my web server runs on is (include version): Alpine Linux v3.14
My hosting provider, if applicable, is: NA
I can login to a root shell on my machine (yes or no, or I don't know): I cannot login to the cert-manager pod
I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no
The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): Unsure - I'd have to check with the infrastructure people
