Too many certificates showing up after yesterday's downtime

My domain is: gumblert.tech

I ran this command:

/usr/bin/env docker run --rm --name=matrix-certbot --user=991:991 --cap-drop=ALL -p 80:8080 --mount type=bind,src=/matrix/ssl/config,dst=/etc/letsencrypt --moun
t type=bind,src=/matrix/ssl/log,dst=/var/log/letsencrypt docker.io/certbot/certbot:amd64-v1.17.0 certonly --non-interactive --work-dir=/tmp --http-01-port 8080  --stan
dalone --preferred-challenges http --agree-tos --email=<my email> -d matrix.gumblert.tech

It produced this output:

 An unexpected error occurred:                                                                                                                                       
   There were too many requests of a given type :: Error creating new order :: too many certificates (5) already issued for this exact set of domains in the last 168  
hours: matrix.gumblert.tech: see https://letsencrypt.org/docs/rate-limits/                                                                                              
   Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more  
details.

My web server is (include version): nginx 1.21.1

The operating system my web server runs on is (include version): Ubuntu 20.4

My hosting provider, if applicable, is: Digital Ocean

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): 1.17.0

I tried renewing my certificates for the root domain and a couple of subdomains during downtime, which failed. Then I waited until the status page was orange, and tried renewing again, and I got this error above. I'm sure this is wrong, because I didn't get a success even once, and the certificates were not created where I would expect them.

Is it possible to reset this rate limit so I can renew my certificates without waiting a whole week?

1 Like

Welcome to the Let's Encrypt Community :slightly_smiling_face:

I'm rather confused. I see a whole lot of certificates issued. Were they not saved into persistent storage?

https://crt.sh/?q=matrix.gumblert.tech

https://crt.sh/?q=gumblert.tech

1 Like

Thanks, this must be a bugged auto-renewal on my side then, I was pretty sure it was off because it wasn't replacing the old certs.

2 Likes

I wondered about that. The history looks pretty consistent and expected up until about a month ago. Did you switch setups at the beginning of July?

It looks like around June 21 that something changed.

1 Like

It's possible that maybe certbot's data folder got corrupted a bit. Sometimes we see the symlinks stop pointing to the recent certificates.

Try running this:

sudo certbot update_symlinks

Then reload your webserver configuration.

2 Likes

Thanks for the help, now that I checked the logs more thoroughly, it was indeed broken symlinks. Do I have to wait a full week or is there anything I can do?

3 Likes

If you update the symlinks and reload your webserver, you should be good. This is assuming that you still have the private keys for the certificates. There are backup copies of them in the archive and keys folders.

If the symlinks are severely corrupted, I may be able to guide you to fix them. I would need the full output of this to do so:

sudo ls -lRa /etc/letsencrypt

You will need your symlinks working to renew going forward anyhow.

1 Like

Since you're operating in a docker container, you might need some additional syntax to interact with certbot.

1 Like

That's your big trouble there. The certonly parameter only acquires a certificate. It doesn't reload your webserver afterwards to install it.

1 Like

That's a bit long. Maybe this instead:

certonly --work-dir /tmp --http-01-port 8080 --standalone -d "matrix.gumblert.tech" --non-interactive --keep

I'm assuming your external port 80 is routed to port 8080 of this server?

Typically with certonly you would want to add a --deploy-hook to automate any type of further deployment of new certificates when the renew.

1 Like

I checked the archived certs but they're very old, and all expired unfortunately.

To manage certificates and deployments, I'm using this ansible playbook: GitHub - spantaleev/matrix-docker-ansible-deploy: Matrix (An open network for secure, decentralized communication) server setup using Ansible and Docker
And after renewal, it should reload the server.

So far, I assumed it works correctly. But it looks like the automatically renewed certificates aren't even saved, because the symlinks were broken and they were just dropped. When I tried renewing manually, I had to delete an old expired certificate that was causing the renewed ones to be dropped.

1 Like