Renewal Problem

Help
#1

My domain is: mpsconline.mizoram.gov.in

I ran this command: ./certbot-auto certonly —webroot -w /site/account_name/public_html -d domain.mizoram.gov.in

It produced this output: An unexpected error occurred:

There were too many requests of a given type :: Error creating new order :: too many certificates already issued for exact set of domains: mpsconline.mizoram.gov.in: see Rate Limits - Let's Encrypt - Free SSL/TLS Certificates

My web server is (include version): Apache

The operating system my web server runs on is (include version): RHEL 7

My hosting provider, if applicable, is: N/A

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): WHM

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):1.10.1

I have been trying to renew my certificate, but because or rate limit its giving this error. But its been like this for more than 7days without any renewal request. And when I manually get the public key from crt.sh | mpsconline.mizoram.gov.in and then use the keys from 'etc/letsencrypt/keys' to install in WHM, the public key is correct but the private key do not matched.

I am running out of options and my site is really need to be up with HTTPS by 1st February. Please Help URGENT!!!

2 Likes
#2

Hello @Lalrinfela,

You are issuing a cert for your domain every day until you reach the 5 certs per same sub set of domains per 7 days rate limit.

CA  CERT TYPE   DOMAIN (CN)                KEY ALG      VALID FROM             VALID TO               EXPIRES IN  SANs
R3  Final cert  mpsconline.mizoram.gov.in  RSA 2048bit  2021-Jan-29 18:18 UTC  2021-Apr-29 18:18 UTC  89 days     mpsconline.mizoram.gov.in
R3  Final cert  mpsconline.mizoram.gov.in  RSA 2048bit  2021-Jan-28 17:44 UTC  2021-Apr-28 17:44 UTC  88 days     mpsconline.mizoram.gov.in
R3  Final cert  mpsconline.mizoram.gov.in  RSA 2048bit  2021-Jan-25 17:46 UTC  2021-Apr-25 17:46 UTC  85 days     mpsconline.mizoram.gov.in
R3  Final cert  mpsconline.mizoram.gov.in  RSA 2048bit  2021-Jan-24 17:44 UTC  2021-Apr-24 17:44 UTC  84 days     mpsconline.mizoram.gov.in
R3  Final cert  mpsconline.mizoram.gov.in  RSA 2048bit  2021-Jan-23 17:43 UTC  2021-Apr-23 17:43 UTC  83 days     mpsconline.mizoram.gov.in
R3  Final cert  mpsconline.mizoram.gov.in  RSA 2048bit  2021-Jan-22 17:43 UTC  2021-Apr-22 17:43 UTC  82 days     mpsconline.mizoram.gov.in
R3  Final cert  mpsconline.mizoram.gov.in  RSA 2048bit  2021-Jan-21 17:43 UTC  2021-Apr-21 17:43 UTC  81 days     mpsconline.mizoram.gov.in
R3  Final cert  mpsconline.mizoram.gov.in  RSA 2048bit  2021-Jan-18 17:44 UTC  2021-Apr-18 17:44 UTC  78 days     mpsconline.mizoram.gov.in
R3  Final cert  mpsconline.mizoram.gov.in  RSA 2048bit  2021-Jan-17 17:43 UTC  2021-Apr-17 17:43 UTC  77 days     mpsconline.mizoram.gov.in
R3  Final cert  mpsconline.mizoram.gov.in  RSA 2048bit  2021-Jan-16 17:46 UTC  2021-Apr-16 17:46 UTC  76 days     mpsconline.mizoram.gov.in
R3  Final cert  mpsconline.mizoram.gov.in  RSA 2048bit  2021-Jan-15 17:42 UTC  2021-Apr-15 17:42 UTC  75 days     mpsconline.mizoram.gov.in
R3  Final cert  mpsconline.mizoram.gov.in  RSA 2048bit  2021-Jan-14 17:43 UTC  2021-Apr-14 17:43 UTC  74 days     mpsconline.mizoram.gov.in
R3  Final cert  mpsconline.mizoram.gov.in  RSA 2048bit  2021-Jan-11 17:42 UTC  2021-Apr-11 17:42 UTC  71 days     mpsconline.mizoram.gov.in
R3  Final cert  mpsconline.mizoram.gov.in  RSA 2048bit  2021-Jan-10 17:42 UTC  2021-Apr-10 17:42 UTC  70 days     mpsconline.mizoram.gov.in
R3  Final cert  mpsconline.mizoram.gov.in  RSA 2048bit  2021-Jan-09 17:42 UTC  2021-Apr-09 17:42 UTC  69 days     mpsconline.mizoram.gov.in
R3  Final cert  mpsconline.mizoram.gov.in  RSA 2048bit  2021-Jan-08 17:42 UTC  2021-Apr-08 17:42 UTC  68 days     mpsconline.mizoram.gov.in
R3  Final cert  mpsconline.mizoram.gov.in  RSA 2048bit  2021-Jan-07 17:42 UTC  2021-Apr-07 17:42 UTC  67 days     mpsconline.mizoram.gov.in
R3  Final cert  mpsconline.mizoram.gov.in  RSA 2048bit  2021-Jan-04 17:42 UTC  2021-Apr-04 17:42 UTC  64 days     mpsconline.mizoram.gov.in
R3  Final cert  mpsconline.mizoram.gov.in  RSA 2048bit  2021-Jan-03 17:43 UTC  2021-Apr-03 17:43 UTC  63 days     mpsconline.mizoram.gov.in
R3  Final cert  mpsconline.mizoram.gov.in  RSA 2048bit  2021-Jan-02 17:43 UTC  2021-Apr-02 17:43 UTC  62 days     mpsconline.mizoram.gov.in
R3  Final cert  mpsconline.mizoram.gov.in  RSA 2048bit  2021-Jan-01 17:42 UTC  2021-Apr-01 17:42 UTC  61 days     mpsconline.mizoram.gov.in
R3  Final cert  mpsconline.mizoram.gov.in  RSA 2048bit  2020-Dec-31 17:43 UTC  2021-Mar-31 17:43 UTC  60 days     mpsconline.mizoram.gov.in

So something is wrong with your setup, did you manually change something inside /etc/letsencrypt/ dirs?

Please show the output of these commands (as root):

certbot-auto certificates

or if certbot-auto is not in your path:

cd /path/where/is/certbot-auto/
./certbot-auto certificates

ls -la /etc/letsencrypt/{archive,live}/mpsconline.mizoram.gov.in

Cheers,
sahsanu

4 Likes
#3

Hi Sahsanu,

Before I get into detail, let me put the context first: - All my government websites are under 'mizoram.gov.in' domain eg: mpsconline.mizoram.gov.in is one of them and there are other 250+ sub domain as the central government mandates subdomain policy for a State. My Government start using LetsEncrypt from sometime back in 2014 (when there was no wildcard system - at least to my knowledge), so we use to manually obtain certificate for each sub domain. So, once in a while we have to run update manualy for the domain that needs to be renewed and we maintain the validity in an excel sheet so that one of our staff keeps track of it. (We avoid the cron and auto renewal till now). We are aware of the rate limits but we never face this problem before. Thanks for reading :slight_smile:

So here are your request:

Did you manually change something inside /etc/letsencrypt/ dirs? No, we do not.

ls -la /etc/letsencrypt/{archive,live}/mpsconline.mizoram.gov.in

  [root@server mpsconline.mizoram.gov.in]# ls -al
    total 20
    drwx------   2 root root    88 Aug 31 10:45 .
    drwx------ 273 root root 12288 Jan 29 19:09 ..
    -rw-------   1 root root   692 Aug 31 10:45 README
    lrwxrwxrwx   1 root root    49 Aug 31 10:45 cert.pem -> ../../archive/mpsconline.mizoram.gov.in/cert1.pem
    lrwxrwxrwx   1 root root    50 Aug 31 10:45 chain.pem -> ../../archive/mpsconline.mizoram.gov.in/chain1.pem
    lrwxrwxrwx   1 root root    54 Aug 31 10:45 fullchain.pem -> ../../archive/mpsconline.mizoram.gov.in/fullchain1.pem
    lrwxrwxrwx   1 root root    52 Aug 31 10:45 privkey.pem -> ../../archive/mpsconline.mizoram.gov.in/privkey1.pem

certbot-auto certificates (this contains a huge list, so i just capture a part of this particular domain)

Screenshot 2021-01-30 at 2.44.38 PM
Screenshot 2021-01-30 at 2.44.38 PM1690×1380 629 KB

I have instructed my staff to stop renewal of any certificate for the next 7days as the one which is in problem now is most critical to be renewed, but i doubt this may not be the problem.. Also, the above list your see could be that we try to manually run the renew multiple times since it is not working.. just a thought.

Looking forward for the help!

BTW, thanks!
Rin

3 Likes
#4

You didn't post the output of this command:

ls -la /etc/letsencrypt/{archive,live}/mpsconline.mizoram.gov.in

I'm missing the archive part.

ls -la /etc/letsencrypt/archive/mpsconline.mizoram.gov.in

You should have all your certs, keys there but the symlinks in live dir are pointing to the first issued cert, key, etc. instead of the last one. If that is the case (I need to see the output) you could fix it using command:

certbot-auto update_symlinks

or recreating the symbolic links in live dir to the latest files in archive dir.

Cheers,
sahsanu

4 Likes
#5

Archive Part:

Screenshot 2021-01-30 at 3.24.15 PM
Screenshot 2021-01-30 at 3.24.15 PM1188×412 143 KB

It appears that all the cert files in the 'live' directory are symbolic links to the 'archive' directory.

Our workflow is that once we generate the key successfully using cerbot-auto, we then add those keys in WHM Cpanel SSL Host Manager.

But as of now with this domain, the problem is even though LetsEncrypt generate the new cert, its not downloading to our server because of rate limit and as a result even though the public cert is visible from crt.sh we do not have the matching private key to feed into WHM.

2 Likes
#6

In the archive dir you should have all the certs issued these days.

Yes, that is the way certbot works.

How do you add them? Do you move or copy files...?

If you hit the rate limit then you can't download your certificated because it has not been issued. You should try to find what is wrong with your workflow when it works because the certs have been downloaded.

Right now, the only way to get a cert for your domain is to add a new domain to that cert to avoid the rate limit so your cert should cover mpsconline.mizoram.gov.in and for example www.mpsconline.mizoram.gov.in but you need to know what is happening in your workflow or it will happen again.

Edit: I forgot to ask whether you have some kind of deploy-hook configured.

cat /etc/letsencrypt/renewal/mpsconline.mizoram.gov.in.conf

4 Likes
#7

How do you add them? Do you move or copy files...?

We copy the text content and paste in WHM Cpanel, no file is moved or rename in the process.

If you hit the rate limit then you can't download your certificated because it has not been issued. You should try to find what is wrong with your workflow when it works because the certs have been downloaded.

Since, the. public cert pem file can be downloaded from crt.sh, i assume that it was issued but just not downloaded. Thanks for clarification.

Right now, the only way to get a cert for your domain is to add a new domain to that cert to avoid the rate limit so your cert should cover mpsconline.mizoram.gov.in and for example www.mpsconline.mizoram.gov.in but you need to know what is happening in your workflow or it will happen again.

I will try but i think WHM SSL Manager might consider 'www.' part to be another sub-domain (just a though- I'll try this one). (I have just try this process, but because of the way we setup our DNS, www.mpsconline.mizoram.gov.in is not valid :cry: )

Edit: I forgot to ask whether you have some kind of deploy-hook configured.

Not that i'm aware of.

cat /etc/letsencrypt/renewal/mpsconline.mizoram.gov.in.conf
[root@server renewal]# cat mpsconline.mizoram.gov.in.conf

# renew_before_expiry = 30 days
version = 1.7.0
archive_dir = /etc/letsencrypt/archive/mpsconline.mizoram.gov.in
cert = /etc/letsencrypt/live/mpsconline.mizoram.gov.in/cert.pem
privkey = /etc/letsencrypt/live/mpsconline.mizoram.gov.in/privkey.pem
chain = /etc/letsencrypt/live/mpsconline.mizoram.gov.in/chain.pem
fullchain = /etc/letsencrypt/live/mpsconline.mizoram.gov.in/fullchain.pem

# Options used in the renewal process
[renewalparams]
authenticator = webroot
account = c6796c488c0c3a3e639ef3344a27579122222
webroot_path = /site/mpsconline/public,
server = https://acme-v02.api.letsencrypt.org/directory
[[webroot_map]]
mpsconline.mizoram.gov.in = /site/mpsconline/public

I hope the 7Day window will work atleast as it is my last resort. This is our workflow since the last 7yrs, it could be just that this past 14days we might have run the limit accidentally.

All our API's are mapped with https and its seems we are doom for the next 7days :bomb: .

Thanks for the HELP.
Cheers!

2 Likes
#8

Well, you should try to find the reason certbot-auto is not saving the files in archive dir because that is really strange.

I would follow these steps:

1.- Try to find the certbot logs and check whether you see some errors the days the certificate was issued, the logs should be in /var/log/letsencrypt/ dir.

2.- All your certificates (except yesterday) were issued between 17:42 and 17:46 UTC so seems you have a cron job launching certbot-auto to renew the certificates, I would check that too.

3.- If you can't find any reason for this strange behavior in above steps, backup your /etc/letsencrypt/ dir and delete the certificate to start fresh.

To backup you can use this for example:

tar pzcvf /root/etc-letsencrypt-backup_2021-01-30.tar.gz /etc/letsencrypt

and to delete the certificate

certbot-auto delete --cert-name mpsconline.mizoram.gov.in

Check with certbot-auto certificates and/or find /etc/letsencrypt -iname "*mpsconline.mizoram.gov.in*" that there is nothing related to this domain in letsencrypt dir.

Note: I'm saying to delete the certificate because seems none of your services are using them directly (you copy all the stuff directly to cPanel).

Now you have 2 options.

1.- Wait till next window where you will be able to issue a new cert for your domain, issue the cert and pray for it to work.

Note: The next time you should be able to issue a new cert is today 2021-Jan-30 at aprox. 18:43 UTC.

2.- Try to issue a new cert now but using staging instead of production (the parameter in certbot-auto is --staging) to check that the process is working fine, indeed I would issue a couple of certificates to check that all issued certs are being saved to archive dir and symlinks in live dir are updated correctly. Once it is working fine, I would delete again the certificate and wait till the window to issue the productive cert.

Good luck,
sahsanu

4 Likes
#9

@sahsanu, @Lalrinfela

Normally, if I had to take a wild guess, I'd say the cron job has --force-renewal in the certbot-auto command. However, if the new certificates are not being saved to the archive folder, the standard renew (without --force-renewal) will just keep on believing that the certificate is near expiration and thus just keep renewing it. Since the standard certbot cron job is set to run once per day, the rate limit is being hit here under the natural setup. Maybe a permission problem with writing to the archive directory? Is certbot-auto being run as root or someone else?

3 Likes
#10

Is certbot-auto being run as root or someone else?

After examining our WHM Cpanel, i see that Auto SSL Manager is enabled at some point when we update our CPanel. Because of which I suppose auto renewal was run by an automated system likely cron. I have disable the LetsEncrypt Auto SSL Renew provided by WHM and instead will just do it manually which suits more to our workflow and need.

BTW, I have been postponing converting our Government SSL to wildcard system for a long time now, maybe i should actually do it. i.e *.mizoram.gov.in. If i were to go ahead with this what would be the ideal flow, we already have many live domains.

My Action Plan is:

  1. Disable Auto SSL (Which i already did about 5hrs ago)
  2. Wait for 24hrs (Just to be safe, anyway its Sunday :grinning: )
  3. Then run renew command : certbot-auto renew --cert-name mpsconline.mizoram.gov.in

PS: I am not entirely sure about the renew command, if you could confirm, it will be very much appreciated.

Thanks Everyone!!

3 Likes
#11

That renewal command looks fine given that the cert-name is correct.

4 Likes
#12

For the sake of simplicity and preventing mistakes (and if you're doing things manually anyhow), the wildcard route could be very appropriate. You need to consider the management of the private key though. If you have servers spread out over different facilities, this could prove complicated. You might need to issue several wildcard certificates for the same apex (with different private keys) just to avoid needing to securely transport private keys.

The apex here would be: mizoram.gov.in

2 Likes
#13

You need to consider the management of the private key though.

If its a wild card system, will i just need to manage the root certificate (*.mizoram.gov.in) or do i still need to manage for each sub domain also (should have probably read the documentation, but i'll take my chances here :grinning: ).

If you have servers spread out over different facilities, this could prove complicated. You might need to issue several wildcard certificates for the same apex (with different private keys) just to avoid needing to securely transport private keys.

Our Data Centre is just one dedicated large cloud server, so this might make it little simple.

But what trouble me is all the active Certificates are having separate validity. Should we all remove them manually, then apply the root domain keys... How will this go (suggestions will be very much appreciated)?

Thanks!

2 Likes
#14

Basically, a certificate covering *.mizoram.gov.in could be substituted anywhere a certificate covering an immediate subdomain of mizoram.gov.in is used. The crucial part here is that a private key would be shared amongst all such servers using the same certificate. You can have as many valid certificates covering *.mizoram.gov.in at a time as you want (and are able to get issued due to rate limits because they're all duplicates of each other even with different private keys). These duplicates would allow you to split the covered subdomains into groups if necessary while still all being covered under the wildcard model.

In essence, you can have more than one certificate for *.mizoram.gov.in if necessary in order to use different private keys for different groups of subdomains.

When acquiring any of these certificates, it helps to have a central server acquire them and then distribute them (and their private keys) to the other servers. You could even designate the task to one server in each group so that each group has a central server. You'll need to be using dns-01 challenges (TXT records) for the wildcard certificate(s) regardless.

Keep in mind that a certificate covering *.mizoram.gov.in does not cover mizoram.gov.in itself, so that will need to either be covered under its own certificate or under one covering both mizoram.gov.in and *.mizoram.gov.in.

3 Likes
#15

Very helpful guide. Appreciated.

So, from what i understand, since we have many live domains, i think my process flow would be something like this:

  1. First, read about obtaining wildcard certificate, and verify if generating one will have any adverse effect on the live certificate. If there will be no side effect, then proceed to obtaining *.mizoram.gov.in certificates

  2. Then next time when deploying new domain / application, start using the wildcard. Then maybe wait for sometime.

  3. If this works, then convert all existing solo certificate to wildcard.

3 Likes
#16

If you want to be ultra-cautious that's a safe plan.

  • There is no affect (and thus no risk) on any live certificate of acquiring another certificate, even for the same SANs (domain names) on the live certificate.
  • You can simply acquire a certificate for *.mizoram.gov.in and immediately install it (and its private key) for a.mizoram.gov.in, b.mizoram.gov.in, and/or whatever.mizoram.gov.in.
  • Rather than having separate certificates, each of these will now just have a copy of the same certificate (and private key). You just acquire what you need and make copies as necessary.
  • Sharing a private key also means sharing potential compromise, so if you need to issue different *.mizoram.gov.in certificates for "higher-sensitivity" servers under different private keys that's something to consider.
  • Obtaining a wildcard certificate using a dns-01 challenge for *.mizoram.gov.in is exactly the same as obtaining a certificate for mizoram.gov.in itself. The TXT records for both have host _acme-challenge.mizoram.gov.in.
3 Likes
#17

As @griffin said, that command looks fine, maybe you could want to use --dry-run to simulate the process and to be sure the command will work.

certbot-auto renew --cert-name mpsconline.mizoram.gov.in --dry-run

Once tested you can remove --dry-run and get your cert:

certbot-auto renew --cert-name mpsconline.mizoram.gov.in

Good luck,
sahsanu

4 Likes
#18

Why do these symlinks have an Aug 31 date?

If certbot is unable to update that link, it might not know that it has and then will try to renew the cert again (twice a day).

I would look into why those files are not being updated.

The LE logs might show something, try adding some verbosity while doing a staging test:
certbot-auto renew --cert-name mpsconline.mizoram.gov.in --dry-run -vv

3 Likes
#19

Hi Guys,

I really appreciate all your support and help. I have just renewed my certificate and it is now reflected even in the 'archive' folder as well.

I will be working on wildcard migration in the coming weeks.

Thank you so much guys!!

4 Likes
#20

Perfect! I'm glad you finally got your cert :wink:

4 Likes