Issue in master server for wildcard certificate!

Help
1

Hi...
I have an issue in my www.mydomain.com server (Ubuntu 20.04) when i want to renew my cert with this command:

certbot certonly --manual -d mydomain.com -d *.mydomain.com --agree-tos --no-bootstrap --manual-public-ip-logging-ok --preferred-challenges dns-01 --server https://acme-v02.api.letsencrypt.org/directory

I copy two _acme02 text in my dns server and when is complete I have errors...
In /var/log/letsencrypt/letsencrypt.log:

2021-04-17 00:01:04,677:DEBUG:acme.client:Received response:
HTTP 429
Server: nginx
Date: Fri, 16 Apr 2021 19:31:04 GMT
Content-Type: application/problem+json
Content-Length: 256
Connection: keep-alive
Boulder-Requester: 99653528
Cache-Control: public, max-age=0, no-cache
Link: https://acme-v02.api.letsencrypt.org/directory;rel="index"
Replay-Nonce: 0104cBjRnd2c-xp38tl2e6eiPAG5eskpxUsaHX4XaAn1vZc

{
"type": "urn:ietf:params:acme:error:rateLimited",
"detail": "Error creating new order :: too many certificates already issued for exact set of domains: *.mydomain.com,mydomain.com: see Rate Limits - Let's Encrypt",
"status": 429
}
2021-04-17 00:01:04,678:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
File "/usr/bin/certbot", line 11, in
load_entry_point('certbot==0.40.0', 'console_scripts', 'certbot')()
File "/usr/lib/python3/dist-packages/certbot/main.py", line 1382, in main
return config.func(config, plugins)
File "/usr/lib/python3/dist-packages/certbot/main.py", line 1265, in certonly
lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
File "/usr/lib/python3/dist-packages/certbot/main.py", line 116, in _get_and_save_cert
renewal.renew_cert(config, domains, le_client, lineage)
File "/usr/lib/python3/dist-packages/certbot/renewal.py", line 320, in renew_cert
new_cert, new_chain, new_key, _ = le_client.obtain_certificate(domains, new_key)
File "/usr/lib/python3/dist-packages/certbot/client.py", line 348, in obtain_certificate
orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
File "/usr/lib/python3/dist-packages/certbot/client.py", line 381, in _get_order_and_authorizations
orderr = self.acme.new_order(csr_pem)
File "/usr/lib/python3/dist-packages/acme/client.py", line 863, in new_order
return self.client.new_order(csr_pem)
File "/usr/lib/python3/dist-packages/acme/client.py", line 666, in new_order
response = self._post(self.directory['newOrder'], order)
File "/usr/lib/python3/dist-packages/acme/client.py", line 95, in _post
return self.net.post(*args, **kwargs)
File "/usr/lib/python3/dist-packages/acme/client.py", line 1171, in post
return self._post_once(*args, **kwargs)
File "/usr/lib/python3/dist-packages/acme/client.py", line 1184, in _post_once
response = self._check_response(response, content_type=content_type)
File "/usr/lib/python3/dist-packages/acme/client.py", line 1042, in _check_response
raise messages.Error.from_json(jobj)
acme.messages.Error: urn:ietf:params:acme:error:rateLimited :: There were too many requests of a given type :: Error creating new order :: too many certificates already issued for exact set of domains: *.mydomain.com,mydomain.com: see Rate Limits - Let's Encrypt
2021-04-17 00:01:04,716:ERROR:certbot.log:An unexpected error occurred:
2021-04-17 00:01:04,717:ERROR:certbot.log:There were too many requests of a given type :: Error creating new order :: too many certificates already issued for exact set of domains: *.mydomain.com,mydomain.com: see Rate Limits - Let's Encrypt
2021-04-17 00:08:01,416:DEBUG:certbot.main:certbot version: 0.40.0
2021-04-17 00:08:01,416:DEBUG:certbot.main:Arguments:
2021-04-17 00:08:01,417:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#manual,PluginEntryPoint#nginx,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2021-04-17 00:08:01,443:DEBUG:certbot.log:Root logging level set at 20
2021-04-17 00:08:01,443:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2021-04-17 00:08:01,447:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/certbot/storage.py", line 211, in get_link_target
target = os.readlink(link)
FileNotFoundError: [Errno 2] No such file or directory: '/etc/letsencrypt/live/mydomain.com-0001/cert.pem'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/bin/certbot", line 11, in
load_entry_point('certbot==0.40.0', 'console_scripts', 'certbot')()
File "/usr/lib/python3/dist-packages/certbot/main.py", line 1382, in main
return config.func(config, plugins)
File "/usr/lib/python3/dist-packages/certbot/main.py", line 1000, in update_symlinks
cert_manager.update_live_symlinks(config)
File "/usr/lib/python3/dist-packages/certbot/cert_manager.py", line 40, in update_live_symlinks
storage.RenewableCert(renewal_file, config, update_symlinks=True)
File "/usr/lib/python3/dist-packages/certbot/storage.py", line 464, in init
self._update_symlinks()
File "/usr/lib/python3/dist-packages/certbot/storage.py", line 533, in _update_symlinks
previous_link = get_link_target(link)
File "/usr/lib/python3/dist-packages/certbot/storage.py", line 213, in get_link_target
raise errors.CertStorageError(
certbot.errors.CertStorageError: Expected /etc/letsencrypt/live/mydomain.com-0001/cert.pem to be a symlink

what should I do?
in this case my another subdomains renew certs successfully!!

1 Like
2

You've successfully created five identical certs within the past week--why not use one of them?

3 Likes
3

thank you for your reply...
It came with an error every time I received it...
Of course, it did not give an error once, but the certificate was not applied, and the next time it asked to delete the previous file, which I had to delete ...
how and where can i see my last certificate...!?

1 Like
4

I'm guessing you misinterpreted this, as that's highly unusual. An ACME client wouldn't ask you to delete a perfectly fine cert to install a cert.

1 Like
5

You are right...
Apparently so ... because it said that the cert2.pem file is available and gave an error ...
Sorry, I deleted the file and tried again!
In our other subdomain servers certificates are ok and updated...
But in master server it has error! What should I do now?

1 Like
6

Hi @asr

please start reading

Doing things wrong again and again -> you have a lot of time to read before doing things wrong again.

1 Like
7

Dear friend;
Thank you for your special guide! You left me alone in the ocean...
I use letsencrypt certs for about 4 years... but new in wildcard...
I will definitely follow your advice and read it again more carefully. But in this particular and careless case that I did, what is your suggestion for restoring the latest released certificate and restoring the site to optimal condition?

1 Like
8

You have deleted the certificates, so you can't.

Rule no 0 working with computers: Never delete things if you don't know if you need these things again.

1 Like
9

I have my last cert1.pem and others...
I mean, I can't do anything right now? Can't I replace or rebuild the certificate?

1 Like
10

What does certbot certificates say?

2 Likes
11

$>certbot certificates

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Renewal configuration file /etc/letsencrypt/renewal/mydomain.com-0001.conf produced an unexpected error: expected /etc/letsencrypt/live/mydomain.com-0001/cert.pem to be a symlink. Skipping.

Found the following certs:
Certificate Name: mydomain.com
Domains: *.mydomain.com
Expiry Date: 2021-04-16 16:07:50+00:00 (INVALID: EXPIRED)
Certificate Path: /etc/letsencrypt/live/mydomain.com/fullchain.pem
Private Key Path: /etc/letsencrypt/live/mydomain.com/privkey.pem

The following renewal configurations were invalid:
/etc/letsencrypt/renewal/mydomain.com-0001.conf

12

Looks like you've manually deleted stuff from the /etc/letsencrypt directory, which is always wrong.

That said, it might be possible to reconstruct the /live/ subdirectory if the contents of /etc/letsencrypt/archive/yourdomain.com-0001/ are still present. The symbolic links in /live/ are as the following example:

server ~ # cd /etc/letsencrypt/live/example.com/
server example.com # ls -l 
total 0
lrwxrwxrwx 1 root root 42 Mar 30 20:07 cert.pem -> ../../archive/example.com/cert27.pem
lrwxrwxrwx 1 root root 43 Mar 30 20:07 chain.pem -> ../../archive/example.com/chain27.pem
lrwxrwxrwx 1 root root 47 Mar 30 20:07 fullchain.pem -> ../../archive/example.com/fullchain27.pem
lrwxrwxrwx 1 root root 45 Mar 30 20:07 privkey.pem -> ../../archive/example.com/privkey27.pem
server example.com #

.. where the "27" in the destination of the symbolic links are the latest files in the /archive/example.com/ directory.

2 Likes
13

Thank you for your reply...
I check that address last night and it is ok!
This is my links, what 's wrong?

~$ sudo ls -la /etc/letsencrypt/live/mydomain.com
total 12
drwxr-xr-x 2 root root 4096 Apr 18 00:59 .
drwx------ 4 root root 4096 Jan 16 20:50 ..
-rw-r--r-- 1 root root 692 Jan 16 20:37 README
lrwxrwxrwx 1 root root 54 Apr 18 00:58 cert.pem -> /etc/letsencrypt/archive/mydomain.com-0001/cert1.pem
lrwxrwxrwx 1 root root 55 Apr 18 00:58 chain.pem -> /etc/letsencrypt/archive/mydomain.com-0001/chain1.pem
lrwxrwxrwx 1 root root 59 Apr 18 00:59 fullchain.pem -> /etc/letsencrypt/archive/mydomain.com-0001/fullchain1.pem
lrwxrwxrwx 1 root root 57 Apr 18 00:59 privkey.pem -> /etc/letsencrypt/archive/mydomain.com-0001/privkey1.pem

and This is my /etc/letsencrypt/renewal/mydomain.com.conf file content:

renew_before_expiry = 30 days

version = 0.40.0
archive_dir = /etc/letsencrypt/archive/mydomain.com
cert = /etc/letsencrypt/live/mydomain.com/cert.pem
privkey = /etc/letsencrypt/live/mydomain.com/privkey.pem
chain = /etc/letsencrypt/live/mydomain.com/chain.pem
fullchain = /etc/letsencrypt/live/mydomain.com/fullchain.pem

Options used in the renewal process

[renewalparams]
account = e9f37bd707dc48a96f9776ce634b6871
pref_challs = dns-01,
server = https://acme-v02.api.letsencrypt.org/directory
authenticator = manual
manual_public_ip_logging_ok = True

14

Plenty, if this are the actual directories.

For example:

Is that really the mydomain.com directory or did you perhaps by any chance remove the -0001 from the end?

Because if it is the directory without the -0001 suffix, why are:

.. those symbolic links pointing to the -0001 suffixed archive directory?

Also, those are symbolic links to absolute paths. While not wrong from the symbolic link point of view, this is not done by certbot: certbot only makes relative paths, i.e., using ../../archive/. This tells me those symbolic links were manually created.

It seems you've managed to professionally disrupt your /etc/letsencrypt/live/ directories by manually manipulating it, which should not have been done.

You should correct the symbolic links to their proper destinations, i.e., -0001 should point to -0001 and directories/symlinks without -0001 should point to files/directories without -0001.

1 Like
15

Thank you for your patience and support

This is my real link:

~$ sudo ls -la /etc/letsencrypt/archive/mydomain.com-0001
total 24
drwxr-xr-x 2 root root 4096 Jan 16 20:37 .
drwx------ 4 root root 4096 Jan 16 20:37 ..
-rw-r--r-- 1 root root 1846 Jan 16 20:37 cert1.pem
-rw-r--r-- 1 root root 1586 Jan 16 20:37 chain1.pem
-rw-r--r-- 1 root root 3432 Jan 16 20:37 fullchain1.pem
-rw------- 1 root root 1704 Jan 16 20:37 privkey1.pem

and I know the hierarchy of letsencrypt directories and symbolic links it made...
I saw that mydomain.com-0001 directory made by certbot after last successful renewal!
last night I made new symbolic link manually to another last certs... and that was not work!
Then I recreate those absolute links.
Fortunately, the -0001 folder is available and has not been deleted, but there is no new certificate in it!
I should change the absolute links to relative now...
These are my real links and directories:

~$ sudo ls -la /etc/letsencrypt/archive/mydomain.com
total 40
drwxr-xr-x 2 root root 4096 Apr 18 00:38 .
drwx------ 4 root root 4096 Jan 16 20:37 ..
-rw-r--r-- 1 root root 1931 Oct 18 19:15 cert1.pem
-rw-r--r-- 1 root root 1866 Apr 16 23:26 cert2.pem
-rw-r--r-- 1 root root 1647 Oct 18 19:15 chain1.pem
-rw-r--r-- 1 root root 1586 Apr 16 23:26 chain2.pem

~$ sudo ls -la /etc/letsencrypt/live
total 20
drwx------ 4 root root 4096 Jan 16 20:50 .
drwxr-xr-x 9 root root 4096 Apr 18 13:06 ..
-rw-r--r-- 1 root root 740 Oct 18 19:15 README
drwxr-xr-x 2 root root 4096 Apr 18 00:59 mydomain.com
drwxr-xr-x 2 root root 4096 Oct 18 19:15 mydomain.com-0000

~$ sudo ls -la /etc/letsencrypt/archive
total 16
drwx------ 4 root root 4096 Jan 16 20:37 .
drwxr-xr-x 9 root root 4096 Apr 18 13:06 ..
drwxr-xr-x 2 root root 4096 Apr 18 00:38 mydomain.com
drwxr-xr-x 2 root root 4096 Jan 16 20:37 mydomain.com-0001

1 Like
16

Looks pretty recent?

2 Likes
17

maybe...
Its manufacturing date is exactly the day of the end of our certification! But other files like fullchain2.pem and privkey2.pem are not in this folder!

1 Like
18

privkey2.pem could be still present in /etc/letsencrypt/keys/ and fullchain2.pem can be made manually by concatenation of cert2.pem and chain2.pem, where the chain isn't very specific for a single certificate.

2 Likes
20

Really, your /etc/letsencrypt directory is very badly broken, and your messing with the symlinks manually is in all likelihood only going to make it worse. I'd be strongly inclined to recommend blowing away the entire directory and starting over--except for the rate limit problem.

Now, since you haven't given us your domain (as the Help topic template says you're required to do in order to get help), none of us can tell you how long it will be before that rate limit expires. You can check yourself, though, on https://check-your-website.server-daten.de/ (and likely other places, but that's the one that comes to mind). IMO, if it expires within a day or two, I'd say tar up /etc/letsencrypt to have a backup, delete the entire directory, and recreate the certificate when the rate limit expires. Then figure out how to automate DNS validation, because doing it manually sucks.

1 Like
21

thanks for your response
Of course, now I prefer to solve the current problem until the work of our site is lightened without doing your suggestion ...
I will definitely do some feasibility studies after these busy days ...
Do you have any comments on the above conversation?

1 Like