Certificate renewal is failing with DNS Challenge

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: cloud.siegert.nl

I ran this command: acme.sh --renew -d cloud.siegert.nl --force

It produced this output:
Create CSR error.
Please check log file for more details: /root/.acme.sh/acme.sh.log

My web server is (include version): nginx/1.20.1

The operating system my web server runs on is (include version): TrueNAS-12.0-U6.1

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): acme.sh v3.0.2

Last entries in acme.sh.log

[Sat Feb 12 12:35:51 CET 2022] Running cmd: renew
[Sat Feb 12 12:35:51 CET 2022] Using config home:/root/.acme.sh
[Sat Feb 12 12:35:51 CET 2022] default_acme_server
[Sat Feb 12 12:35:51 CET 2022] ACME_DIRECTORY='https://acme.zerossl.com/v2/DV90'
[Sat Feb 12 12:35:51 CET 2022] DOMAIN_PATH='/root/.acme.sh/cloud.siegert.nl'
[Sat Feb 12 12:35:51 CET 2022] Renew: 'cloud.siegert.nl'
[Sat Feb 12 12:35:51 CET 2022] Le_API='https://acme-v02.api.letsencrypt.org/directory'
[Sat Feb 12 12:35:51 CET 2022] Using config home:/root/.acme.sh
[Sat Feb 12 12:35:51 CET 2022] ACME_DIRECTORY='https://acme-v02.api.letsencrypt.org/directory'
[Sat Feb 12 12:35:51 CET 2022] _main_domain='cloud.siegert.nl'
[Sat Feb 12 12:35:51 CET 2022] _alt_domains='no'
[Sat Feb 12 12:35:51 CET 2022] Le_NextRenewTime='1647731792'
[Sat Feb 12 12:35:51 CET 2022] Using ACME_DIRECTORY: https://acme-v02.api.letsencrypt.org/directory
[Sat Feb 12 12:35:51 CET 2022] _init api for server: https://acme-v02.api.letsencrypt.org/directory
[Sat Feb 12 12:35:51 CET 2022] Retrying GET
[Sat Feb 12 12:35:51 CET 2022] GET
[Sat Feb 12 12:35:51 CET 2022] url='https://acme-v02.api.letsencrypt.org/directory'
[Sat Feb 12 12:35:51 CET 2022] timeout=
[Sat Feb 12 12:35:51 CET 2022] displayError='1'
[Sat Feb 12 12:35:51 CET 2022] _CURL='curl --silent --dump-header /root/.acme.sh/http.header  -L '
[Sat Feb 12 12:35:52 CET 2022] ret='0'
[Sat Feb 12 12:35:52 CET 2022] _hcode='0'
[Sat Feb 12 12:35:52 CET 2022] ACME_KEY_CHANGE='https://acme-v02.api.letsencrypt.org/acme/key-change'
[Sat Feb 12 12:35:52 CET 2022] ACME_NEW_AUTHZ
[Sat Feb 12 12:35:52 CET 2022] ACME_NEW_ORDER='https://acme-v02.api.letsencrypt.org/acme/new-order'
[Sat Feb 12 12:35:52 CET 2022] ACME_NEW_ACCOUNT='https://acme-v02.api.letsencrypt.org/acme/new-acct'
[Sat Feb 12 12:35:52 CET 2022] ACME_REVOKE_CERT='https://acme-v02.api.letsencrypt.org/acme/revoke-cert'
[Sat Feb 12 12:35:52 CET 2022] ACME_AGREEMENT='https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf'
[Sat Feb 12 12:35:52 CET 2022] ACME_NEW_NONCE='https://acme-v02.api.letsencrypt.org/acme/new-nonce'
[Sat Feb 12 12:35:52 CET 2022] Using CA: https://acme-v02.api.letsencrypt.org/directory
[Sat Feb 12 12:35:52 CET 2022] _on_before_issue
[Sat Feb 12 12:35:52 CET 2022] _chk_main_domain='cloud.siegert.nl'
[Sat Feb 12 12:35:52 CET 2022] _chk_alt_domains
[Sat Feb 12 12:35:52 CET 2022] Le_LocalAddress
[Sat Feb 12 12:35:52 CET 2022] d='cloud.siegert.nl'
[Sat Feb 12 12:35:52 CET 2022] Check for domain='cloud.siegert.nl'
[Sat Feb 12 12:35:52 CET 2022] _currentRoot='dns_nederhost'
[Sat Feb 12 12:35:52 CET 2022] d
[Sat Feb 12 12:35:52 CET 2022] _saved_account_key_hash is not changed, skip register account.
[Sat Feb 12 12:35:52 CET 2022] Read key length:
[Sat Feb 12 12:35:52 CET 2022] _createcsr
[Sat Feb 12 12:35:52 CET 2022] Single domain='cloud.siegert.nl'
[Sat Feb 12 12:35:52 CET 2022] Create CSR error.
[Sat Feb 12 12:35:52 CET 2022] pid
[Sat Feb 12 12:35:52 CET 2022] No need to restore nginx, skip.
[Sat Feb 12 12:35:52 CET 2022] _clearupdns
[Sat Feb 12 12:35:52 CET 2022] dns_entries
[Sat Feb 12 12:35:52 CET 2022] skip dns.
[Sat Feb 12 12:35:52 CET 2022] _on_issue_err
[Sat Feb 12 12:35:52 CET 2022] Please check log file for more details: /root/.acme.sh/acme.sh.log

All keys seems to be empty after renewal
drwxr-xr-x 2 root wheel 5 Nov 20 21:54 backup/
-rw-r--r-- 1 root wheel 3751 Jan 20 00:16 ca.cer
-rw-r--r-- 1 root wheel 0 Feb 12 12:31 cloud.siegert.nl.cer
-rw-r--r-- 1 root wheel 956 Feb 12 12:35 cloud.siegert.nl.conf
-rw-r--r-- 1 root wheel 964 Jan 20 00:16 cloud.siegert.nl.csr
-rw-r--r-- 1 root wheel 151 Feb 12 12:35 cloud.siegert.nl.csr.conf
-rw-r--r-- 1 root wheel 0 Feb 12 12:31 cloud.siegert.nl.key
-rw-r--r-- 1 root wheel 0 Feb 12 12:31 fullchain.cer

Restarting ngin fails:
Performing sanity check on nginx configuration:
nginx: [emerg] cannot load certificate "/root/.acme.sh/cloud.siegert.nl/fullchain.cer": PEM_read_bio_X509_AUX() failed (SSL: error:0909006C:PEM routines:get_name:no start line:Expecting: TRUSTED CERTIFICATE)
nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed

2

Don't use this. No good comes out of that.

And?

1 Like
3

It seems something is wrong with the contens of your acme.sh directory. What's the contents of the directory /root/.acme.sh/cloud.siegert.nl/ ?

Yeah, people seem to think all errors magically go away when adding --force :roll_eyes:

Also, when looking at your certificate history, it shows a certificate issued on 19 Januari:

That cert isn't even a single month old.. Why are you renewing already?

3 Likes
4

Hi @9peppe,
That was the last line of the log file.
Last relevant entries of the log are above this line.

5

Content of /root/.acme.sh/cloud.siegert.nl is in my post, but for your convenience here it is:
root@cloud:~/.acme.sh/cloud.siegert.nl # ll
total 20
drwxr-xr-x 2 root wheel 5 Nov 20 21:54 backup/
-rw-r--r-- 1 root wheel 3751 Jan 20 00:16 ca.cer
-rw-r--r-- 1 root wheel 0 Feb 12 12:31 cloud.siegert.nl.cer
-rw-r--r-- 1 root wheel 956 Feb 12 12:35 cloud.siegert.nl.conf
-rw-r--r-- 1 root wheel 964 Jan 20 00:16 cloud.siegert.nl.csr
-rw-r--r-- 1 root wheel 151 Feb 12 12:35 cloud.siegert.nl.csr.conf
-rw-r--r-- 1 root wheel 0 Feb 12 12:31 cloud.siegert.nl.key
-rw-r--r-- 1 root wheel 0 Feb 12 12:31 fullchain.cer

6

The certificate is renewed on schedule, nothing wrong so far.
But the certificate entries in ~/.acme.sh/cloud.siegert.nl are all empty.
Due to the error the fullchain.cer nginx will not restart, so after renewal the site is still issuing the old cert.

7

I did not realise, nor was it stated, that it was the contents of that specific directory.

The scheduled renewal task might be fine, but acme.sh should not try to renew already. Renewal isn't due for another month, if you use the "renew after 60 days" advisement of Let's Encrypt.

That said, it seems your acme.sh files are broken indeed. What's the content of the /backup/ dir?

5 Likes
8

Hi Osiris,
Thank you for your time!
The certificate was first issued on the 21st of November 2021. The renewal took place at 20th of January. All seemed good, but the server kept using the old (still active) certificate. So according to acme I have a valid certificate, but the server is not using it.
That is why I tried a forced renewal.

The content of the backup directory is:
-rw-r--r-- 1 root wheel 0 Feb 12 15:11 cert.bak
-rw-r--r-- 1 root wheel 0 Feb 12 15:11 fullchain.bak
-rw-r--r-- 1 root wheel 0 Feb 12 15:11 key.bak

All empty files.

9

Strange, the date of all those empty files is today.. Just over an hour ago.

Seems acme.sh is destroying your files? Assuming you didn't destroy them yourself :stuck_out_tongue:

Also, why would the contents of the backup directory have a timestamp of 15:11 while the contents of the "regular" directory has timestamps of 12:31? Did you make another attempt which failed again?

Further more, I have to say I don't have any experience with acme.sh and I don't like the client at all (it's lacking documentation, it's lacking proper verbose output/logging and it has been bought by a commercial CA). So for me personally I wouldn't even know where to start debugging this strange behaviour.. Maybe it might be better to just start over? Or switch to a different ACME client, might also be a good idea.. :roll_eyes: Although you're probably quite limited on a TrueNAS.

3 Likes
10

You're completely right!
I've tried renewal several times today, so that explains the strange timestamps.
Didn't know acme was sold to a commercial CA. Hate that.
TrueNAS isn't limiting me on a ACME client, so I can always use a different one.
Do you have any recommendations for an other client?

1 Like
11

Personally, I just use Certbot, but that's because I'm running it on a regular host with enough CPU power and it's written in Python, so I can easily make modifications if I require to do so.

However, Certbot is, I think, quite a "heavy" client. It might have features you don't require while still loading all the Python modules for them. And perhaps your TrueNAS' hardware doesn't like that.

There are of course other clients to consider, see ACME Client Implementations - Let's Encrypt

4 Likes
12

CPU/MEM is not the problem.
I noticed, in your link that the second best option for a client is acme.

I'm a bit hardheaded, I just want to make acme work.

As far as I can see, all is working like it should, only when renewing the certificate, according to the acme.log all certs are downloaded and saved but in reality all certs are empty.

I have stopped nginx, and tried again to renew, but again all certs are empty.

Anyone any idea what is going wrong?

13

What is the output of the

df -h ~/.acme.sh/cloud.siegert.nl

command?

2 Likes
14

Hi bruncsak,

root@cloud:~/.acme.sh # df -h ~/.acme.sh/cloud.siegert.nl
Filesystem Size Used Avail Capacity Mounted on
Pool1/iocage/jails/cloud/root 5.6T 5.9G 5.6T 0% /

15

The list isn't ranked by preference/endorsement of any kind. Only the part at the top about Certbot.

2 Likes
16

It's no negative comment Osiris.
I only noticed there are not that many options.
Acme is very light, but as you said very little documentation, and log file is not very informative.
At least in my case the log does not reveal anything.

17

I think you mean acme.sh? Because "ACME" is the name of the protocol and there are many other clients with "acme" in the name.

Such as acme-tiny (GitHub - diafygi/acme-tiny: A tiny script to issue and renew TLS certs from Let's Encrypt) which is a lightweight, single file Python application. And, if required, a fork which uses the dns-01 challenge instead called acme-dns-tiny (https://acme-dns-tiny.adorsaz.ch/). Not sure if it also supports your DNS provider though.

And there are other Python clients listed :slight_smile: I doooo love Python :stuck_out_tongue: Although acme.sh probably has the most DNS plugins available.. Which are open source and could be used elsewhere too of course :stuck_out_tongue: Or ported.

2 Likes
18

Yeah, you're right. I was a bit sluggish with the acme,sh name, sorry for that.

But my problem is still the same. acme.sh worked for me in the past, after rebuilding my server, and set-up encryption, acme.sh gave up on me.
And I don't understand. According the acme.sh.log the certs were stored. But all cert files are empty.
I can restore the certs from an earlier backup, and everything works.

As far as I can see, all access rights are in order, so why are the cert files not written to disk?

1 Like
19

I dunno, perhaps it's an idea to file an issue on the acme.sh github repository?

3 Likes
20

Thanks Osiris,
I'll try that!

1 Like