Autorenewal failure nextcloud

My second site, which is a nextcloud snap server, is now failing to renew with the following message:

Renewal configuration file /etc/letsencrypt/renewal/cloud.ddoherty.net-0001.conf is broken.
The error was: expected /etc/letsencrypt/live/cloud.ddoherty.net-0001/cert.pem to be a symlink
Skipping.
Failed to renew certificate cloud.ddoherty.net with error: [Errno 17] File exists: '/etc/letsencrypt/archive/cloud.ddoherty.net/privkey2.pem'
All renewals failed. The following certificates could not be renewed:
  /etc/letsencrypt/live/cloud.ddoherty.net/fullchain.pem (failure)
1 renew failure(s), 1 parse failure(s)

It looks like a whole new error. Is there a simple fix?

That generally implies something hasn't gone to plan.

Let's see what we're dealing with:

  • certbot certificates
  • ls -l /etc/letsencrypt/live/cloud.ddoherty.net-0001/
3 Likes
# certbot certificates
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Found the following certs:
  Certificate Name: cloud.ddoherty.net
    Serial Number: 3f3d436f3676bb5fc6895b658f54b6ebcf1
    Key Type: ECDSA
    Domains: cloud.ddoherty.net
    Expiry Date: 2024-02-15 11:47:31+00:00 (VALID: 21 days)
    Certificate Path: /etc/letsencrypt/live/cloud.ddoherty.net/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/cloud.ddoherty.net/privkey.pem
  Certificate Name: s16.ddoherty.net
    Serial Number: 4f41f409063b079db44dce1898a85e671f5
    Key Type: RSA
    Domains: s16.ddoherty.net
    Expiry Date: 2024-04-22 22:00:40+00:00 (VALID: 89 days)
    Certificate Path: /etc/letsencrypt/live/s16.ddoherty.net/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/s16.ddoherty.net/privkey.pem

The following renewal configurations were invalid:
  /etc/letsencrypt/renewal/cloud.ddoherty.net-0001.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

# ls -l /etc/letsencrypt/live/cloud.ddoherty.net-0001/
ls: cannot access '/etc/letsencrypt/live/cloud.ddoherty.net-0001/': No such file or directory

-------

ls -l /etc/letsencrypt/live/cloud.ddoherty.net
total 4
lrwxrwxrwx 1 root root  47 Nov 17 06:47 cert.pem -> ../../archive/cloud.ddoherty.net-0001/cert1.pem
lrwxrwxrwx 1 root root  48 Nov 17 06:47 chain.pem -> ../../archive/cloud.ddoherty.net-0001/chain1.pem
lrwxrwxrwx 1 root root  52 Nov 17 06:47 fullchain.pem -> ../../archive/cloud.ddoherty.net-0001/fullchain1.pem
lrwxrwxrwx 1 root root  50 Nov 17 06:47 privkey.pem -> ../../archive/cloud.ddoherty.net-0001/privkey1.pem
-rw-r--r-- 1 root root 692 Nov 17 06:47 README

You might as well just delete this file:

[the simple fix is to:]
And start over.

What FQDN(s) do you need to get a cert for?

3 Likes

It seems to be working for s16.ddoherty.net, so I just need it for cloud.ddoherty.net.

For that one, exactly what do I do to "start over"?

Thanks, @rg305

1 Like

But you have a valid cert for that name...

It is ready to be renewed though...
What shows?:
certbot renew

2 Likes

There seems to be a slight TYPO in the redirection for cloud:

curl -Ii cloud.ddoherty.net/.well-known/acme-challenge/Test_File-1234
HTTP/1.1 301 Moved Permanently
Server: nginx/1.18.0 (Ubuntu)
Date: Thu, 25 Jan 2024 12:47:56 GMT
Content-Type: text/html
Content-Length: 178
Connection: keep-alive
Location: https://cloud.ddoherty.net:/.well-known/acme-challenge/Test_File-1234

Notice how s16 doesn't include that extra ":" at the end of the domain name:

curl -Ii s16.ddoherty.net/.well-known/acme-challenge/Test_File-1234
HTTP/1.1 301 Moved Permanently
Server: nginx/1.18.0 (Ubuntu)
Date: Thu, 25 Jan 2024 12:49:13 GMT
Content-Type: text/html
Content-Length: 178
Connection: keep-alive
Location: https://s16.ddoherty.net/.well-known/acme-challenge/Test_File-1234
2 Likes

certbot renew

certbot renew
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/cloud.ddoherty.net.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Renewing an existing certificate for cloud.ddoherty.net
Failed to renew certificate cloud.ddoherty.net with error: urn:ietf:params:acme:error:rateLimited :: There were too many requests of a given type :: Error creating new order :: too many certificates (5) already issued for this exact set of domains in the last 168 hours: cloud.ddoherty.net, retry after 2024-01-26T21:20:50Z: see https://letsencrypt.org/docs/duplicate-certificate-limit/

But in the log file:

2024-01-25 06:46:10,474:ERROR:certbot._internal.renewal:All renewals failed. The following certificates could not be renewed:
2024-01-25 06:46:10,474:ERROR:certbot._internal.renewal:  /etc/letsencrypt/live/cloud.ddoherty.net/fullchain.pem (failure)
2024-01-25 06:46:10,474:DEBUG:certbot._internal.display.obj:Notifying user: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2024-01-25 06:46:10,475:DEBUG:certbot._internal.log:Exiting abnormally:
Traceback (most recent call last):
  File "/snap/certbot/3566/bin/certbot", line 8, in <module>
    sys.exit(main())
  File "/snap/certbot/3566/lib/python3.8/site-packages/certbot/main.py", line 19, in main
    return internal_main.main(cli_args)
  File "/snap/certbot/3566/lib/python3.8/site-packages/certbot/_internal/main.py", line 1869, in main
    return config.func(config, plugins)
  File "/snap/certbot/3566/lib/python3.8/site-packages/certbot/_internal/main.py", line 1642, in renew
    renewed_domains, failed_domains = renewal.handle_renewal_request(config)
  File "/snap/certbot/3566/lib/python3.8/site-packages/certbot/_internal/renewal.py", line 568, in handle_renewal_request
    raise errors.Error(
certbot.errors.Error: 1 renew failure(s), 0 parse failure(s)
2024-01-25 06:46:10,475:ERROR:certbot._internal.log:1 renew failure(s), 0 parse failure(s)

Is this expected?

OK that is a problem.
It seems certbot is renewing but the cert isn't being saved to disk properly.

2 Likes

Let's have a look at these folders:
ls -l /etc/letsencrypt/live/cloud.ddoherty.net/
ls -l /etc/letsencrypt/archive/cloud.ddoherty.net/

2 Likes

Good catch. I think I fixed it in the nginx config. I mean the colon problem.

1 Like

Yes, the ":" is gone.

That said, I do see a 403 error on cloud when a 404 error is expected:

curl -Ii https://cloud.ddoherty.net/.well-known/acme-challenge/Test_File-1234
HTTP/2 403
server: nginx/1.18.0 (Ubuntu)
date: Thu, 25 Jan 2024 12:58:13 GMT
content-type: text/html; charset=iso-8859-1

Compare with:

curl -Ii https://s16.ddoherty.net/.well-known/acme-challenge/Test_File-1234
HTTP/2 404
server: nginx/1.18.0 (Ubuntu)
date: Thu, 25 Jan 2024 12:58:31 GMT
content-type: text/html
content-length: 162
1 Like
# ls -l /etc/letsencrypt/live/cloud.ddoherty.net/
total 4
lrwxrwxrwx 1 root root  47 Nov 17 06:47 cert.pem -> ../../archive/cloud.ddoherty.net-0001/cert1.pem
lrwxrwxrwx 1 root root  48 Nov 17 06:47 chain.pem -> ../../archive/cloud.ddoherty.net-0001/chain1.pem
lrwxrwxrwx 1 root root  52 Nov 17 06:47 fullchain.pem -> ../../archive/cloud.ddoherty.net-0001/fullchain1.pem
lrwxrwxrwx 1 root root  50 Nov 17 06:47 privkey.pem -> ../../archive/cloud.ddoherty.net-0001/privkey1.pem
-rw-r--r-- 1 root root 692 Nov 17 06:47 README
# ls -l /etc/letsencrypt/archive/cloud.ddoherty.net/
total 124
-rw-r--r-- 1 root root 1822 Jul 16  2023 cert10.pem
-rw-r--r-- 1 root root 1822 Sep 14 23:52 cert11.pem
-rw-r--r-- 1 root root 1822 Nov 14 06:50 cert12.pem
-rw-r--r-- 1 root root 1874 Jan 22  2023 cert7.pem
-rw-r--r-- 1 root root 1874 Mar 23  2023 cert8.pem
-rw-r--r-- 1 root root 1879 May 22  2023 cert9.pem
-rw-r--r-- 1 root root 3749 Jul 16  2023 chain10.pem
-rw-r--r-- 1 root root 3749 Sep 14 23:52 chain11.pem
-rw-r--r-- 1 root root 3749 Nov 14 06:50 chain12.pem
-rw-r--r-- 1 root root 3749 Jan 22  2023 chain7.pem
-rw-r--r-- 1 root root 3749 Mar 23  2023 chain8.pem
-rw-r--r-- 1 root root 3749 May 22  2023 chain9.pem
-rw-r--r-- 1 root root 5571 Jul 16  2023 fullchain10.pem
-rw-r--r-- 1 root root 5571 Sep 14 23:52 fullchain11.pem
-rw-r--r-- 1 root root 5571 Nov 14 06:50 fullchain12.pem
-rw-r--r-- 1 root root 5623 Jan 22  2023 fullchain7.pem
-rw-r--r-- 1 root root 5623 Mar 23  2023 fullchain8.pem
-rw-r--r-- 1 root root 5628 May 22  2023 fullchain9.pem
-rw-r--r-- 1 root root 1704 Jul 16  2023 privkey10.pem
-rw-r--r-- 1 root root 1704 Sep 14 23:52 privkey11.pem
-rw-r--r-- 1 root root 1704 Nov 14 06:50 privkey12.pem
-rw------- 1 root root 1704 Jan 23 21:48 privkey2.pem
-rw-r--r-- 1 root root 1704 Jan 22  2023 privkey7.pem
-rw-r--r-- 1 root root 1704 Mar 23  2023 privkey8.pem
-rw-r--r-- 1 root root 1704 May 22  2023 privkey9.pem

Those symlinks are completely WRONG.
They point to (now gone) -0001 folder.

We need to delete and recreate those symlinks (correctly).

2 Likes

Aha. I do that manually, right?

It can be done manually, yes.

I'm not too familiar with this way...
But you might try:

  • delete the symlinks
  • certbot reconfigure --cert-name cloud.ddoherty.net
1 Like

Should I link to the 10 versions, like cert10.pem, etc?

To the highest number = "12"
[twelve is higher than ten]
LOL

2 Likes
1 Like

Yes, of course! Need to brush up on my arithmetic!

Anyway, all done. Next?

1 Like