Problems with certbot renew

Well this gets stranger by the day. Port scanner shows 80 and 443 open. certbot renew --dry-run shows no problems. Only congratulations all succeeded.
People can connect to the forum, but logwatch is still showing this:

    certbot-renew.service: Failed with result 'exit-code'.: 2 Time(s)
    certbot-renew.service: Main process exited, code=exited, status=1/FAILURE: 2 Time(s)

What's ther new problem now? Am I going to get problems like this all the time? This is SO frustrating

1 Like

Are there any logs to go with that?
What is the certbot renewal command being run?

9 Likes

The renewal command is built into Alma 8.6. Where would I find it?
The only log I have is Logwatch
Which log should I be looking at?
Is there any way to improve the letsencrypt detail delivered by Logwatch?

I ran certbot renew

[root@alma-86 ~]# certbot renew
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/corp.networkingtechnology.org.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Failed to renew certificate corp.networkingtechnology.org with error: Account at /etc/letsencrypt/accounts/acme-v01.api.letsencrypt.org/directory/90eb7ed71db53fd117f72c0855591879 does not exist

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/support.corp.networkingtechnology.org.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Certificate not yet due for renewal

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/writers.corp.networkingtechnology.org.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Certificate not yet due for renewal

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
The following certificates are not due for renewal yet:
  /etc/letsencrypt/live/support.corp.networkingtechnology.org/fullchain.pem expires on 2022-12-24 (skipped)
  /etc/letsencrypt/live/writers.corp.networkingtechnology.org/fullchain.pem expires on 2022-11-19 (skipped)
All renewals failed. The following certificates could not be renewed:
  /etc/letsencrypt/live/corp.networkingtechnology.org/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
[root@alma-86 ~]#

Here is the complete logfile for 27/09/2022

2022-09-27 09:44:59,255:DEBUG:certbot._internal.main:certbot version: 1.22.0
2022-09-27 09:44:59,258:DEBUG:certbot._internal.main:Location of certbot entry point: /usr/bin/certbot
2022-09-27 09:44:59,258:DEBUG:certbot._internal.main:Arguments: []
2022-09-27 09:44:59,259:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2022-09-27 09:44:59,302:DEBUG:certbot._internal.log:Root logging level set at 30
2022-09-27 09:44:59,305:DEBUG:certbot._internal.display.obj:Notifying user: Processing /etc/letsencrypt/renewal/corp.networkingtechnology.org.conf
2022-09-27 09:44:59,348:DEBUG:certbot._internal.plugins.selection:Requested authenticator <certbot._internal.cli.cli_utils._Default object at 0x7f391d3a4c50> and installer <certbot._internal.cli.cli_utils._Default object at 0x7f391d3a4c50>
2022-09-27 09:44:59,391:DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): r3.o.lencr.org:80
2022-09-27 09:44:59,437:DEBUG:urllib3.connectionpool:http://r3.o.lencr.org:80 "POST / HTTP/1.1" 200 503
2022-09-27 09:44:59,439:DEBUG:certbot.ocsp:OCSP response for certificate /etc/letsencrypt/archive/corp.networkingtechnology.org/cert1.pem is signed by the certificate's issuer.
2022-09-27 09:44:59,445:DEBUG:certbot.ocsp:OCSP certificate status for /etc/letsencrypt/archive/corp.networkingtechnology.org/cert1.pem is: OCSPCertStatus.GOOD
2022-09-27 09:44:59,450:DEBUG:certbot._internal.storage:Should renew, less than 30 days before certificate expiry 2022-10-08 19:39:36 UTC.
2022-09-27 09:44:59,450:INFO:certbot._internal.renewal:Certificate is due for renewal, auto-renewing...
2022-09-27 09:44:59,450:DEBUG:certbot._internal.plugins.selection:Requested authenticator apache and installer apache
2022-09-27 09:44:59,596:DEBUG:certbot_apache._internal.configurator:Apache version is 2.4.37
2022-09-27 09:44:59,978:DEBUG:certbot._internal.plugins.selection:Single candidate plugin: * apache
Description: Apache Web Server plugin
Interfaces: Installer, Authenticator, Plugin
Entry point: apache = certbot_apache._internal.entrypoint:ENTRYPOINT
Initialized: <certbot_apache._internal.override_fedora.FedoraConfigurator object at 0x7f391d329f98>
Prep: True
2022-09-27 09:44:59,982:DEBUG:certbot._internal.plugins.selection:Single candidate plugin: * apache
Description: Apache Web Server plugin
Interfaces: Installer, Authenticator, Plugin
Entry point: apache = certbot_apache._internal.entrypoint:ENTRYPOINT
Initialized: <certbot_apache._internal.override_fedora.FedoraConfigurator object at 0x7f391d329f98>
Prep: True
2022-09-27 09:44:59,982:DEBUG:certbot._internal.plugins.selection:Selected authenticator <certbot_apache._internal.override_fedora.FedoraConfigurator object at 0x7f391d329f98> and installer <certbot_apache._internal.override_fedora.FedoraConfigurator object at 0x7f391d329f98>
2022-09-27 09:44:59,983:INFO:certbot._internal.plugins.selection:Plugins selected: Authenticator apache, Installer apache
2022-09-27 09:44:59,983:ERROR:certbot._internal.renewal:Failed to renew certificate corp.networkingtechnology.org with error: Account at /etc/letsencrypt/accounts/acme-v01.api.letsencrypt.org/directory/90eb7ed71db53fd117f72c0855591879 does not exist
2022-09-27 09:44:59,985:DEBUG:certbot._internal.renewal:Traceback was:
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/certbot/_internal/renewal.py", line 485, in handle_renewal_request
    main.renew_cert(lineage_config, plugins, renewal_candidate)
  File "/usr/lib/python3.6/site-packages/certbot/_internal/main.py", line 1439, in renew_cert
    le_client = _init_le_client(config, auth, installer)
  File "/usr/lib/python3.6/site-packages/certbot/_internal/main.py", line 788, in _init_le_client
    acc, acme = _determine_account(config)
  File "/usr/lib/python3.6/site-packages/certbot/_internal/main.py", line 688, in _determine_account
    acc = account_storage.load(config.account)
  File "/usr/lib/python3.6/site-packages/certbot/_internal/account.py", line 247, in load
    return self._load_for_server_path(account_id, self.config.server_path)
  File "/usr/lib/python3.6/site-packages/certbot/_internal/account.py", line 218, in _load_for_server_path
    prev_loaded_account = self._load_for_server_path(account_id, prev_server_path)
  File "/usr/lib/python3.6/site-packages/certbot/_internal/account.py", line 228, in _load_for_server_path
    "Account at %s does not exist" % account_dir_path)
certbot.errors.AccountNotFound: Account at /etc/letsencrypt/accounts/acme-v01.api.letsencrypt.org/directory/90eb7ed71db53fd117f72c0855591879 does not exist

2022-09-27 09:44:59,986:DEBUG:certbot._internal.display.obj:Notifying user: Processing /etc/letsencrypt/renewal/support.corp.networkingtechnology.org.conf
2022-09-27 09:45:00,029:DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): r3.o.lencr.org:80
2022-09-27 09:45:00,067:DEBUG:urllib3.connectionpool:http://r3.o.lencr.org:80 "POST / HTTP/1.1" 200 503
2022-09-27 09:45:00,070:DEBUG:certbot.ocsp:OCSP response for certificate /etc/letsencrypt/archive/support.corp.networkingtechnology.org/cert2.pem is signed by the certificate's issuer.
2022-09-27 09:45:00,071:DEBUG:certbot.ocsp:OCSP certificate status for /etc/letsencrypt/archive/support.corp.networkingtechnology.org/cert2.pem is: OCSPCertStatus.GOOD
2022-09-27 09:45:00,073:DEBUG:certbot._internal.display.obj:Notifying user: Certificate not yet due for renewal
2022-09-27 09:45:00,074:DEBUG:certbot._internal.plugins.selection:Requested authenticator apache and installer apache
2022-09-27 09:45:00,080:DEBUG:certbot._internal.plugins.selection:Selecting plugin: * apache
Description: Apache Web Server plugin
Interfaces: Installer, Authenticator, Plugin
Entry point: apache = certbot_apache._internal.entrypoint:ENTRYPOINT
Initialized: <certbot_apache._internal.override_fedora.FedoraConfigurator object at 0x7f391d3294e0>
2022-09-27 09:45:00,081:DEBUG:certbot.plugins.storage:Plugin storage file /etc/letsencrypt/.pluginstorage.json was empty, no values loaded
2022-09-27 09:45:00,081:DEBUG:certbot._internal.display.obj:Notifying user: Processing /etc/letsencrypt/renewal/writers.corp.networkingtechnology.org.conf
2022-09-27 09:45:00,120:DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): r3.o.lencr.org:80
2022-09-27 09:45:00,159:DEBUG:urllib3.connectionpool:http://r3.o.lencr.org:80 "POST / HTTP/1.1" 200 503
2022-09-27 09:45:00,161:DEBUG:certbot.ocsp:OCSP response for certificate /etc/letsencrypt/archive/writers.corp.networkingtechnology.org/cert1.pem is signed by the certificate's issuer.
2022-09-27 09:45:00,162:DEBUG:certbot.ocsp:OCSP certificate status for /etc/letsencrypt/archive/writers.corp.networkingtechnology.org/cert1.pem is: OCSPCertStatus.GOOD
2022-09-27 09:45:00,163:DEBUG:certbot._internal.display.obj:Notifying user: Certificate not yet due for renewal
2022-09-27 09:45:00,165:DEBUG:certbot._internal.plugins.selection:Requested authenticator apache and installer apache
2022-09-27 09:45:00,171:DEBUG:certbot._internal.plugins.selection:Selecting plugin: * apache
Description: Apache Web Server plugin
Interfaces: Installer, Authenticator, Plugin
Entry point: apache = certbot_apache._internal.entrypoint:ENTRYPOINT
Initialized: <certbot_apache._internal.override_fedora.FedoraConfigurator object at 0x7f391ccc2be0>
2022-09-27 09:45:00,172:DEBUG:certbot.plugins.storage:Plugin storage file /etc/letsencrypt/.pluginstorage.json was empty, no values loaded
2022-09-27 09:45:00,172:DEBUG:certbot._internal.display.obj:Notifying user: 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2022-09-27 09:45:00,173:DEBUG:certbot._internal.display.obj:Notifying user: The following certificates are not due for renewal yet:
2022-09-27 09:45:00,173:DEBUG:certbot._internal.display.obj:Notifying user:   /etc/letsencrypt/live/support.corp.networkingtechnology.org/fullchain.pem expires on 2022-12-24 (skipped)
  /etc/letsencrypt/live/writers.corp.networkingtechnology.org/fullchain.pem expires on 2022-11-19 (skipped)
2022-09-27 09:45:00,173:ERROR:certbot._internal.renewal:All renewals failed. The following certificates could not be renewed:
2022-09-27 09:45:00,173:ERROR:certbot._internal.renewal:  /etc/letsencrypt/live/corp.networkingtechnology.org/fullchain.pem (failure)
2022-09-27 09:45:00,174:DEBUG:certbot._internal.display.obj:Notifying user: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2022-09-27 09:45:00,174:DEBUG:certbot._internal.log:Exiting abnormally:
Traceback (most recent call last):
  File "/usr/bin/certbot", line 11, in <module>
    load_entry_point('certbot==1.22.0', 'console_scripts', 'certbot')()
  File "/usr/lib/python3.6/site-packages/certbot/main.py", line 19, in main
    return internal_main.main(cli_args)
  File "/usr/lib/python3.6/site-packages/certbot/_internal/main.py", line 1632, in main
    return config.func(config, plugins)
  File "/usr/lib/python3.6/site-packages/certbot/_internal/main.py", line 1518, in renew
    renewal.handle_renewal_request(config)
  File "/usr/lib/python3.6/site-packages/certbot/_internal/renewal.py", line 512, in handle_renewal_request
    len(renew_failures), len(parse_failures)))
certbot.errors.Error: 1 renew failure(s), 0 parse failure(s)
2022-09-27 09:45:00,175:ERROR:certbot._internal.log:1 renew failure(s), 0 parse failure(s)

The problem might be related to "acme-v01.api.letsencrypt.org" having been retired.

8 Likes

Great!!! So what do I DO about it? Reinstall everything or what?

Read these for start:

7 Likes

On Alma-86 certbot --version gives me 1,22.0 and that's where I get the error

On Hermes certbot --version give me 1.22.0 and this IS working

I only installed these certificates a month or so ago. So please explain:
(a) How I have an out of date version?
(b) Why both servers have the identical versions and were installed within 30 days of each other and one works and the other doesn't?

So upgrading is NOT going to solve the problem. Quo vadis?

(A) The out-of-date version applies to the account being used:

Account at /etc/letsencrypt/accounts/acme-v01.api.letsencrypt.org/directory/90eb7ed71db53fd117f72c0855591879 does not exist

[which is also not found/missing]

(B) Because one seems to have been previously used and not completely wiped before install/upgrade.

9 Likes

So where does that leave me? I still have no clue how to fix it. The documents that the other poster refers to are meaningless. I repeat I am NOT a Linux expert (or lover). I've got it and I have to live with it. You can't go from being able to install Centos and a few programs to a Linux Guru overnight, especially at 81.

I'm just annoyed that yet again, I have to get hold of some Linux thing that DOESN'T WORK.
All I get for it is a link to an aricle which may as well be written in Chinese.

We are not sure either. The acme-v01 endpoint has not been used for some time. And, you seem to have started using Let's Encrypt only in last few months. You shouldn't have any references to that endpoint. It is almost as if someone modified it manually.

Can you show contents of this file.

/etc/letsencrypt/renewal/corp.networkingtechnology.org.conf

and, also output of these

ls -l /etc/letsencrypt/renewal
ls -l /etc/letsencrypt/accounts
9 Likes
# renew_before_expiry = 30 days
version = 1.22.0
archive_dir = /etc/letsencrypt/archive/corp.networkingtechnology.org
cert = /etc/letsencrypt/live/corp.networkingtechnology.org/cert.pem
privkey = /etc/letsencrypt/live/corp.networkingtechnology.org/privkey.pem
chain = /etc/letsencrypt/live/corp.networkingtechnology.org/chain.pem
fullchain = /etc/letsencrypt/live/corp.networkingtechnology.org/fullchain.pem

# Options used in the renewal process
[renewalparams]
authenticator = apache
account = 1b757b9724299c0f5ab3269e0c9cd7b4
server = https://acme-v02.api.letsencrypt.org/directory
[root@alma-86 ~]# ls -l /etc/letsencrypt/renewal
total 12
-rw-r--r-- 1 root root 585 Sep 28 13:55 corp.networkingtechnology.org.conf
-rw-r--r-- 1 root root 625 Sep 28 13:53 support.corp.networkingtechnology.org.conf
-rw-r--r-- 1 root root 625 Sep 28 13:54 writers.corp.networkingtechnology.org.conf
[root@alma-86 ~]# ls -l /etc/letsencrypt/accounts
total 0
drwx------ 3 root root 23 Jul 27 14:47 acme-staging-v02.api.letsencrypt.org
drwx------ 3 root root 23 Jul 27 14:29 acme-v02.api.letsencrypt.org

Hmm. I don't see any reason for the acme-v01. What does this command show?

certbot renew --cert-name corp.networkingtechnology.org --dry-run
9 Likes

Can we see this file?:

8 Likes

It is in post #25 Rudy (see as response to #24)

8 Likes

Congratulations all certificates can be renewed (I already told you that).

And, then this (again). Please show the output

certbot renew --cert-name corp.networkingtechnology.org 

Note I noticed that something touched your renewal conf files very recently. So, commands may behave differently and I need to verify step by step. So, please can the snark.

9 Likes

I missed the file content; as it wasn't labled...

This account:

doesn't match the error account:

I don't know what's going on.
I say:

  • remove the whole /etc/letsencrypt/ directory
  • restore it from a known working backup [or from a copy from another working system]
    [if that's not possible, then just reissue brand new certs]
8 Likes

Me neither. I'll restore a backup which was working. We had a power failure a week or more ago. I'll restore before the power failure and see what happens.

I'll do it tomorrow.

2 Likes

I'm out of words. I have no idea what's going on now.
Logwatch this morning:

 **Unmatched Entries**
    Reloading The Apache HTTP Server.: 5 Time(s)
    certbot-renew.service: Failed with result 'exit-code'.: 1 Time(s)
    certbot-renew.service: Main process exited, code=exited, status=1/FAILURE: 1 Time(s)
    certbot-renew.service: Succeeded.: 1 Time(s)

A few minutes ago I get this:

Hello, Your certificate (or certificates) for the names listed below will expire in 13 days (on 13 Oct 22 11:39 +0000). Please make sure to renew your certificate before then, or visitors to your web site will encounter errors. We recommend renewing certificates automatically when they have a third of their total lifetime left. For Let's Encrypt's current 90-day certificates, that means renewing 30 days before expiration. See Integration Guide - Let's Encrypt for details. www.corp.networkingtechnology.org For details about when we send these emails, please visit: Expiration Emails - Let's Encrypt In particular, note that this reminder email is still sent if you've obtained a slightly different certificate by adding or removing names. If you've replaced this certificate with a newer one that covers more or fewer names than the list above, you may be able to ignore this message. For any questions or support, please visit: https://community.letsencrypt.org/ Unfortunately, we can't provide support by email. If you are receiving this email in error, unsubscribe at: You've been unsubscribed Please note that this would also unsubscribe you from other Let's Encrypt service notices, including expiration reminders for any other certificates. Regards, The Let's Encrypt Team

What is going on?

Has any of that happened?

9 Likes