Problems with certbot renew

HankM · September 24, 2022, 2:27pm

My domain is: Corp,networkingtechnology.org
Logwatch produced this output: certbot-renew.service: Failed with result 'exit-code'.:2
Apache: latest update
The operating system my web server runs on is (include version): Alma 8.6
My hosting provider, if applicable, is: Self
I can login to a root shell on my machine (yes):
I'm using a control panel to manage my site (no, or provide the name and version of the control panel): NO server is in the next room.
The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): 1.22.0

I get so frustrated with everything to do with Linux.
I created 3 certificates on this server:
corp.networkingtechnology.org
support.networkingtechnology.org
writers.networkingtechnology.org.

A few day ago we had a power failure. It lasted longer than the UPS and my Proliant Server running VMWare ESX 6 went down with all the virtual machines.

When the power came back on, everything SEEMED to be working fine (still is), but now I'm getting this error message in Logwatch. I'm not sure if it's due to the power outage or something else.

I tried certbot renew --dry-run and all three certificates come up with error messages. The DNS hasn't changed The only thing that's been done since the Certificates were installed was to create a DKIM record. But, despite that, suddenly there's yet another Linux headache. It really is depressing, firefighting for no apparent reason.

Simulating renewal of an existing certificate for corp.networkingtechnology.org

Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:
  Domain: corp.networkingtechnology.org
  Type:   connection
  Detail: 79.132.230.60: Fetching http://corp.networkingtechnology.org/.well-known/acme-challenge/FIBH061ii42bl_4Iq--qbHu-1M8Te0fThQg06Zh7_bY: Timeout during connect (likely firewall problem)

Hint: The Certificate Authority failed to verify the temporary Apache configuration changes made by Certbot. Ensure that the listed domains point to this Apache server and that it is accessible from the internet.

Failed to renew certificate corp.networkingtechnology.org with error: Some challenges have failed.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/support.corp.networkingtechnology.org.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Simulating renewal of an existing certificate for support.corp.networkingtechnology.org

Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:
  Domain: support.corp.networkingtechnology.org
  Type:   connection
  Detail: 79.132.230.60: Fetching http://support.corp.networkingtechnology.org/.well-known/acme-challenge/UujYDC8mZMo5T7immjBg7vaBRUkOrbrcLbv4JynZd-w: Timeout during connect (likely firewall problem)

Hint: The Certificate Authority failed to verify the temporary Apache configuration changes made by Certbot. Ensure that the listed domains point to this Apache server and that it is accessible from the internet.

Failed to renew certificate support.corp.networkingtechnology.org with error: Some challenges have failed.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/writers.corp.networkingtechnology.org.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Simulating renewal of an existing certificate for writers.corp.networkingtechnology.org

Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:
  Domain: writers.corp.networkingtechnology.org
  Type:   connection
  Detail: 79.132.230.60: Fetching http://writers.corp.networkingtechnology.org/.well-known/acme-challenge/0BYmksTOB9Yh41Di-yLRhEaLiGnyvjEViyJ6e-q1Skw: Timeout during connect (likely firewall problem)

Hint: The Certificate Authority failed to verify the temporary Apache configuration changes made by Certbot. Ensure that the listed domains point to this Apache server and that it is accessible from the internet.

Failed to renew certificate writers.corp.networkingtechnology.org with error: Some challenges have failed.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
All simulated renewals failed. The following certificates could not be renewed:
  /etc/letsencrypt/live/corp.networkingtechnology.org/fullchain.pem (failure)
  /etc/letsencrypt/live/support.corp.networkingtechnology.org/fullchain.pem (failure)
  /etc/letsencrypt/live/writers.corp.networkingtechnology.org/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
3 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

MikeMcQ · September 24, 2022, 2:53pm

Hello again @HankM Sorry to hear about your new troubles.

Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:
  Domain: corp.networkingtechnology.org
  Type:   connection
  Detail: 79.132.230.60: 
Fetching http://corp.networkingtechnology.org/.well-known/acme-challenge/FIBH061ii42bl_4Iq--qbHu-1M8Te0fThQg06Zh7_bY: 
Timeout during connect (likely firewall problem)

The Let's Encrypt servers are not able to connect using your domain name. And, as noted in the message, that is most likely a firewall issue on your end.

I also cannot connect to it from my test server. And, another site I use to test connectivity cannot reach your site from various points around the world. I don't see how anyone could connect to your site.

You know your equipment better than we do. I'm not sure what else to say. Perhaps some other volunteer will offer advice.

I know you don't have any way to test connectivity from outside your local network. But, you could try using a site like this one to test

rg305 · September 24, 2022, 4:09pm

An actual version number is preferred.

Name:    corp.networkingtechnology.org
Address: 79.132.230.60

Name:    79.132.230.60.static.edpnet.net
Address: 79.132.230.60

Does your ISP (edpnet.be) block (HTTP) port 80?

Bruce5051 · September 24, 2022, 4:37pm

Also has support here: Help & Support | edpnet.be

HankM · September 24, 2022, 4:56pm

It definitely has nothing to do with my OPNsense Firewall, I have 4 servers on 4 of my static IP Addresses. 79.132.230.58. 59 and 61 (offline) are identical with 60, which is the one not working.

I've created a Ticket with EDPNet. I doubt anything will happen before next week. Things only die on weekends around here

Bruce5051 · September 24, 2022, 5:04pm

Only 79.132.230.58 has some connection Check website performance and response: Check host - online website monitoring
No Port 80 connection for the rest

79.132.230.59 Check website performance and response: Check host - online website monitoring
79.132.230.60 Check website performance and response: Check host - online website monitoring
79.132.230.61 Check website performance and response: Check host - online website monitoring

Let's Encrypt uses Multi-Perspective Validation Improves Domain Validation Security - Let's Encrypt

rg305 · September 24, 2022, 6:20pm

I'd check what I could check, and that is the first place I would start.

HankM · September 24, 2022, 6:42pm

I have NO idea how you come up with that. I suggest you try a different Port Scanner.

If I use TCP Port Scanner, Online Port Scan, Port Scanning | IPVoid
58 AND 59 are both OK, but 59 is a GroupWise mail server, so why would I have Port 80 and 443 open? It has NO SSL Certificate, that's why I'm moving to Postfix.

GroupWise 7 uses NEITHER Port 80 or Port 443. Only 25, 81, 110, 7206 and 8009. That server is online and sending and receiving mail

Pegasus, 72.132.230.61 is offlline (I did say that I'm sure).

So, as I said originally 58, 59 and 61 ARE working (if I bring up 61). I did test with TCP Port Scanner, Online Port Scan, Port Scanning | IPVoid and ONLY 60 is not working.

I may be old but NOT stupid.

Bruce5051 · September 24, 2022, 6:45pm

@HankM,

I see this:

Bruce5051 · September 24, 2022, 6:49pm

Because you has posted this

Bruce5051 · September 24, 2022, 6:51pm

Do you know if TCP Port Scanner, Online Port Scan, Port Scanning | IPVoid checks from different locations for different geographic locations and/or time?

Best Practice - Keep Port 80 Open

Let's Encrypt uses Multi-Perspective Validation Improves Domain Validation Security - Let's Encrypt

rg305 · September 25, 2022, 3:23am

Certificates can be used on more ports than just 443.
Some of those can be placed behind a reverse proxy [which can use a cert].

HankM · September 25, 2022, 10:15am

Well, I asked about the possibility of adding a DKIM Cert to GW 7, but was told by everyone it was impossible. That's why I created the new server.

HankM · September 25, 2022, 10:27am

This is insane. It WAS the power outage and this awful bloody Fritz!Box we have. It's a pile of junk. It's tucked away (in the same place as our previous Sagem was, that NEVER gave us a problem). I noticed last night that the FB had a red light showing.

I rebooted (well, the only way is to unplug it and then plug the stupid thing back in). After about 10 minutes, I saw both lights green.

When I got up this morning, one light was red again. 4 reboots later, I had another idea so I switched it off and restored the OPNsense Firewall back to the day BEFORE we had the power outage.

When the restore completed, I plugged the FB back in and everything SEEMS to be working again. Which corrupted what? Who knows.

I think we are all in for MANY power failures this winter. I'll have to put a UPS on my workstation so I can login to VMware ESX and bring the servers down gracefully.

HankM · September 26, 2022, 10:08pm

Well this gets stranger by the day. Port scanner shows 80 and 443 open. certbot renew --dry-run shows no problems. Only congratulations all succeeded.
People can connect to the forum, but logwatch is still showing this:

    certbot-renew.service: Failed with result 'exit-code'.: 2 Time(s)
    certbot-renew.service: Main process exited, code=exited, status=1/FAILURE: 2 Time(s)

What's ther new problem now? Am I going to get problems like this all the time? This is SO frustrating

rg305 · September 27, 2022, 12:16am

Are there any logs to go with that?
What is the certbot renewal command being run?

HankM · September 27, 2022, 6:17am

The renewal command is built into Alma 8.6. Where would I find it?
The only log I have is Logwatch
Which log should I be looking at?
Is there any way to improve the letsencrypt detail delivered by Logwatch?

I ran certbot renew

[root@alma-86 ~]# certbot renew
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/corp.networkingtechnology.org.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Failed to renew certificate corp.networkingtechnology.org with error: Account at /etc/letsencrypt/accounts/acme-v01.api.letsencrypt.org/directory/90eb7ed71db53fd117f72c0855591879 does not exist

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/support.corp.networkingtechnology.org.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Certificate not yet due for renewal

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/writers.corp.networkingtechnology.org.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Certificate not yet due for renewal

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
The following certificates are not due for renewal yet:
  /etc/letsencrypt/live/support.corp.networkingtechnology.org/fullchain.pem expires on 2022-12-24 (skipped)
  /etc/letsencrypt/live/writers.corp.networkingtechnology.org/fullchain.pem expires on 2022-11-19 (skipped)
All renewals failed. The following certificates could not be renewed:
  /etc/letsencrypt/live/corp.networkingtechnology.org/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
[root@alma-86 ~]#

Here is the complete logfile for 27/09/2022

2022-09-27 09:44:59,255:DEBUG:certbot._internal.main:certbot version: 1.22.0
2022-09-27 09:44:59,258:DEBUG:certbot._internal.main:Location of certbot entry point: /usr/bin/certbot
2022-09-27 09:44:59,258:DEBUG:certbot._internal.main:Arguments: []
2022-09-27 09:44:59,259:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2022-09-27 09:44:59,302:DEBUG:certbot._internal.log:Root logging level set at 30
2022-09-27 09:44:59,305:DEBUG:certbot._internal.display.obj:Notifying user: Processing /etc/letsencrypt/renewal/corp.networkingtechnology.org.conf
2022-09-27 09:44:59,348:DEBUG:certbot._internal.plugins.selection:Requested authenticator <certbot._internal.cli.cli_utils._Default object at 0x7f391d3a4c50> and installer <certbot._internal.cli.cli_utils._Default object at 0x7f391d3a4c50>
2022-09-27 09:44:59,391:DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): r3.o.lencr.org:80
2022-09-27 09:44:59,437:DEBUG:urllib3.connectionpool:http://r3.o.lencr.org:80 "POST / HTTP/1.1" 200 503
2022-09-27 09:44:59,439:DEBUG:certbot.ocsp:OCSP response for certificate /etc/letsencrypt/archive/corp.networkingtechnology.org/cert1.pem is signed by the certificate's issuer.
2022-09-27 09:44:59,445:DEBUG:certbot.ocsp:OCSP certificate status for /etc/letsencrypt/archive/corp.networkingtechnology.org/cert1.pem is: OCSPCertStatus.GOOD
2022-09-27 09:44:59,450:DEBUG:certbot._internal.storage:Should renew, less than 30 days before certificate expiry 2022-10-08 19:39:36 UTC.
2022-09-27 09:44:59,450:INFO:certbot._internal.renewal:Certificate is due for renewal, auto-renewing...
2022-09-27 09:44:59,450:DEBUG:certbot._internal.plugins.selection:Requested authenticator apache and installer apache
2022-09-27 09:44:59,596:DEBUG:certbot_apache._internal.configurator:Apache version is 2.4.37
2022-09-27 09:44:59,978:DEBUG:certbot._internal.plugins.selection:Single candidate plugin: * apache
Description: Apache Web Server plugin
Interfaces: Installer, Authenticator, Plugin
Entry point: apache = certbot_apache._internal.entrypoint:ENTRYPOINT
Initialized: <certbot_apache._internal.override_fedora.FedoraConfigurator object at 0x7f391d329f98>
Prep: True
2022-09-27 09:44:59,982:DEBUG:certbot._internal.plugins.selection:Single candidate plugin: * apache
Description: Apache Web Server plugin
Interfaces: Installer, Authenticator, Plugin
Entry point: apache = certbot_apache._internal.entrypoint:ENTRYPOINT
Initialized: <certbot_apache._internal.override_fedora.FedoraConfigurator object at 0x7f391d329f98>
Prep: True
2022-09-27 09:44:59,982:DEBUG:certbot._internal.plugins.selection:Selected authenticator <certbot_apache._internal.override_fedora.FedoraConfigurator object at 0x7f391d329f98> and installer <certbot_apache._internal.override_fedora.FedoraConfigurator object at 0x7f391d329f98>
2022-09-27 09:44:59,983:INFO:certbot._internal.plugins.selection:Plugins selected: Authenticator apache, Installer apache
2022-09-27 09:44:59,983:ERROR:certbot._internal.renewal:Failed to renew certificate corp.networkingtechnology.org with error: Account at /etc/letsencrypt/accounts/acme-v01.api.letsencrypt.org/directory/90eb7ed71db53fd117f72c0855591879 does not exist
2022-09-27 09:44:59,985:DEBUG:certbot._internal.renewal:Traceback was:
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/certbot/_internal/renewal.py", line 485, in handle_renewal_request
    main.renew_cert(lineage_config, plugins, renewal_candidate)
  File "/usr/lib/python3.6/site-packages/certbot/_internal/main.py", line 1439, in renew_cert
    le_client = _init_le_client(config, auth, installer)
  File "/usr/lib/python3.6/site-packages/certbot/_internal/main.py", line 788, in _init_le_client
    acc, acme = _determine_account(config)
  File "/usr/lib/python3.6/site-packages/certbot/_internal/main.py", line 688, in _determine_account
    acc = account_storage.load(config.account)
  File "/usr/lib/python3.6/site-packages/certbot/_internal/account.py", line 247, in load
    return self._load_for_server_path(account_id, self.config.server_path)
  File "/usr/lib/python3.6/site-packages/certbot/_internal/account.py", line 218, in _load_for_server_path
    prev_loaded_account = self._load_for_server_path(account_id, prev_server_path)
  File "/usr/lib/python3.6/site-packages/certbot/_internal/account.py", line 228, in _load_for_server_path
    "Account at %s does not exist" % account_dir_path)
certbot.errors.AccountNotFound: Account at /etc/letsencrypt/accounts/acme-v01.api.letsencrypt.org/directory/90eb7ed71db53fd117f72c0855591879 does not exist

2022-09-27 09:44:59,986:DEBUG:certbot._internal.display.obj:Notifying user: Processing /etc/letsencrypt/renewal/support.corp.networkingtechnology.org.conf
2022-09-27 09:45:00,029:DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): r3.o.lencr.org:80
2022-09-27 09:45:00,067:DEBUG:urllib3.connectionpool:http://r3.o.lencr.org:80 "POST / HTTP/1.1" 200 503
2022-09-27 09:45:00,070:DEBUG:certbot.ocsp:OCSP response for certificate /etc/letsencrypt/archive/support.corp.networkingtechnology.org/cert2.pem is signed by the certificate's issuer.
2022-09-27 09:45:00,071:DEBUG:certbot.ocsp:OCSP certificate status for /etc/letsencrypt/archive/support.corp.networkingtechnology.org/cert2.pem is: OCSPCertStatus.GOOD
2022-09-27 09:45:00,073:DEBUG:certbot._internal.display.obj:Notifying user: Certificate not yet due for renewal
2022-09-27 09:45:00,074:DEBUG:certbot._internal.plugins.selection:Requested authenticator apache and installer apache
2022-09-27 09:45:00,080:DEBUG:certbot._internal.plugins.selection:Selecting plugin: * apache
Description: Apache Web Server plugin
Interfaces: Installer, Authenticator, Plugin
Entry point: apache = certbot_apache._internal.entrypoint:ENTRYPOINT
Initialized: <certbot_apache._internal.override_fedora.FedoraConfigurator object at 0x7f391d3294e0>
2022-09-27 09:45:00,081:DEBUG:certbot.plugins.storage:Plugin storage file /etc/letsencrypt/.pluginstorage.json was empty, no values loaded
2022-09-27 09:45:00,081:DEBUG:certbot._internal.display.obj:Notifying user: Processing /etc/letsencrypt/renewal/writers.corp.networkingtechnology.org.conf
2022-09-27 09:45:00,120:DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): r3.o.lencr.org:80
2022-09-27 09:45:00,159:DEBUG:urllib3.connectionpool:http://r3.o.lencr.org:80 "POST / HTTP/1.1" 200 503
2022-09-27 09:45:00,161:DEBUG:certbot.ocsp:OCSP response for certificate /etc/letsencrypt/archive/writers.corp.networkingtechnology.org/cert1.pem is signed by the certificate's issuer.
2022-09-27 09:45:00,162:DEBUG:certbot.ocsp:OCSP certificate status for /etc/letsencrypt/archive/writers.corp.networkingtechnology.org/cert1.pem is: OCSPCertStatus.GOOD
2022-09-27 09:45:00,163:DEBUG:certbot._internal.display.obj:Notifying user: Certificate not yet due for renewal
2022-09-27 09:45:00,165:DEBUG:certbot._internal.plugins.selection:Requested authenticator apache and installer apache
2022-09-27 09:45:00,171:DEBUG:certbot._internal.plugins.selection:Selecting plugin: * apache
Description: Apache Web Server plugin
Interfaces: Installer, Authenticator, Plugin
Entry point: apache = certbot_apache._internal.entrypoint:ENTRYPOINT
Initialized: <certbot_apache._internal.override_fedora.FedoraConfigurator object at 0x7f391ccc2be0>
2022-09-27 09:45:00,172:DEBUG:certbot.plugins.storage:Plugin storage file /etc/letsencrypt/.pluginstorage.json was empty, no values loaded
2022-09-27 09:45:00,172:DEBUG:certbot._internal.display.obj:Notifying user: 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2022-09-27 09:45:00,173:DEBUG:certbot._internal.display.obj:Notifying user: The following certificates are not due for renewal yet:
2022-09-27 09:45:00,173:DEBUG:certbot._internal.display.obj:Notifying user:   /etc/letsencrypt/live/support.corp.networkingtechnology.org/fullchain.pem expires on 2022-12-24 (skipped)
  /etc/letsencrypt/live/writers.corp.networkingtechnology.org/fullchain.pem expires on 2022-11-19 (skipped)
2022-09-27 09:45:00,173:ERROR:certbot._internal.renewal:All renewals failed. The following certificates could not be renewed:
2022-09-27 09:45:00,173:ERROR:certbot._internal.renewal:  /etc/letsencrypt/live/corp.networkingtechnology.org/fullchain.pem (failure)
2022-09-27 09:45:00,174:DEBUG:certbot._internal.display.obj:Notifying user: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2022-09-27 09:45:00,174:DEBUG:certbot._internal.log:Exiting abnormally:
Traceback (most recent call last):
  File "/usr/bin/certbot", line 11, in <module>
    load_entry_point('certbot==1.22.0', 'console_scripts', 'certbot')()
  File "/usr/lib/python3.6/site-packages/certbot/main.py", line 19, in main
    return internal_main.main(cli_args)
  File "/usr/lib/python3.6/site-packages/certbot/_internal/main.py", line 1632, in main
    return config.func(config, plugins)
  File "/usr/lib/python3.6/site-packages/certbot/_internal/main.py", line 1518, in renew
    renewal.handle_renewal_request(config)
  File "/usr/lib/python3.6/site-packages/certbot/_internal/renewal.py", line 512, in handle_renewal_request
    len(renew_failures), len(parse_failures)))
certbot.errors.Error: 1 renew failure(s), 0 parse failure(s)
2022-09-27 09:45:00,175:ERROR:certbot._internal.log:1 renew failure(s), 0 parse failure(s)

rg305 · September 27, 2022, 4:00pm

The problem might be related to "acme-v01.api.letsencrypt.org" having been retired.

HankM · September 27, 2022, 4:22pm

Great!!! So what do I DO about it? Reinstall everything or what?

harrycranston53 · September 27, 2022, 4:44pm

Read these for start: