Like many others, LE cert failing

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:

I ran this command:
certbot --nginx
certbot --nginx -d "ameriquests.org" -d "www.ameriquests.org"

It produced this output:
certbot.errors.AuthorizationError: Some challenges have failed.

My web server is (include version):
nginx version: nginx/1.20.1

The operating system my web server runs on is (include version):
CentOS Linux release 7.9.2009 (Core)

My hosting provider, if applicable, is: self-hosted

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no - everything done from command line

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 1.27.0

This started when I tried to renew our certificates. So I thought maybe I needed to start from scratch and even uninstalled certbot and snap and reinstalling them. But I am getting the same result (below). Suggestions?

2022-05-09 20:55:40,281:DEBUG:urllib3.connectionpool:http://localhost:None "GET /v2/connections?snap=certbot&interface=content HTTP/1.1" 200 97
2022-05-09 20:55:40,661:DEBUG:certbot._internal.main:certbot version: 1.27.0
2022-05-09 20:55:40,661:DEBUG:certbot._internal.main:Location of certbot entry point: /snap/certbot/2035/bin/certbot
2022-05-09 20:55:40,661:DEBUG:certbot._internal.main:Arguments: ['--nginx', '-d', 'ameriquests.org', '-d', 'www.ameriquests.org', '--preconfigured-renewal']
2022-05-09 20:55:40,661:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#nginx,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2022-05-09 20:55:40,697:DEBUG:certbot._internal.log:Root logging level set at 30
2022-05-09 20:55:40,698:DEBUG:certbot._internal.plugins.selection:Requested authenticator nginx and installer nginx
2022-05-09 20:55:41,096:DEBUG:certbot._internal.plugins.selection:Single candidate plugin: * nginx
Description: Nginx Web Server plugin
Interfaces: Installer, Authenticator, Plugin
Entry point: nginx = certbot_nginx._internal.configurator:NginxConfigurator
Initialized: <certbot_nginx._internal.configurator.NginxConfigurator object at 0x7fa0ed42ddc0>
Prep: True
2022-05-09 20:55:41,097:DEBUG:certbot._internal.plugins.selection:Selected authenticator <certbot_nginx._internal.configurator.NginxConfigurator object at 0x7fa0ed42ddc0> and installer <certbot_nginx._internal.configurator.NginxConfigurator object at 0x7fa0ed42ddc0>
2022-05-09 20:55:41,097:INFO:certbot._internal.plugins.selection:Plugins selected: Authenticator nginx, Installer nginx
2022-05-09 20:55:41,108:DEBUG:certbot._internal.main:Picked account: <Account(RegistrationResource(body=Registration(key=None, contact=(), agreement=None, status=None, terms_of_service_agreed=None, only_return_existing=None, external_account_binding=None), uri='https://acme-v02.api.letsencrypt.org/acme/acct/231523810', new_authzr_uri=None, terms_of_service=None), 9e56d19b522284e6b5ec08ee6d659c4c, Meta(creation_dt=datetime.datetime(2021, 10, 8, 15, 48, 37, tzinfo=<UTC>), creation_host='libvm13.library.vanderbilt.edu', register_to_eff=None))>
2022-05-09 20:55:41,109:DEBUG:acme.client:Sending GET request to https://acme-v02.api.letsencrypt.org/directory.
2022-05-09 20:55:41,111:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org:443
2022-05-09 20:55:41,275:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "GET /directory HTTP/1.1" 200 658
2022-05-09 20:55:41,276:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Tue, 10 May 2022 01:55:41 GMT
Content-Type: application/json
Content-Length: 658
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "gI9iUpvSWfU": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
  "keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
    "website": "https://letsencrypt.org"
  },
  "newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
  "revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert"
}
2022-05-09 20:55:41,277:DEBUG:certbot._internal.cert_manager:Renewal conf file /etc/letsencrypt/renewal/ameriquests.org.conf is broken. Skipping.
2022-05-09 20:55:41,278:DEBUG:certbot._internal.cert_manager:Traceback was:
Traceback (most recent call last):
  File "/var/lib/snapd/snap/certbot/2035/lib/python3.8/site-packages/certbot/_internal/cert_manager.py", line 437, in _search_lineages
    candidate_lineage = storage.RenewableCert(renewal_file, cli_config)
  File "/var/lib/snapd/snap/certbot/2035/lib/python3.8/site-packages/certbot/_internal/storage.py", line 504, in __init__
    self._check_symlinks()
  File "/var/lib/snapd/snap/certbot/2035/lib/python3.8/site-packages/certbot/_internal/storage.py", line 577, in _check_symlinks
    raise errors.CertStorageError(
certbot.errors.CertStorageError: expected /etc/letsencrypt/live/ameriquests.org/cert.pem to be a symlink

Welcome to the community @jmcgranahan

First off, you look to be a victim of a Palo Alto brand firewall. They recently changed a default setting which is blocking the ACME challenge path URLs. See below post for details on correcting that. Requests to your domain fail in the same way as these others with "reset by peer". This was likely the reason your cert did not renew a month ago.

Your other problem is you have a corrupt set of Let's Encrypt folders. There is an error saying a file should be a symlink but it is not. I don't have time at the moment to help with that but either someone else will pickup or I might later tonight.

3 Likes

"the curse of the extra slash"

404 expected - 404 returned

curl -I http://ameriquests.org/.well-known/acme-challenge
HTTP/1.1 404 Not Found
Server: nginx/1.20.1
Date: Tue, 10 May 2022 02:55:35 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
X-Powered-By: PHP/7.3.33
Set-Cookie: OJSSID=6hfjlhe4p8patrn211fqu31aj0; path=/; domain=ameriquests.org

add a slash at the end and ...
404 expected - 404 NOT returned

curl -I http://ameriquests.org/.well-known/acme-challenge/
curl: (52) Empty reply from server
2 Likes

Yes, but I get the dreaded "reset by peer". And, an expected 404 when missing the slash after challenge (or shorter URLs). Regardless, likely Palo Alto setting

curl -I http://ameriquests.org/.well-known/acme-challenge/
curl: (56) Recv failure: Connection reset by peer

curl -I http://ameriquests.org/.well-known/acme-challenge/SampleToken
curl: (56) Recv failure: Connection reset by peer
2 Likes

Agreed.

2 Likes

Just wanted to report out that the issue was the Palo Alto firewall. Thanks all for pointing us in the right direction!

3 Likes