Unable to renew: challenge failed with apache backend

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: capablehost.ca (and www.capablehost.ca)

I ran this command: sudo certbot renew --dry-run -v

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log


Processing /etc/letsencrypt/renewal/www.capablehost.ca.conf


Certificate not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator apache, Installer apache
Simulating renewal of an existing certificate for www.capablehost.ca and capablehost.ca
Performing the following challenges:
http-01 challenge for capablehost.ca
http-01 challenge for www.capablehost.ca
Waiting for verification...
Challenge failed for domain capablehost.ca
Challenge failed for domain www.capablehost.ca
http-01 challenge for capablehost.ca
http-01 challenge for www.capablehost.ca

Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:
Domain: capablehost.ca
Type: unauthorized
Detail: Invalid response from http://capablehost.ca/.well-known/acme-challenge/qTN0ijn6ypRCgg81OEk6NsM01-XjV-ZNdU8Y6k_LTRg [2600:3c03::f03c:91ff:fee7:7da4]: 403

Domain: www.capablehost.ca
Type: unauthorized
Detail: Invalid response from http://www.capablehost.ca/.well-known/acme-challenge/WITTUYE9aKe9KMAQhzt7dYl2yk5gVKUO7q3zA_gynsM [2600:3c03::f03c:91ff:fee7:7da4]: 403

Hint: The Certificate Authority failed to verify the temporary Apache configuration changes made by Certbot. Ensure that the listed domains point to this Apache server and that it is accessible from the internet.

Cleaning up challenges
Failed to renew certificate www.capablehost.ca with error: Some challenges have failed.


All simulated renewals failed. The following certificates could not be renewed:
/etc/letsencrypt/live/www.capablehost.ca/fullchain.pem (failure)


1 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

My web server is (include version):
apache httpd 2.4.38-3+deb10u7

The operating system my web server runs on is (include version):
debian 10 (buster)

My hosting provider, if applicable, is:
linode

I can login to a root shell on my machine (yes or no, or I don't know):
yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
certbot 1.25.0

Here is the output of apachectl -S:

AH00112: Warning: DocumentRoot [/var/www/html/pedigree-db/sandbox] does not exist
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message
VirtualHost configuration:
*:443 is a NameVirtualHost
default server capablehost.ca (/etc/apache2/sites-enabled/000-default.conf:78)
port 443 namevhost capablehost.ca (/etc/apache2/sites-enabled/000-default.conf:78)
alias www.capablehost.ca
port 443 namevhost bluebirdlane.com (/etc/apache2/sites-enabled/bluebirdlane.com.conf:15)
alias www.bluebirdlane.com
port 443 namevhost elix.ca (/etc/apache2/sites-enabled/elix.ca.conf:13)
alias www.elix.ca
port 443 namevhost mogly.ca (/etc/apache2/sites-enabled/mogly.ca.conf:13)
alias www.mogly.ca
port 443 namevhost sonnenhofstables.ca (/etc/apache2/sites-enabled/sonnenhofstables.ca.conf:13)
alias www.sonnenhofstables.ca
port 443 namevhost stefanv.com (/etc/apache2/sites-enabled/stefanv.com.conf:15)
alias www.stefanv.com
port 443 namevhost summerbaillie.com (/etc/apache2/sites-enabled/summerbaillie.com.conf:13)
alias www.summerbaillie.com
*:80 is a NameVirtualHost
default server capablehost.ca (/etc/apache2/sites-enabled/000-default.conf:1)
port 80 namevhost capablehost.ca (/etc/apache2/sites-enabled/000-default.conf:1)
alias www.capablehost.ca
port 80 namevhost adca.pedigree-db.com (/etc/apache2/sites-enabled/adca.pedigree-db.com.conf:1)
port 80 namevhost akkps.pedigree-db.com (/etc/apache2/sites-enabled/akkps.pedigree-db.com.conf:1)
VirtualHost configuration:
*:443 is a NameVirtualHost
default server capablehost.ca (/etc/apache2/sites-enabled/000-default.conf:78)
port 443 namevhost capablehost.ca (/etc/apache2/sites-enabled/000-default.conf:78)
alias www.capablehost.ca
port 443 namevhost bluebirdlane.com (/etc/apache2/sites-enabled/bluebirdlane.com.conf:15)
alias www.bluebirdlane.com
port 443 namevhost elix.ca (/etc/apache2/sites-enabled/elix.ca.conf:13)
alias www.elix.ca
port 443 namevhost mogly.ca (/etc/apache2/sites-enabled/mogly.ca.conf:13)
alias www.mogly.ca
port 443 namevhost sonnenhofstables.ca (/etc/apache2/sites-enabled/sonnenhofstables.ca.conf:13)
alias www.sonnenhofstables.ca
port 443 namevhost stefanv.com (/etc/apache2/sites-enabled/stefanv.com.conf:15)
alias www.stefanv.com
port 443 namevhost summerbaillie.com (/etc/apache2/sites-enabled/summerbaillie.com.conf:13)
alias www.summerbaillie.com
*:80 is a NameVirtualHost
default server capablehost.ca (/etc/apache2/sites-enabled/000-default.conf:1)
port 80 namevhost capablehost.ca (/etc/apache2/sites-enabled/000-default.conf:1)
alias www.capablehost.ca
port 80 namevhost adca.pedigree-db.com (/etc/apache2/sites-enabled/adca.pedigree-db.com.conf:1)
port 80 namevhost akkps.pedigree-db.com (/etc/apache2/sites-enabled/akkps.pedigree-db.com.conf:1)
port 80 namevhost bluebirdlane.com (/etc/apache2/sites-enabled/bluebirdlane.com.conf:1)
alias www.bluebirdlane.com
port 80 namevhost capable.ca (/etc/apache2/sites-enabled/capable.ca.conf:1)
alias www.capable.ca
port 80 namevhost elix.bluebirdlane.com (/etc/apache2/sites-enabled/elix.bluebirdlane.com.conf:1)
port 80 namevhost elix.ca (/etc/apache2/sites-enabled/elix.ca.conf:1)
alias www.elix.ca
port 80 namevhost hearthigh.pedigree-db.com (/etc/apache2/sites-enabled/hearthigh.pedigree-db.com.conf:1)
port 80 namevhost kiko.pedigree-db.com (/etc/apache2/sites-enabled/kiko.pedigree-db.com.conf:1)
port 80 namevhost mogly.ca (/etc/apache2/sites-enabled/mogly.ca.conf:1)
alias www.mogly.ca
port 80 namevhost motocalc.com (/etc/apache2/sites-enabled/motocalc.com.conf:1)
alias www.motocalc.com
port 80 namevhost pedigree-db.com (/etc/apache2/sites-enabled/pedigree-db.com.conf:1)
alias www.pedigree-db.com
port 80 namevhost sandbox.pedigree-db.com (/etc/apache2/sites-enabled/sandbox.pedigree-db.com.conf:1)
port 80 namevhost sonnenhof.capablehost.ca (/etc/apache2/sites-enabled/sonnenhof.capablehost.ca.conf:1)
port 80 namevhost sonnenhofstables.ca (/etc/apache2/sites-enabled/sonnenhofstables.ca.conf:1)
alias www.sonnenhofstables.ca
port 80 namevhost stefanv.com (/etc/apache2/sites-enabled/stefanv.com.conf:1)
alias www.stefanv.com
port 80 namevhost summerbaillie.com (/etc/apache2/sites-enabled/summerbaillie.com.conf:1)
alias www.summerbaillie.com

Here are some relevant entries of the letsencrypt log file:

2022-03-29 16:34:14,205:INFO:certbot._internal.auth_handler:http-01 challenge for capablehost.ca
2022-03-29 16:34:14,205:INFO:certbot._internal.auth_handler:http-01 challenge for www.capablehost.ca
2022-03-29 16:34:14,220:DEBUG:certbot_apache._internal.http_01:Adding a temporary challenge validation Inclu
de for name: capablehost.ca in: /etc/apache2/sites-enabled/000-default.conf
2022-03-29 16:34:14,220:DEBUG:certbot_apache._internal.http_01:Adding a temporary challenge validation Inclu
de for name: capablehost.ca in: /etc/apache2/sites-enabled/000-default.conf
2022-03-29 16:34:14,220:DEBUG:certbot_apache.internal.http_01:writing a pre config file with text:
RewriteEngine on
RewriteRule ^/.well-known/acme-challenge/([A-Za-z0-9-
=]+)$ /var/lib/letsencrypt/http_challenges/$1
[END]

2022-03-29 16:34:14,221:DEBUG:certbot_apache._internal.http_01:writing a post config file with text:
<Directory /var/lib/letsencrypt/http_challenges>
Require all granted

<Location /.well-known/acme-challenge>
Require all granted

2022-03-29 16:34:14,258:DEBUG:certbot.reverter:Creating backup of /etc/apache2/sites-enabled/000-default.con

and later:

2022-03-29 16:34:18,716:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Tue, 29 Mar 2022 20:34:18 GMT
Content-Type: application/json
Content-Length: 1109
Connection: keep-alive
Boulder-Requester: 48962718
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: 0001zWQNcElIQAucu6dYHISDgMIFWdCRid1sDUAuvgDN3nU
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{ 
  "identifier": {
    "type": "dns",
    "value": "capablehost.ca"
  },
  "status": "invalid",
  "expires": "2022-04-05T20:26:49Z",
  "challenges": [
    { 
      "type": "http-01",
      "status": "invalid",
      "error": {
        "type": "urn:ietf:params:acme:error:unauthorized",
        "detail": "Invalid response from http://capablehost.ca/.well-known/acme-challenge/qTN0ijn6ypRCgg81OEk6NsM01-XjV-ZNdU8Y6k_LTRg [2600:3c03::f03c:91ff:fee7:7da4]: 403",
        "status": 403
      },
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/2041290178/yiTsBw",
      "token": "qTN0ijn6ypRCgg81OEk6NsM01-XjV-ZNdU8Y6k_LTRg",
      "validationRecord": [
        { 
          "url": "http://capablehost.ca/.well-known/acme-challenge/qTN0ijn6ypRCgg81OEk6NsM01-XjV-ZNdU8Y6k_LTRg",
          "hostname": "capablehost.ca",
          "port": "80",
          "addressesResolved": [
            "45.33.77.33",
            "2600:3c03::f03c:91ff:fee7:7da4"
          ],
          "addressUsed": "2600:3c03::f03c:91ff:fee7:7da4"
        }
      ],
      "validated": "2022-03-29T20:34:17Z"
    }
  ]
}

The HTTP 403 is the most unusual part of what's happening here.

That request has probably generated a corresponding error message in your Apache's error_log. It would be helpful if you could find it and see what it says.

2 Likes

Thanks @_az; I think this pointed me to the solution. The log says:

[Thu Mar 31 14:46:44.448330 2022] [core:error] [pid 2535] (13)Permission denied: [client 2a05:d014:531:8602:ec7e:b8b4:45c3:1a3b:13098] AH00035: access to /.well-known/acme-challenge/Oc1omQCTabbIoKKDlzW2Dqo9h76vOPhGNWXdtIXtPps denied (filesystem path '/var/lib/letsencrypt/http_challenges') because search permissions are missing on a component of the path
[Thu Mar 31 14:46:44.486511 2022] [core:error] [pid 2534] (13)Permission denied: [client 2600:1f14:a8b:502:b1fc:e4c3:fd66:2675:45690] AH00035: access to /.well-known/acme-challenge/tCFhGkt96Aqu_n2wlP3Jj3II4ZPP8gVjOqMyZoKKiRg denied (filesystem path '/var/lib/letsencrypt/http_challenges') because search permissions are missing on a component of the path
[Thu Mar 31 14:46:44.488643 2022] [core:error] [pid 2537] (13)Permission denied: [client 2600:1f14:a8b:502:b1fc:e4c3:fd66:2675:45688] AH00035: access to /.well-known/acme-challenge/Oc1omQCTabbIoKKDlzW2Dqo9h76vOPhGNWXdtIXtPps denied (filesystem path '/var/lib/letsencrypt/http_challenges') because search permissions are missing on a component of the path
[Thu Mar 31 14:46:44.517546 2022] [core:error] [pid 2536] (13)Permission denied: [client 2600:3000:2710:300::21:24118] AH00035: access to /.well-known/acme-challenge/Oc1omQCTabbIoKKDlzW2Dqo9h76vOPhGNWXdtIXtPps denied (filesystem path '/var/lib/letsencrypt/http_challenges') because search permissions are missing on a component of the path
[Thu Mar 31 14:46:44.544897 2022] [core:error] [pid 2538] (13)Permission denied: [client 2a05:d014:531:8602:ec7e:b8b4:45c3:1a3b:13100] AH00035: access to /.well-known/acme-challenge/tCFhGkt96Aqu_n2wlP3Jj3II4ZPP8gVjOqMyZoKKiRg denied (filesystem path '/var/lib/letsencrypt/http_challenges') because search permissions are missing on a component of the path
[Thu Mar 31 14:46:44.872031 2022] [core:error] [pid 2540] (13)Permission denied: [client 2600:3000:2710:300::22:11402] AH00035: access to /.well-known/acme-challenge/tCFhGkt96Aqu_n2wlP3Jj3II4ZPP8gVjOqMyZoKKiRg denied (filesystem path '/var/lib/letsencrypt/http_challenges') because search permissions are missing on a component of the path

I note that /var/lib/letsencrypt has permissions 750 (and /var/lib/letsencrypt/http_challenges has permissions 755). Both are owned by root.

Doing a chmod o+rx /var/lib/letsencrypt let the dry run succeed. I think that should be considered safe (especially for our case, where we don't expect untrusted users on the server) -- please let me know if there are reasons why it wouldn't be.

That leaves figuring out if there's anything we can do to help diagnose what went wrong here, if that's useful to you guys. Should this directory have read and execute permissions? Is there an obvious reason why it wouldn't have those permissions in our case?

1 Like

As far as I can tell, Certbot will always create this directory with 0755.

Is it possible the permissions were changed by something or someone external?

3 Likes

Glad to hear that 0755 is the expected set of permissions.

I can't imagine who or what would have set it to 0750; there are no other (human) shell accounts on the server. Maybe something in the snap system? But that's a wild guess. Okay, I'm fine with this remaining a mystery. Thanks again for the pointer.

2 Likes

Well, I tried a fresh install of Debian 10 and the Certbot snap and it seems to work as expected:

root@oregano-fluid:~# stat /var/lib/letsencrypt/
File: /var/lib/letsencrypt/
Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: fe01h/65025d    Inode: 134218      Links: 4
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-04-01 16:27:24.087618710 +1100
Modify: 2022-04-01 16:27:36.835618710 +1100
Change: 2022-04-01 16:27:36.835618710 +1100
Birth: -

No idea, sorry!

4 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.