Certbot --apache failing challenge with 404

My domains are: www.sojournersgame.com (failing) and www.beyondthefarplane.com (working)
Both are on same Apache2 server.

I ran this command:
sudo certbot certonly -d sojournersgame.com --debug-challenges -v

It produced this output:
/var/log/letsencrypt/letsencrypt.log:

The following URLs should be accessible from the internet and return the value
mentioned:

URL:
http://sojournersgame.com/.well-known/acme-challenge/bNLDaQeKrumv8a7KTSB9Dq6CkWnG4Tk-MgAL_MIB0NU
Expected value:
bNLDaQeKrumv8a7KTSB9Dq6CkWnG4Tk-MgAL_MIB0NU.MxouZMBlfGmU5EV_Q9GctVcrQgy9g2kDOzRZHRP2ckE
2023-03-11 21:48:28,115:DEBUG:acme.client:JWS payload:
b'{}'
2023-03-11 21:48:28,119:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/chall-v3/210085800307/cPC7mw:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2FjY3QvMTAwMTk4MTE1NyIsICJub25jZSI6ICIzMjdDTWhzeUNUeVFHc3BiVTlRcTB2Y00tZzFVQk9NR0JJejFJX3RJS2EwX2VZZyIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvY2hhbGwtdjMvMjEwMDg1ODAwMzA3L2NQQzdtdyJ9",
  "signature": "HgMnN7xk1VEmkLP8Tj0ZZx7HeZgv4KBInwqg9nJcoWIcRBykKdrfBtpnnOsrkDenObRKviReITwVgk_3gfjYEybDHaxHPYN_3g1NzalQg5ldQ991CydUR_5w7UJhsmllv4bUBxVW7dGm_BWW2adsP37VJwDYpAwwFTH2OWF9ajFaqoWKpzmMTVGlYMOlirJnopE5ONLzrfJhpC2-hmWEo7ftTkM42jnl39Mn-RQU9EG0u8a96Ysmolf1L_8BzTgbwQQ_TR2vHSbPrgYa-LpmkxqR4gnTzTEUZ9THE6RshTdQ-CE6rRfb84WWZUMRvvHwx425Qee4-KdkzdbGw5wnJg",
  "payload": "e30"
}
2023-03-11 21:48:28,185:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "POST /acme/chall-v3/210085800307/cPC7mw HTTP/1.1" 200 187
2023-03-11 21:48:28,186:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Sat, 11 Mar 2023 21:48:28 GMT
Content-Type: application/json
Content-Length: 187
Connection: keep-alive
Boulder-Requester: 1001981157
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index", <https://acme-v02.api.letsencrypt.org/acme/authz-v3/210085800307>;rel="up"
Location: https://acme-v02.api.letsencrypt.org/acme/chall-v3/210085800307/cPC7mw
Replay-Nonce: 1DFAYMmPOA8F4hhOA8F6UTpe-ROLSbhJNoPICNohtCV1PHM
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "type": "http-01",
  "status": "pending",
  "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/210085800307/cPC7mw",
  "token": "bNLDaQeKrumv8a7KTSB9Dq6CkWnG4Tk-MgAL_MIB0NU"
}
2023-03-11 21:48:28,186:DEBUG:acme.client:Storing nonce: 1DFAYMmPOA8F4hhOA8F6UTpe-ROLSbhJNoPICNohtCV1PHM
2023-03-11 21:48:28,187:INFO:certbot._internal.auth_handler:Waiting for verification...
2023-03-11 21:48:29,189:DEBUG:acme.client:JWS payload:
b''
2023-03-11 21:48:29,191:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/authz-v3/210085800307:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2FjY3QvMTAwMTk4MTE1NyIsICJub25jZSI6ICIxREZBWU1tUE9BOEY0aGhPQThGNlVUcGUtUk9MU2JoSk5vUElDTm9odENWMVBITSIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYXV0aHotdjMvMjEwMDg1ODAwMzA3In0",
  "signature": "MeN6hvZuf2W2LgttvOzUqwJHY2QXmcLgOlxUDrV9r8tYn1KbBfe_IXUeGK7eDxsHRcJer1Fi5CPZnIbkQkh6suyu9ayMQaghyH8-xTT3wj4NGGVVhfc-ZdnEg7cbf-DkN6eUwHMqHVT_K-TtCc-bGtpwFUo2sUq6M4Tc9JoxYa_-gFFMB0GfiCyf4cvQhQHALc6Mt0x7M-JkocIg_JBKeCpR5P_7GVoqlNa3uiKVzqtY9tndaWX0CGRzVj4zXl_rwqkc3RCpBGHgc3I96aoJNUKhP1a84Pxb2HG2o_Y1iYGP4A-wwmmu6sWY1ahTfHpB-INNwqcFV0wRN7U0-nloiA",
  "payload": ""
}
2023-03-11 21:48:29,241:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/210085800307 HTTP/1.1" 200 1112
2023-03-11 21:48:29,242:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Sat, 11 Mar 2023 21:48:29 GMT
Content-Type: application/json
Content-Length: 1112
Connection: keep-alive
Boulder-Requester: 1001981157
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: C878Uz4eNNOWK5NFcKVUZWnbYtTLt_jNYeNAQ1MrGAFuneU
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "sojournersgame.com"
  },
  "status": "invalid",
  "expires": "2023-03-18T21:46:45Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "invalid",
      "error": {
        "type": "urn:ietf:params:acme:error:unauthorized",
        "detail": "2604:a880:400:d0::1ddb:7001: Invalid response from http://sojournersgame.com/.well-known/acme-challenge/bNLDaQeKrumv8a7KTSB9Dq6CkWnG4Tk-MgAL_MIB0NU: 404",
        "status": 403
      },
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/210085800307/cPC7mw",
      "token": "bNLDaQeKrumv8a7KTSB9Dq6CkWnG4Tk-MgAL_MIB0NU",
      "validationRecord": [
        {
          "url": "http://sojournersgame.com/.well-known/acme-challenge/bNLDaQeKrumv8a7KTSB9Dq6CkWnG4Tk-MgAL_MIB0NU",
          "hostname": "sojournersgame.com",
          "port": "80",
          "addressesResolved": [
            "198.199.74.230",
            "2604:a880:400:d0::1ddb:7001"
          ],
          "addressUsed": "2604:a880:400:d0::1ddb:7001"
        }
      ],
      "validated": "2023-03-11T21:48:28Z"
    }
  ]
}
2023-03-11 21:48:29,243:DEBUG:acme.client:Storing nonce: C878Uz4eNNOWK5NFcKVUZWnbYtTLt_jNYeNAQ1MrGAFuneU
2023-03-11 21:48:29,244:INFO:certbot._internal.auth_handler:Challenge failed for domain sojournersgame.com
2023-03-11 21:48:29,244:INFO:certbot._internal.auth_handler:http-01 challenge for sojournersgame.com
2023-03-11 21:48:29,245:DEBUG:certbot._internal.display.obj:Notifying user: 
Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:
  Domain: sojournersgame.com
  Type:   unauthorized
  Detail: 2604:a880:400:d0::1ddb:7001: Invalid response from http://sojournersgame.com/.well-known/acme-challenge/bNLDaQeKrumv8a7KTSB9Dq6CkWnG4Tk-MgAL_MIB0NU: 404

Hint: The Certificate Authority failed to verify the temporary Apache configuration changes made by Certbot. Ensure that the listed domains point to this Apache server and that it is accessible from the internet.

2023-03-11 21:48:29,248:DEBUG:certbot._internal.error_handler:Encountered exception:
Traceback (most recent call last):
  File "/snap/certbot/2836/lib/python3.8/site-packages/certbot/_internal/auth_handler.py", line 108, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, max_time_mins, best_effort)
  File "/snap/certbot/2836/lib/python3.8/site-packages/certbot/_internal/auth_handler.py", line 212, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.

2023-03-11 21:48:29,249:DEBUG:certbot._internal.error_handler:Calling registered functions
2023-03-11 21:48:29,249:INFO:certbot._internal.auth_handler:Cleaning up challenges
2023-03-11 21:48:29,536:DEBUG:certbot._internal.log:Exiting abnormally:
Traceback (most recent call last):
  File "/snap/certbot/2836/bin/certbot", line 8, in <module>
    sys.exit(main())
  File "/snap/certbot/2836/lib/python3.8/site-packages/certbot/main.py", line 19, in main
    return internal_main.main(cli_args)
  File "/snap/certbot/2836/lib/python3.8/site-packages/certbot/_internal/main.py", line 1864, in main
    return config.func(config, plugins)
  File "/snap/certbot/2836/lib/python3.8/site-packages/certbot/_internal/main.py", line 1597, in certonly
    lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
  File "/snap/certbot/2836/lib/python3.8/site-packages/certbot/_internal/main.py", line 141, in _get_and_save_cert
    lineage = le_client.obtain_and_enroll_certificate(domains, certname)
  File "/snap/certbot/2836/lib/python3.8/site-packages/certbot/_internal/client.py", line 516, in obtain_and_enroll_certificate
    cert, chain, key, _ = self.obtain_certificate(domains)
  File "/snap/certbot/2836/lib/python3.8/site-packages/certbot/_internal/client.py", line 428, in obtain_certificate
    orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
  File "/snap/certbot/2836/lib/python3.8/site-packages/certbot/_internal/client.py", line 496, in _get_order_and_authorizations
    authzr = self.auth_handler.handle_authorizations(orderr, self.config, best_effort)
  File "/snap/certbot/2836/lib/python3.8/site-packages/certbot/_internal/auth_handler.py", line 108, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, max_time_mins, best_effort)
  File "/snap/certbot/2836/lib/python3.8/site-packages/certbot/_internal/auth_handler.py", line 212, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.
2023-03-11 21:48:29,542:ERROR:certbot._internal.log:Some challenges have failed.

My web server is: Apache/2.4.56
The operating system my web server runs on is: Ubuntu 22.04
My hosting provider, if applicable, is: digitalocean
I can login to a root shell on my machine: yes
I'm using a control panel to manage my site: no
The version of my client is: 2.4.0

Details:

So I've tried disabling ufw, removing DNS for ipv6, checked Apache VirtualHost and .htaccess and appears fine, created a test text at /var/www/sojournersgame/.well-known/acme-challenge/test1234.txt file manually and accessed it successfully from the browser... feels like I'm running out of options here!

For context: I originally did certbot on a Wordpress site I've been running for some time, and it worked like a charm... but then I made another Wordpress site from scratch, and no matter what I try I just can't get certbot to succeed.

Failing site's Apache conf:

 <VirtualHost *:80>

    ServerName sojournersgame.com
    ServerAlias www.sojournersgame.com *.sojournersgame.com
    DocumentRoot /var/www/sojournersgame

    <Directory /var/www/sojournersgame>
        Options FollowSymLinks
        AllowOverride Limit Options FileInfo
        DirectoryIndex index.php
        Require all granted
    </Directory>

    <Directory /var/www/sojournersgame/wp-content>
        Options FollowSymLinks
        Require all granted
    </Directory>

    RewriteEngine Off
    RewriteCond %{SERVER_NAME} =*.sojournersgame.com [OR]
    RewriteCond %{SERVER_NAME} =www.sojournersgame.com [OR]
    RewriteCond %{SERVER_NAME} =sojournersgame.com
    RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]

</VirtualHost>

Only applicable .htaccess (in /var/www/sojournersgame/.htaccess)

# BEGIN WordPress
# The directives (lines) between "BEGIN WordPress" and "END WordPress" are
# dynamically generated, and should only be modified via WordPress filters.
# Any changes to the directives between these markers will be overwritten.
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

# END WordPress

php_value upload_max_filesize 1024M

(I tried setting RewriteEngine to Off here, but to no avail. I assume I don't have to run anything to update this change.)

Anyways, I'm stumped! What could the issue be? Thanks in advance.

I tried reproducing this myself and it seems like this directive here is the culprit:

Certbot's Apache plugin relies on mod_rewrite to do its magic. It does insert a RewriteEngine On at the beginning of your virtual host temporarily, but that gets undone by this line.

5 Likes

Ahhhhhh that fixed it! THANK YOU!!!!!!

2 Likes

For future reference...
This request lacks the "www" name:

This would get a cert with both names on it:

sudo certbot certonly \
-d sojournersgame.com -d www.sojournersgame.com \
--debug-challenges -v
3 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.