Certbot renew suddenly failing (Challenge failed for domain)

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: elm.kroitor.ca

I ran this command: certbot renew (manually because autorenew started failing)

It produced this output: Challenge failed for domain

My web server is (include version): nginx 1.20.1

The operating system my web server runs on is (include version): rocky 9.5

My hosting provider, if applicable, is: Linode

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): 3.1.0

Hi, autorenew has been working for more than a year on this server and I'm suddenly getting emails saying my cert will expire. Manually checking shows this to be true.
Running certbot renew manually results in:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/elm.kroitor.ca.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Certificate is due for renewal, auto-renewing...
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate for elm.kroitor.ca
Performing the following challenges:
http-01 challenge for elm.kroitor.ca
Using the webroot path /var/www/html for all unmatched domains.
Waiting for verification...
Challenge failed for domain elm.kroitor.ca
http-01 challenge for elm.kroitor.ca

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
  Domain: elm.kroitor.ca
  Type:   unauthorized
  Detail: 172.105.103.72: Invalid response from http://elm.kroitor.ca/.well-known/acme-challenge/OI84iS-9RdWjnhDy9ooQYPhh9KfTHJLQtBsTgijPrls: 404

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

Cleaning up challenges
Failed to renew certificate elm.kroitor.ca with error: Some challenges have failed.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
All renewals failed. The following certificates could not be renewed:
  /etc/letsencrypt/live/elm.kroitor.ca/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

There are no known changes to our underlying website or OS other than regular version updates.

I believe there are recent certbot there are recent changes about cert life and automatic email reminders, but I can't see if this issue is related to either of those...

Thanks for all help as always,
Paul

2

PS: I have the verbose log (as I ran it with -v) but it's a million miles long. I can post it if useful.

3

Is this the right webroot for this domain? Has it changed recently? I see roundcube responding on that fqdn.

2 Likes
4

I have no reason to believe it's changed (ever).
That directory is present and has only a robots.txt and an index.html. The later simply contains:

<html><head><meta HTTP-EQUIV="REFRESH" content="0; url=/mail/"></head></html>

I'm not familiar enough with html redirects to understand if that's kosher or not, but the only thing that's been done on the box in the last year or so are update scripts published for the server it runs (iredmail). I haven't read anything about changing the web root, and their update notes are very complete. They certainly would have noted something major like a change of web root.

5

I'm seeing something much worse...

1 Like
6

IPV6 was disabled when the server was installed a couple of years ago.

Is IPV6 mandatory for certbot now?

7

Let's Encrypt (usually) uses IPv6 if it's available (AAAA record present in DNS). Not sure why that's not happening here. :thinking:

Screenshot_20250509_152651_Samsung Internet
Screenshot_20250509_152651_Samsung Internet1440×1742 175 KB

Screenshot_20250509_152726_Samsung Internet
Screenshot_20250509_152726_Samsung Internet1440×1747 163 KB

2 Likes
8

LE will fallback to IPv4 if the IPv6 times out "cleanly". Based on your Let's Debug test that's what it looks like.

A bug in LE server has the error message showing the IPv4 address rather than the IPv6 one.

See the IPv4 retry details here: IPv6 Support - Let's Encrypt

That does not explain why they get a 404 Not Found. Just explaining this confusing part of it :slight_smile: They should remove the AAAA record from their DNS if they don't support IPv6. It could confuse other people trying to access their site.

4 Likes
9

There is an AAAA record in the name server, and it's set to the correct address. However, something was done on the machine itself to disable IPv6.

But nothing about this has changed. Did certbot change its algorithm for dealing with IPv6 failures?

10

The AAAA record is gone. Will it stay cached at your end for some hours or days?

11

Let's Encrypt handles the IPv6 failure, per @MikeMcQ's post above. That's my bad.

1 Like
12

Let me check...

:hourglass_not_done:

It has a time-to-live off 6 hours, but may propagate much more quickly than that.

You can enter elm.kroitor.ca here to check:

1 Like
13

It's still failing to renew, but I'll try again in six hours. I still have a few days before the current cert expires.
Thanks for your help (so far!),
Paul

14

That's not causing the failure. It's a side issue.

2 Likes
15

LE looks directly at the authoritative DNS Servers so as soon as all your DNS servers sync amongst themselves LE won't see it anymore.

The AAAA record still is there so either your servers take a long time or something more needs to be done .

Still, this isn't causing the 404. We should focus on that and not this.

See: https://unboundtest.com/m/AAAA/elm.kroitor.ca/CL5V7Z7K

4 Likes
16

Oh!
So how can I debug the actual failure? As above, I have the verbose log if that's helpful.

17

@MikeMcQ is on the right track.

@MikeMcQ:

I'm not seeing any HTTP redirects and LE doesn't follow equivs, so still curious why this is failing.

@Nummer378:

Tagging you into this thread in case you see room for some improvements for Let's Debug.

3 Likes
18

At the risk of spamming the forum, here's the log:

2025-05-09 18:13:38,030:DEBUG:certbot._internal.main:certbot version: 3.1.0
2025-05-09 18:13:38,030:DEBUG:certbot._internal.main:Location of certbot entry point: /bin/certbot
2025-05-09 18:13:38,030:DEBUG:certbot._internal.main:Arguments: ['-v']
2025-05-09 18:13:38,031:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2025-05-09 18:13:38,038:DEBUG:certbot._internal.log:Root logging level set at 20
2025-05-09 18:13:38,040:DEBUG:certbot._internal.display.obj:Notifying user: Processing /etc/letsencrypt/renewal/elm.kroitor.ca.conf
2025-05-09 18:13:38,041:DEBUG:certbot._internal.plugins.selection:Requested authenticator None and installer None
2025-05-09 18:13:38,063:DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): e5.o.lencr.org:80
2025-05-09 18:13:38,274:DEBUG:urllib3.connectionpool:http://e5.o.lencr.org:80 "POST / HTTP/1.1" 200 345
2025-05-09 18:13:38,274:DEBUG:certbot.ocsp:OCSP response for certificate /etc/letsencrypt/archive/elm.kroitor.ca/cert13.pem is signed by the certificate's issuer.
2025-05-09 18:13:38,276:DEBUG:certbot.ocsp:OCSP certificate status for /etc/letsencrypt/archive/elm.kroitor.ca/cert13.pem is: OCSPCertStatus.GOOD
2025-05-09 18:13:38,278:DEBUG:certbot._internal.storage:Should renew, less than 30 days before certificate expiry 2025-05-27 19:17:28 UTC.
2025-05-09 18:13:38,278:INFO:certbot._internal.renewal:Certificate is due for renewal, auto-renewing...
2025-05-09 18:13:38,278:DEBUG:certbot._internal.plugins.selection:Requested authenticator webroot and installer None
2025-05-09 18:13:38,279:DEBUG:certbot._internal.plugins.selection:Single candidate plugin: * webroot
Description: Saves the necessary validation files to a .well-known/acme-challenge/ directory within the nominated webroot path. A separate HTTP server must be running and serving files from the webroot path. HTTP challenge only (wildcards not supported).
Interfaces: Authenticator, Plugin
Entry point: EntryPoint(name='webroot', value='certbot._internal.plugins.webroot:Authenticator', group='certbot.plugins')
Initialized: <certbot._internal.plugins.webroot.Authenticator object at 0x7f04b1d15fd0>
Prep: True
2025-05-09 18:13:38,279:DEBUG:certbot._internal.plugins.selection:Selected authenticator <certbot._internal.plugins.webroot.Authenticator object at 0x7f04b1d15fd0> and installer None
2025-05-09 18:13:38,279:INFO:certbot._internal.plugins.selection:Plugins selected: Authenticator webroot, Installer None
2025-05-09 18:13:38,327:DEBUG:certbot._internal.main:Picked account: <Account(RegistrationResource(body=Registration(key=None, contact=(), agreement=None, status=None, terms_of_service_agreed=None, only_return_existing=None, external_account_binding=None), uri='https://acme-v02.api.letsencrypt.org/acme/acct/995245107', new_authzr_uri=None, terms_of_service=None), 0761b8b1367b5582f3e017c61aefc43a, Meta(creation_dt=datetime.datetime(2023, 3, 5, 17, 45, 39, tzinfo=<UTC>), creation_host='elm.kroitor.ca', register_to_eff=None))>
2025-05-09 18:13:38,327:DEBUG:acme.client:Sending GET request to https://acme-v02.api.letsencrypt.org/directory.
2025-05-09 18:13:38,328:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org:443
2025-05-09 18:13:38,576:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "GET /directory HTTP/1.1" 200 1012
2025-05-09 18:13:38,576:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Fri, 09 May 2025 18:13:38 GMT
Content-Type: application/json
Content-Length: 1012
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "TjtopBM5d-I": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
  "keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "profiles": {
      "classic": "https://letsencrypt.org/docs/profiles#classic",
      "shortlived": "https://letsencrypt.org/docs/profiles#shortlived (not yet generally available)",
      "tlsserver": "https://letsencrypt.org/docs/profiles#tlsserver"
    },
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.5-February-24-2025.pdf",
    "website": "https://letsencrypt.org"
  },
  "newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
  "renewalInfo": "https://acme-v02.api.letsencrypt.org/draft-ietf-acme-ari-03/renewalInfo",
  "revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert"
}
2025-05-09 18:13:38,577:DEBUG:certbot._internal.display.obj:Notifying user: Renewing an existing certificate for elm.kroitor.ca
2025-05-09 18:13:38,579:DEBUG:acme.client:Requesting fresh nonce
2025-05-09 18:13:38,579:DEBUG:acme.client:Sending HEAD request to https://acme-v02.api.letsencrypt.org/acme/new-nonce.
2025-05-09 18:13:38,625:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "HEAD /acme/new-nonce HTTP/1.1" 200 0
2025-05-09 18:13:38,625:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Fri, 09 May 2025 18:13:38 GMT
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: ww5-sb6NFmnwSpCllffQpM9Ezys1AMuM_5hX9DbuA2MKB055ws4
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800


2025-05-09 18:13:38,625:DEBUG:acme.client:Storing nonce: ww5-sb6NFmnwSpCllffQpM9Ezys1AMuM_5hX9DbuA2MKB055ws4
2025-05-09 18:13:38,625:DEBUG:acme.client:JWS payload:
b'{\n  "identifiers": [\n    {\n      "type": "dns",\n      "value": "elm.kroitor.ca"\n    }\n  ]\n}'
2025-05-09 18:13:38,627:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/new-order:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2FjY3QvOTk1MjQ1MTA3IiwgIm5vbmNlIjogInd3NS1zYjZORm1ud1NwQ2xsZmZRcE05RXp5czFBTXVNXzVoWDlEYnVBMk1LQjA1NXdzNCIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvbmV3LW9yZGVyIn0",
  "signature": "aAo0UHChcK9ywT0w-LnHbXsgQGjl_HEtVcMSldCpOIE0xQf_cib-JiNJrVdW7lcbIMRL2cqmGHS1S1QqZuKwgXVE9-lrw2587QLCksSI-8pwlBz7bu5SMji3HwS6xLgrmFDU31GlaU4x_8abd9kF2JKuq06cODNdbs9iLuKfgILJGSfqmTsUNu3dhaQ4xseZ25pBc1T33UNjVMBqwnWSGFP1Fg9UnYgOUQFHllq6eFvFQYXSgDdx-HVAIb0LU8RTcpwYqwaFLbeNg8-rkykhu0ho2u2zr9b-cljolAu3NdZJqCTfnsodBuT5O9xi9Or4YppZ3cGZmkqrFskXoY_W2w",
  "payload": "ewogICJpZGVudGlmaWVycyI6IFsKICAgIHsKICAgICAgInR5cGUiOiAiZG5zIiwKICAgICAgInZhbHVlIjogImVsbS5rcm9pdG9yLmNhIgogICAgfQogIF0KfQ"
}
2025-05-09 18:13:38,702:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "POST /acme/new-order HTTP/1.1" 201 346
2025-05-09 18:13:38,702:DEBUG:acme.client:Received response:
HTTP 201
Server: nginx
Date: Fri, 09 May 2025 18:13:38 GMT
Content-Type: application/json
Content-Length: 346
Connection: keep-alive
Boulder-Requester: 995245107
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
Location: https://acme-v02.api.letsencrypt.org/acme/order/995245107/382438269347
Replay-Nonce: QmouDpB2hcxbCDFoWK6oKDboWTt1NMG3M-x3prPo8Prls0QHQYU
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "status": "pending",
  "expires": "2025-05-16T18:13:38Z",
  "identifiers": [
    {
      "type": "dns",
      "value": "elm.kroitor.ca"
    }
  ],
  "authorizations": [
    "https://acme-v02.api.letsencrypt.org/acme/authz/995245107/517912165237"
  ],
  "finalize": "https://acme-v02.api.letsencrypt.org/acme/finalize/995245107/382438269347"
}
2025-05-09 18:13:38,702:DEBUG:acme.client:Storing nonce: QmouDpB2hcxbCDFoWK6oKDboWTt1NMG3M-x3prPo8Prls0QHQYU
2025-05-09 18:13:38,703:DEBUG:acme.client:JWS payload:
b''
2025-05-09 18:13:38,704:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/authz/995245107/517912165237:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2FjY3QvOTk1MjQ1MTA3IiwgIm5vbmNlIjogIlFtb3VEcEIyaGN4YkNERm9XSzZvS0Rib1dUdDFOTUczTS14M3ByUG84UHJsczBRSFFZVSIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYXV0aHovOTk1MjQ1MTA3LzUxNzkxMjE2NTIzNyJ9",
  "signature": "qdl1mJVqbAVZjkPwBNupvK17u4GKGoZtGkSwlPoXY2SZ1r70BL4LznDINd9M4c734pRiU15HJS5AEnaAsj1V5kupLe3wOPiHrDHfcxILZ6jc8kli9T0SSojGpilZpjm0A61C4ziqSJsTcar-EFx2JEtQKonguOUuMiLsePyPUxR3M24PMcJ7dqy-enRntIdeFqw6kI-TV-t1Z7wejuPHdYTPcYQO8BemxvLy4_IDzvVS2mWWTyiRAWn1lQN4lXJzmRWlk0AjCn0k_xne8GMaoM79w4PlgHqBBIQ7s3VyboktxU22TMn0BBxJdN_cD8ju_kIGN13QDNy-SRmGuRZy9A",
  "payload": ""
}
2025-05-09 18:13:38,754:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "POST /acme/authz/995245107/517912165237 HTTP/1.1" 200 819
2025-05-09 18:13:38,754:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Fri, 09 May 2025 18:13:38 GMT
Content-Type: application/json
Content-Length: 819
Connection: keep-alive
Boulder-Requester: 995245107
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: ww5-sb6NSn0iIcGu4SHYqTLfMwcjVtKVKy0oYGamEctbKj_V-_c
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "elm.kroitor.ca"
  },
  "status": "pending",
  "expires": "2025-05-16T18:13:38Z",
  "challenges": [
    {
      "type": "dns-01",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall/995245107/517912165237/RWXe7w",
      "status": "pending",
      "token": "nkljeLdk3W8SnylbPrvjD8t5FtGNMEm9rzNBABIuVz8"
    },
    {
      "type": "http-01",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall/995245107/517912165237/WnnF8Q",
      "status": "pending",
      "token": "nkljeLdk3W8SnylbPrvjD8t5FtGNMEm9rzNBABIuVz8"
    },
    {
      "type": "tls-alpn-01",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall/995245107/517912165237/8eT9JA",
      "status": "pending",
      "token": "nkljeLdk3W8SnylbPrvjD8t5FtGNMEm9rzNBABIuVz8"
    }
  ]
}
2025-05-09 18:13:38,754:DEBUG:acme.client:Storing nonce: ww5-sb6NSn0iIcGu4SHYqTLfMwcjVtKVKy0oYGamEctbKj_V-_c
2025-05-09 18:13:38,755:INFO:certbot._internal.auth_handler:Performing the following challenges:
2025-05-09 18:13:38,755:INFO:certbot._internal.auth_handler:http-01 challenge for elm.kroitor.ca
2025-05-09 18:13:38,755:INFO:certbot._internal.plugins.webroot:Using the webroot path /var/www/html for all unmatched domains.
2025-05-09 18:13:38,755:DEBUG:certbot._internal.plugins.webroot:Creating root challenges validation dir at /var/www/html/.well-known/acme-challenge
2025-05-09 18:13:38,757:DEBUG:certbot._internal.plugins.webroot:Attempting to save validation to /var/www/html/.well-known/acme-challenge/nkljeLdk3W8SnylbPrvjD8t5FtGNMEm9rzNBABIuVz8
2025-05-09 18:13:38,757:DEBUG:acme.client:JWS payload:
b'{}'
2025-05-09 18:13:38,759:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/chall/995245107/517912165237/WnnF8Q:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2FjY3QvOTk1MjQ1MTA3IiwgIm5vbmNlIjogInd3NS1zYjZOU24waUljR3U0U0hZcVRMZk13Y2pWdEtWS3kwb1lHYW1FY3RiS2pfVi1fYyIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvY2hhbGwvOTk1MjQ1MTA3LzUxNzkxMjE2NTIzNy9Xbm5GOFEifQ",
  "signature": "H1kC0Ed3vZQrnhPpB-q6xEfIbeVMscy2d60KRdoqEDBeBAA5z_iK5Un1BbCBnespdv_n40QReQ4b5aha6le0wzJ3lVRECdfyXTes0a59H-P1FPII-PjdrVYIpBtjBDaaO_5Bn4tMZx-9_PRe_1eIZwZ0xUErTpzPgiCd8ahJQZxzkja_9rQ51haY7nkSQ9iUeOLsDuHP1qhuj_nttX3hthlofYkKZmgXMLbTiyiguTRQaOrHz_7Q1Qzvlikb5r1kE5tgzDXYlEgWJqPbjyn-KQNJqA4FLSt8YYBTfui5SdZnhWPhBD0bgPhWhOiNruR3njInrguVRaG5rKLYqThRaA",
  "payload": "e30"
}
2025-05-09 18:13:38,813:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "POST /acme/chall/995245107/517912165237/WnnF8Q HTTP/1.1" 200 194
2025-05-09 18:13:38,813:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Fri, 09 May 2025 18:13:38 GMT
Content-Type: application/json
Content-Length: 194
Connection: keep-alive
Boulder-Requester: 995245107
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index", <https://acme-v02.api.letsencrypt.org/acme/authz/995245107/517912165237>;rel="up"
Location: https://acme-v02.api.letsencrypt.org/acme/chall/995245107/517912165237/WnnF8Q
Replay-Nonce: ww5-sb6N38f1n1Dyjw9qE8pY0mx8bJRkzLUrZEwxXPH0ynCYeSQ
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "type": "http-01",
  "url": "https://acme-v02.api.letsencrypt.org/acme/chall/995245107/517912165237/WnnF8Q",
  "status": "pending",
  "token": "nkljeLdk3W8SnylbPrvjD8t5FtGNMEm9rzNBABIuVz8"
}
2025-05-09 18:13:38,814:DEBUG:acme.client:Storing nonce: ww5-sb6N38f1n1Dyjw9qE8pY0mx8bJRkzLUrZEwxXPH0ynCYeSQ
2025-05-09 18:13:38,814:INFO:certbot._internal.auth_handler:Waiting for verification...
2025-05-09 18:13:39,815:DEBUG:acme.client:JWS payload:
b''
2025-05-09 18:13:39,817:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/authz/995245107/517912165237:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2FjY3QvOTk1MjQ1MTA3IiwgIm5vbmNlIjogInd3NS1zYjZOMzhmMW4xRHlqdzlxRThwWTBteDhiSlJrekxVclpFd3hYUEgweW5DWWVTUSIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYXV0aHovOTk1MjQ1MTA3LzUxNzkxMjE2NTIzNyJ9",
  "signature": "h3dbkhdzZyd8tQ_JKGEZy8AVHyG2O0SQlDhOwM7eo1HyBHXB6A9IDnBVaf5dF8uC9nfHR33r9-1J6gl91BhPSNwmB3wVQXlTFnrt3Kz3ciN2EwdT1WD7DVPqnImJWsvrclKgjmt2Gu5SNd_eLMPul74EYROmztRjuQBL7RxY1u842yvbyXgg6x_r3X3m4RVWgpAXE2PiQvobIfGSyFL3_E-SGKH1-G6OKTjaUzfUB_cXuUtKyrExht_iVHIzxtvo9IK8A3-7EeKKUnhqvsCHCioXwAR-Znmkf1ldtgyXOUixBwxIJn0xTxBcEk8QlVuZO3I1wn5zSc2ObllvkiPqxg",
  "payload": ""
}
2025-05-09 18:13:39,865:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "POST /acme/authz/995245107/517912165237 HTTP/1.1" 200 1457
2025-05-09 18:13:39,865:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Fri, 09 May 2025 18:13:39 GMT
Content-Type: application/json
Content-Length: 1457
Connection: keep-alive
Boulder-Requester: 995245107
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: ww5-sb6NOba5uImbgJnYqzZD-A6THDLGQZHfXc3Ds4d_nYmhw2E
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "elm.kroitor.ca"
  },
  "status": "invalid",
  "expires": "2025-05-16T18:13:38Z",
  "challenges": [
    {
      "type": "http-01",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall/995245107/517912165237/WnnF8Q",
      "status": "invalid",
      "validated": "2025-05-09T18:13:38Z",
      "error": {
        "type": "urn:ietf:params:acme:error:unauthorized",
        "detail": "172.105.103.72: Invalid response from http://elm.kroitor.ca/.well-known/acme-challenge/nkljeLdk3W8SnylbPrvjD8t5FtGNMEm9rzNBABIuVz8: 404",
        "status": 403
      },
      "token": "nkljeLdk3W8SnylbPrvjD8t5FtGNMEm9rzNBABIuVz8",
      "validationRecord": [
        {
          "url": "http://elm.kroitor.ca/.well-known/acme-challenge/nkljeLdk3W8SnylbPrvjD8t5FtGNMEm9rzNBABIuVz8",
          "hostname": "elm.kroitor.ca",
          "port": "80",
          "addressesResolved": [
            "172.105.103.72",
            "2600:3c04::f03c:93ff:fe9f:2f70"
          ],
          "addressUsed": "2600:3c04::f03c:93ff:fe9f:2f70"
        },
        {
          "url": "http://elm.kroitor.ca/.well-known/acme-challenge/nkljeLdk3W8SnylbPrvjD8t5FtGNMEm9rzNBABIuVz8",
          "hostname": "elm.kroitor.ca",
          "port": "80",
          "addressesResolved": [
            "172.105.103.72",
            "2600:3c04::f03c:93ff:fe9f:2f70"
          ],
          "addressUsed": "172.105.103.72"
        }
      ]
    }
  ]
}
2025-05-09 18:13:39,865:DEBUG:acme.client:Storing nonce: ww5-sb6NOba5uImbgJnYqzZD-A6THDLGQZHfXc3Ds4d_nYmhw2E
2025-05-09 18:13:39,866:INFO:certbot._internal.auth_handler:Challenge failed for domain elm.kroitor.ca
2025-05-09 18:13:39,866:INFO:certbot._internal.auth_handler:http-01 challenge for elm.kroitor.ca
2025-05-09 18:13:39,866:DEBUG:certbot._internal.display.obj:Notifying user:
Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
  Domain: elm.kroitor.ca
  Type:   unauthorized
  Detail: 172.105.103.72: Invalid response from http://elm.kroitor.ca/.well-known/acme-challenge/nkljeLdk3W8SnylbPrvjD8t5FtGNMEm9rzNBABIuVz8: 404

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

2025-05-09 18:13:39,867:DEBUG:certbot._internal.error_handler:Encountered exception:
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/certbot/_internal/auth_handler.py", line 108, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, max_time_mins, best_effort)
  File "/usr/lib/python3.9/site-packages/certbot/_internal/auth_handler.py", line 212, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.

2025-05-09 18:13:39,867:DEBUG:certbot._internal.error_handler:Calling registered functions
2025-05-09 18:13:39,867:INFO:certbot._internal.auth_handler:Cleaning up challenges
2025-05-09 18:13:39,867:DEBUG:certbot._internal.plugins.webroot:Removing /var/www/html/.well-known/acme-challenge/nkljeLdk3W8SnylbPrvjD8t5FtGNMEm9rzNBABIuVz8
2025-05-09 18:13:39,867:DEBUG:certbot._internal.plugins.webroot:All challenges cleaned up
2025-05-09 18:13:39,868:ERROR:certbot._internal.renewal:Failed to renew certificate elm.kroitor.ca with error: Some challenges have failed.
2025-05-09 18:13:39,870:DEBUG:certbot._internal.renewal:Traceback was:
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/certbot/_internal/renewal.py", line 540, in handle_renewal_request
    main.renew_cert(lineage_config, plugins, renewal_candidate)
  File "/usr/lib/python3.9/site-packages/certbot/_internal/main.py", line 1529, in renew_cert
    renewed_lineage = _get_and_save_cert(le_client, config, lineage=lineage)
  File "/usr/lib/python3.9/site-packages/certbot/_internal/main.py", line 130, in _get_and_save_cert
    renewal.renew_cert(config, domains, le_client, lineage)
  File "/usr/lib/python3.9/site-packages/certbot/_internal/renewal.py", line 399, in renew_cert
    new_cert, new_chain, new_key, _ = le_client.obtain_certificate(domains, new_key)
  File "/usr/lib/python3.9/site-packages/certbot/_internal/client.py", line 429, in obtain_certificate
    orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
  File "/usr/lib/python3.9/site-packages/certbot/_internal/client.py", line 497, in _get_order_and_authorizations
    authzr = self.auth_handler.handle_authorizations(orderr, self.config, best_effort)
  File "/usr/lib/python3.9/site-packages/certbot/_internal/auth_handler.py", line 108, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, max_time_mins, best_effort)
  File "/usr/lib/python3.9/site-packages/certbot/_internal/auth_handler.py", line 212, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.

2025-05-09 18:13:39,871:DEBUG:certbot._internal.display.obj:Notifying user:
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2025-05-09 18:13:39,871:ERROR:certbot._internal.renewal:All renewals failed. The following certificates could not be renewed:
2025-05-09 18:13:39,871:ERROR:certbot._internal.renewal:  /etc/letsencrypt/live/elm.kroitor.ca/fullchain.pem (failure)
2025-05-09 18:13:39,871:DEBUG:certbot._internal.display.obj:Notifying user: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2025-05-09 18:13:39,872:DEBUG:certbot._internal.log:Exiting abnormally:
Traceback (most recent call last):
  File "/bin/certbot", line 8, in <module>
    sys.exit(main())
  File "/usr/lib/python3.9/site-packages/certbot/main.py", line 19, in main
    return internal_main.main(cli_args)
  File "/usr/lib/python3.9/site-packages/certbot/_internal/main.py", line 1873, in main
    return config.func(config, plugins)
  File "/usr/lib/python3.9/site-packages/certbot/_internal/main.py", line 1621, in renew
    renewed_domains, failed_domains = renewal.handle_renewal_request(config)
  File "/usr/lib/python3.9/site-packages/certbot/_internal/renewal.py", line 568, in handle_renewal_request
    raise errors.Error(
certbot.errors.Error: 1 renew failure(s), 0 parse failure(s)
2025-05-09 18:13:39,872:ERROR:certbot._internal.log:1 renew failure(s), 0 parse failure(s)
[root@elm ~]#
2 Likes
19

Would you show the entire active nginx config shown by this command?

sudo nginx -T

Ideally upload the config.txt file

sudo nginx -T >config.txt

An upper case T is essential. The output will be very long.

Or, at least show the nginx server block for this domain.

UPDATE: Sadly, the certbot log is not very helpful for 404 errors.

3 Likes
20

Not really. The fallback from Let's Encrypt on IPv6 failures is a band-aid at best, and it won't work properly in all cases. Red-flagging an unreachable AAAA record is the proper thing IMHO. If IPv6 is not intended, it should be clearly marked as such.

3 Likes