Unauthorized - Invalid Response 400

Overnight my entire LAMP stack is no longer renewing certs. I have many virtual hosts on the same IP address but never had a problem.

Here is a sample one I am currently experiencing issues with.

Domain: sdmx2024.adriacongrex.it
Type: unauthorized
Detail: 194.243.14.157: Invalid response from http://sdmx2024.adriacongrex.it/.well-known/acme-challenge/i9SgfbZHr7sHN-NFFXpsoLiRqqvFaW0m6Sq3AjbnQzA: 400

I ran this command: certbot renew --cert-name sdmx2024.adriacongrex.it --dry-run

Here is the entire debug log:

2025-04-07 11:19:30,277:DEBUG:certbot._internal.main:certbot version: 1.12.0
2025-04-07 11:19:30,278:DEBUG:certbot._internal.main:Location of certbot entry point: /usr/bin/certbot
2025-04-07 11:19:30,278:DEBUG:certbot._internal.main:Arguments: ['--cert-name', 'sdmx2024.adriacongrex.it', '--dry-run']
2025-04-07 11:19:30,278:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#w>
2025-04-07 11:19:30,297:DEBUG:certbot._internal.log:Root logging level set at 20
2025-04-07 11:19:30,297:INFO:certbot._internal.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2025-04-07 11:19:30,298:DEBUG:certbot.display.util:Notifying user: Processing /etc/letsencrypt/renewal/sdmx2024.adriacongrex.it.conf
2025-04-07 11:19:30,312:DEBUG:certbot._internal.plugins.selection:Requested authenticator <certbot._internal.cli.cli_utils._Default object at 0x7f11aa99b490> and installer <certbot._internal.cli.cli_utils.>
2025-04-07 11:19:30,312:DEBUG:certbot._internal.cli:Var dry_run=True (set by user).
2025-04-07 11:19:30,312:DEBUG:certbot._internal.cli:Var server={'staging', 'dry_run'} (set by user).
2025-04-07 11:19:30,312:DEBUG:certbot._internal.cli:Var dry_run=True (set by user).
2025-04-07 11:19:30,312:DEBUG:certbot._internal.cli:Var server={'staging', 'dry_run'} (set by user).
2025-04-07 11:19:30,312:DEBUG:certbot._internal.cli:Var account={'server'} (set by user).
2025-04-07 11:19:30,323:DEBUG:certbot._internal.storage:Should renew, less than 30 days before certificate expiry 2025-04-01 18:24:23 UTC.
2025-04-07 11:19:30,324:INFO:certbot._internal.renewal:Cert is due for renewal, auto-renewing...
2025-04-07 11:19:30,324:DEBUG:certbot._internal.plugins.selection:Requested authenticator apache and installer apache
2025-04-07 11:19:30,422:DEBUG:certbot_apache._internal.configurator:Apache version is 2.4.62
2025-04-07 11:19:30,942:DEBUG:certbot._internal.plugins.selection:Single candidate plugin: * apache
Description: Apache Web Server plugin
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: apache = certbot_apache._internal.entrypoint:ENTRYPOINT
Initialized: <certbot_apache._internal.override_debian.DebianConfigurator object at 0x7f11aa9d6ca0>
Prep: True
2025-04-07 11:19:30,944:DEBUG:certbot._internal.plugins.selection:Single candidate plugin: * apache
Description: Apache Web Server plugin
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: apache = certbot_apache._internal.entrypoint:ENTRYPOINT
Initialized: <certbot_apache._internal.override_debian.DebianConfigurator object at 0x7f11aa9d6ca0>
Prep: True
2025-04-07 11:19:30,944:DEBUG:certbot._internal.plugins.selection:Selected authenticator <certbot_apache._internal.override_debian.DebianConfigurator object at 0x7f11aa9d6ca0> and installer <certbot_apache>
2025-04-07 11:19:30,944:INFO:certbot._internal.plugins.selection:Plugins selected: Authenticator apache, Installer apache
2025-04-07 11:19:30,957:DEBUG:certbot._internal.main:Picked account: <Account(RegistrationResource(body=Registration(key=None, contact=(), agreement=None, status=None, terms_of_service_agreed=None, only_re>
2025-04-07 11:19:30,958:DEBUG:acme.client:Sending GET request to https://acme-staging-v02.api.letsencrypt.org/directory.
2025-04-07 11:19:30,961:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): acme-staging-v02.api.letsencrypt.org:443
2025-04-07 11:19:31,415:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "GET /directory HTTP/1.1" 200 1116
2025-04-07 11:19:31,416:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Mon, 07 Apr 2025 09:19:31 GMT
Content-Type: application/json
Content-Length: 1116
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "K52LuNrEF8Q": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
  "keyChange": "https://acme-staging-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "profiles": {
      "classic": "https://letsencrypt.org/docs/profiles#classic",
      "shortlived": "https://letsencrypt.org/docs/profiles#shortlived (not yet generally available)",
      "tlsserver": "https://letsencrypt.org/docs/profiles#tlsserver (not yet generally available)"
    },
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.5-February-24-2025.pdf",
    "website": "https://letsencrypt.org/docs/staging-environment/"
  },
  "newAccount": "https://acme-staging-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-staging-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-staging-v02.api.letsencrypt.org/acme/new-order",
  "renewalInfo": "https://acme-staging-v02.api.letsencrypt.org/draft-ietf-acme-ari-03/renewalInfo",
  "revokeCert": "https://acme-staging-v02.api.letsencrypt.org/acme/revoke-cert"
}
2025-04-07 11:19:31,417:DEBUG:certbot.display.util:Notifying user: Simulating renewal of an existing certificate for sdmx2024.adriacongrex.it
2025-04-07 11:19:31,551:DEBUG:acme.client:Requesting fresh nonce
2025-04-07 11:19:31,551:DEBUG:acme.client:Sending HEAD request to https://acme-staging-v02.api.letsencrypt.org/acme/new-nonce.
2025-04-07 11:19:31,703:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "HEAD /acme/new-nonce HTTP/1.1" 200 0
2025-04-07 11:19:31,704:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Mon, 07 Apr 2025 09:19:31 GMT
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: cFzjDXVwLMriwRRAB6FzOHGfr9l4Z582gU-baTzQq85vbMDZtIc
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800


2025-04-07 11:19:31,704:DEBUG:acme.client:Storing nonce: cFzjDXVwLMriwRRAB6FzOHGfr9l4Z582gU-baTzQq85vbMDZtIc

2025-04-07 11:19:31,705:DEBUG:acme.client:JWS payload:
b'{\n  "identifiers": [\n    {\n      "type": "dns",\n      "value": "sdmx2024.adriacongrex.it"\n    }\n  ]\n}'
2025-04-07 11:19:31,708:DEBUG:acme.client:Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/new-order:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS1zdGFnaW5nLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYWNjdC8xNzIxOTU3MDMiLCAibm9uY2UiOiAiY0Z6akRYVndMTXJpd1JSQUI2RnpPSEdmcjlsNFo1ODJnVS1iYVR6U>
  "signature": "PHnS2WRiytdLEjBR1BHKu0P0tnpkNywd8zqX7pB4L4Zlvc-ThMLa0Me2vDpwYNFcdG9jR-0-fisGUk4fS7OEEYE3_qLvX2ITVUbinZtDf5BjV0x3_ZDBo5tXFEXRcIljlf4wkbD8xLMUOkolOTauuFggPWWSWTRNWSrNTpIhijAM_qVC5C1Dg-Walexsf>
  "payload": "ewogICJpZGVudGlmaWVycyI6IFsKICAgIHsKICAgICAgInR5cGUiOiAiZG5zIiwKICAgICAgInZhbHVlIjogInNkbXgyMDI0LmFkcmlhY29uZ3JleC5pdCIKICAgIH0KICBdCn0"
}
2025-04-07 11:19:31,889:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/new-order HTTP/1.1" 201 370
2025-04-07 11:19:31,890:DEBUG:acme.client:Received response:
HTTP 201
Server: nginx
Date: Mon, 07 Apr 2025 09:19:31 GMT
Content-Type: application/json
Content-Length: 370
Connection: keep-alive
Boulder-Requester: 172195703
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
Location: https://acme-staging-v02.api.letsencrypt.org/acme/order/172195703/23721928664
Replay-Nonce: cFzjDXVwkNlCWYlNLGyERN5AvRznJQbxA_tozrMPlIt7ZxrdkR8
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "status": "pending",
  "expires": "2025-04-14T09:19:31Z",
  "identifiers": [
    {
      "type": "dns",
      "value": "sdmx2024.adriacongrex.it"
    }
  ],
  "authorizations": [
    "https://acme-staging-v02.api.letsencrypt.org/acme/authz/172195703/16706932224"
  ],
  "finalize": "https://acme-staging-v02.api.letsencrypt.org/acme/finalize/172195703/23721928664"
}
2025-04-07 11:19:31,891:DEBUG:acme.client:Storing nonce: cFzjDXVwkNlCWYlNLGyERN5AvRznJQbxA_tozrMPlIt7ZxrdkR8
2025-04-07 11:19:31,891:DEBUG:acme.client:JWS payload:
b''
2025-04-07 11:19:31,894:DEBUG:acme.client:Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/authz/172195703/16706932224:

{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS1zdGFnaW5nLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYWNjdC8xNzIxOTU3MDMiLCAibm9uY2UiOiAiY0Z6akRYVndrTmxDV1lsTkxHeUVSTjVBdlJ6bkpRYnhBX3RvenJNU>
  "signature": "JNcbUktff2VjLXeMyTPVbED3g1-CX9B6ryMlv7YHvoPKtP3juvmv-mEN9inkLG-iLFiqYZxymsF2FBtMBSKIOwg--AUbJJO2C4I1gK0fBRcTfHVRJGDgq6wCeus57keeYCDFumkpZYe2uDT1vyVLu0gO5FHM9KtCt5EsqIULuL7-pj0V3p6dNOF_SRNsf>
  "payload": ""
}
2025-04-07 11:19:32,053:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/authz/172195703/16706932224 HTTP/1.1" 200 850
2025-04-07 11:19:32,054:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Mon, 07 Apr 2025 09:19:31 GMT
Content-Type: application/json
Content-Length: 850
Connection: keep-alive
Boulder-Requester: 172195703
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: 2tykYePSA28zFzGmpdLW-wzH2AzP9GiecAgNOpvRcGIWEINaMQU
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "sdmx2024.adriacongrex.it"
  },
  "status": "pending",
  "expires": "2025-04-14T09:19:31Z",
  "challenges": [
    {
      "type": "dns-01",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall/172195703/16706932224/-evcVA",
      "status": "pending",
      "token": "i9SgfbZHr7sHN-NFFXpsoLiRqqvFaW0m6Sq3AjbnQzA"
    },
    {
      "type": "http-01",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall/172195703/16706932224/PlCq3w",
      "status": "pending",
      "token": "i9SgfbZHr7sHN-NFFXpsoLiRqqvFaW0m6Sq3AjbnQzA"
    },
    {
      "type": "tls-alpn-01",

"url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall/172195703/16706932224/3cTX2Q",
      "status": "pending",
      "token": "i9SgfbZHr7sHN-NFFXpsoLiRqqvFaW0m6Sq3AjbnQzA"
    }
  ]
}
2025-04-07 11:19:32,055:DEBUG:acme.client:Storing nonce: 2tykYePSA28zFzGmpdLW-wzH2AzP9GiecAgNOpvRcGIWEINaMQU
2025-04-07 11:19:32,055:INFO:certbot._internal.auth_handler:Performing the following challenges:
2025-04-07 11:19:32,056:INFO:certbot._internal.auth_handler:http-01 challenge for sdmx2024.adriacongrex.it
2025-04-07 11:19:32,081:DEBUG:certbot_apache._internal.http_01:Adding a temporary challenge validation Include for name: sdmx2024.adriacongrex.it in: /etc/apache2/sites-enabled/sdmx2024.adriacongrex.it.conf
2025-04-07 11:19:32,081:DEBUG:certbot_apache._internal.http_01:Adding a temporary challenge validation Include for name: sdmx2024.adriacongrex.it in: /etc/apache2/sites-enabled/sdmx2024.adriacongrex.it.conf
2025-04-07 11:19:32,082:DEBUG:certbot_apache._internal.http_01:writing a pre config file with text:
         RewriteEngine on
        RewriteRule ^/\.well-known/acme-challenge/([A-Za-z0-9-_=]+)$ /var/lib/letsencrypt/http_challenges/$1 [END]

2025-04-07 11:19:32,082:DEBUG:certbot_apache._internal.http_01:writing a post config file with text:
         <Directory /var/lib/letsencrypt/http_challenges>
            Require all granted
        </Directory>
        <Location /.well-known/acme-challenge>
            Require all granted
        </Location>

2025-04-07 11:19:32,142:DEBUG:certbot.reverter:Creating backup of /etc/apache2/sites-enabled/sdmx2024.adriacongrex.it.conf
2025-04-07 11:19:35,472:INFO:certbot._internal.auth_handler:Waiting for verification...
2025-04-07 11:19:35,473:DEBUG:acme.client:JWS payload:
b'{}'
2025-04-07 11:19:35,477:DEBUG:acme.client:Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/chall/172195703/16706932224/PlCq3w:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS1zdGFnaW5nLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYWNjdC8xNzIxOTU3MDMiLCAibm9uY2UiOiAiMnR5a1llUFNBMjh6RnpHbXBkTFctd3pIMkF6UDlHaWVjQWdOT3B2U>
  "signature": "DogUWDcv5d7loknXIbz0fE4FH_kqL1YpyNdwodbk6899EoOqM27Hx-WdDjjcV1-hTBbdtoQFOc7Q2aPB-bSXWWCtVTfIW5bqb3i89I4alw2Xok85bsYsH_MIQGjLVErfNr-loOAmzBo5ud5EbpPU5UiWGU6Od7pMw4XPiMrnuaf7XeOpsP5scuRk21fuc>
  "payload": "e30"
}
2025-04-07 11:19:35,638:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/chall/172195703/16706932224/PlCq3w HTTP/1.1" 200 201
2025-04-07 11:19:35,639:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Mon, 07 Apr 2025 09:19:35 GMT
Content-Type: application/json
Content-Length: 201
Connection: keep-alive
Boulder-Requester: 172195703
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index", <https://acme-staging-v02.api.letsencrypt.org/acme/authz/172195703/16706932224>;rel="up"
Location: https://acme-staging-v02.api.letsencrypt.org/acme/chall/172195703/16706932224/PlCq3w
Replay-Nonce: 2tykYePSmbiCHV_hYcLMT6spRnYNmVacaPadek7nETHf2eHWd4I
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "type": "http-01",
  "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall/172195703/16706932224/PlCq3w",
  "status": "pending",
  "token": "i9SgfbZHr7sHN-NFFXpsoLiRqqvFaW0m6Sq3AjbnQzA"
}
2025-04-07 11:19:35,639:DEBUG:acme.client:Storing nonce: 2tykYePSmbiCHV_hYcLMT6spRnYNmVacaPadek7nETHf2eHWd4I
2025-04-07 11:19:36,641:DEBUG:acme.client:JWS payload:
b''
2025-04-07 11:19:36,644:DEBUG:acme.client:Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/authz/172195703/16706932224:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS1zdGFnaW5nLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYWNjdC8xNzIxOTU3MDMiLCAibm9uY2UiOiAiMnR5a1llUFNtYmlDSFZfaFljTE1UNnNwUm5ZTm1WYWNhUGFkZWs3b>
  "signature": "AItmVSrXD0Xm2qsuGkCf3kxD_KOLO8l0htNQ5FUzZF7eALv6HSI9Ch5uWXFAXpXGMAPkdy7bN0Wp2UmxZOCItDY9o73wvFGWbTddl61GoRvKIIzn0qCzh1yNIfiKfq1L7rpR63MLc-v3epkTz9rc334sU-ihLr_YVe45Ih59AQIvAWH2zkJ5ThGu6bfes>
  "payload": ""
}
2025-04-07 11:19:36,799:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/authz/172195703/16706932224 HTTP/1.1" 200 850
2025-04-07 11:19:36,800:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Mon, 07 Apr 2025 09:19:36 GMT
Content-Type: application/json
Content-Length: 850
Connection: keep-alive
Boulder-Requester: 172195703
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: 2tykYePStHsIGlcGLHY7PtNI2SX1z2MwoE1PUdnILeQylNhZw0c
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "sdmx2024.adriacongrex.it"
  },
 "status": "pending",
  "expires": "2025-04-14T09:19:31Z",
  "challenges": [
    {
      "type": "dns-01",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall/172195703/16706932224/-evcVA",
      "status": "pending",
      "token": "i9SgfbZHr7sHN-NFFXpsoLiRqqvFaW0m6Sq3AjbnQzA"
    },
    {
      "type": "http-01",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall/172195703/16706932224/PlCq3w",
      "status": "pending",
      "token": "i9SgfbZHr7sHN-NFFXpsoLiRqqvFaW0m6Sq3AjbnQzA"
    },
    {
      "type": "tls-alpn-01",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall/172195703/16706932224/3cTX2Q",
      "status": "pending",
      "token": "i9SgfbZHr7sHN-NFFXpsoLiRqqvFaW0m6Sq3AjbnQzA"
    }
  ]
}
2025-04-07 11:19:36,800:DEBUG:acme.client:Storing nonce: 2tykYePStHsIGlcGLHY7PtNI2SX1z2MwoE1PUdnILeQylNhZw0c
2025-04-07 11:19:39,804:DEBUG:acme.client:JWS payload:
b''
2025-04-07 11:19:39,807:DEBUG:acme.client:Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/authz/172195703/16706932224:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS1zdGFnaW5nLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYWNjdC8xNzIxOTU3MDMiLCAibm9uY2UiOiAiMnR5a1llUFN0SHNJR2xjR0xIWTdQdE5JMlNYMXoyTXdvRTFQVWRuS>
  "signature": "z5EvPTrfluffc6AAdPqYMjA7LAw0S_g7OkSbOZHGK6DGCNSabM53Y2fDQuYmTfjq3SlumvHdSq9dGXoItcO3sYTnl20Dh3_1jp_AIaMRrT-dKdYdWp8fEx0AQvzW5y8dhmugZnW0mWAjQYteQ1ZTtYqzokIHcdhd6eCSGGQgCpk5kKPKZHDUyYgADMfcS>
  "payload": ""
}
2025-04-07 11:19:39,965:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/authz/172195703/16706932224 HTTP/1.1" 200 1081
2025-04-07 11:19:39,966:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Mon, 07 Apr 2025 09:19:39 GMT
Content-Type: application/json
Content-Length: 1081
Connection: keep-alive
Boulder-Requester: 172195703
Cache-Control: public, max-age=0, no-cacheLink: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: cFzjDXVwWUO7Gm3MRNz2K1DqvrOYCbfUMI_2Fp89JCt5GrLhZ5Q
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "sdmx2024.adriacongrex.it"
  },
  "status": "invalid",
  "expires": "2025-04-14T09:19:31Z",
  "challenges": [
    {
      "type": "http-01",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall/172195703/16706932224/PlCq3w",
      "status": "invalid",
      "validated": "2025-04-07T09:19:35Z",
      "error": {
        "type": "urn:ietf:params:acme:error:unauthorized",
        "detail": "194.243.14.157: Invalid response from http://sdmx2024.adriacongrex.it/.well-known/acme-challenge/i9SgfbZHr7sHN-NFFXpsoLiRqqvFaW0m6Sq3AjbnQzA: 400",
        "status": 403
      },
      "token": "i9SgfbZHr7sHN-NFFXpsoLiRqqvFaW0m6Sq3AjbnQzA",
      "validationRecord": [
        {
          "url": "http://sdmx2024.adriacongrex.it/.well-known/acme-challenge/i9SgfbZHr7sHN-NFFXpsoLiRqqvFaW0m6Sq3AjbnQzA",
          "hostname": "sdmx2024.adriacongrex.it",
          "port": "80",
          "addressesResolved": [
            "194.243.14.157"
          ],
          "addressUsed": "194.243.14.157"
        }
      ]
    }
  ]
}
2025-04-07 11:19:39,966:DEBUG:acme.client:Storing nonce: cFzjDXVwWUO7Gm3MRNz2K1DqvrOYCbfUMI_2Fp89JCt5GrLhZ5Q
2025-04-07 11:19:39,967:WARNING:certbot._internal.auth_handler:Challenge failed for domain sdmx2024.adriacongrex.it
2025-04-07 11:19:39,967:INFO:certbot._internal.auth_handler:http-01 challenge for sdmx2024.adriacongrex.it
2025-04-07 11:19:39,967:DEBUG:certbot._internal.reporter:Reporting to user: The following errors were reported by the server:Domain: sdmx2024.adriacongrex.it
Type:   unauthorized
Detail: 194.243.14.157: Invalid response from http://sdmx2024.adriacongrex.it/.well-known/acme-challenge/i9SgfbZHr7sHN-NFFXpsoLiRqqvFaW0m6Sq3AjbnQzA: 400

To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address.
2025-04-07 11:19:39,968:DEBUG:certbot._internal.error_handler:Encountered exception:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/certbot/_internal/auth_handler.py", line 91, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, best_effort)
  File "/usr/lib/python3/dist-packages/certbot/_internal/auth_handler.py", line 180, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.

2025-04-07 11:19:39,968:DEBUG:certbot._internal.error_handler:Calling registered functions
2025-04-07 11:19:39,968:INFO:certbot._internal.auth_handler:Cleaning up challenges
2025-04-07 11:19:40,303:ERROR:certbot._internal.renewal:Failed to renew certificate sdmx2024.adriacongrex.it with error: Some challenges have failed.
2025-04-07 11:19:40,305:DEBUG:certbot._internal.renewal:Traceback was:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/certbot/_internal/renewal.py", line 485, in handle_renewal_request
    main.renew_cert(lineage_config, plugins, renewal_candidate)
  File "/usr/lib/python3/dist-packages/certbot/_internal/main.py", line 1234, in renew_cert
    renewed_lineage = _get_and_save_cert(le_client, config, lineage=lineage)
  File "/usr/lib/python3/dist-packages/certbot/_internal/main.py", line 123, in _get_and_save_cert
    renewal.renew_cert(config, domains, le_client, lineage)
  File "/usr/lib/python3/dist-packages/certbot/_internal/renewal.py", line 345, in renew_cert
    new_cert, new_chain, new_key, _ = le_client.obtain_certificate(domains, new_key)
  File "/usr/lib/python3/dist-packages/certbot/_internal/client.py", line 374, in obtain_certificate
    orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
  File "/usr/lib/python3/dist-packages/certbot/_internal/client.py", line 421, in _get_order_and_authorizations
    authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
  File "/usr/lib/python3/dist-packages/certbot/_internal/auth_handler.py", line 91, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, best_effort)
  File "/usr/lib/python3/dist-packages/certbot/_internal/auth_handler.py", line 180, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.

2025-04-07 11:19:40,305:DEBUG:certbot.display.util:Notifying user:
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2025-04-07 11:19:40,305:ERROR:certbot._internal.renewal:All simulated renewals failed. The following certificates could not be renewed:
2025-04-07 11:19:40,305:ERROR:certbot._internal.renewal:  /etc/letsencrypt/live/sdmx2024.adriacongrex.it/fullchain.pem (failure)
2025-04-07 11:19:40,305:DEBUG:certbot.display.util:Notifying user: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2025-04-07 11:19:40,305:DEBUG:certbot._internal.log:Exiting abnormally:
Traceback (most recent call last):
  File "/usr/bin/certbot", line 33, in <module>
    sys.exit(load_entry_point('certbot==1.12.0', 'console_scripts', 'certbot')())
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 15, in main
    return internal_main.main(cli_args)
  File "/usr/lib/python3/dist-packages/certbot/_internal/main.py", line 1413, in main
    return config.func(config, plugins)
  File "/usr/lib/python3/dist-packages/certbot/_internal/main.py", line 1317, in renew
    renewal.handle_renewal_request(config)
  File "/usr/lib/python3/dist-packages/certbot/_internal/renewal.py", line 510, in handle_renewal_request
    raise errors.Error("{0} renew failure(s), {1} parse failure(s)".format(
certbot.errors.Error: 1 renew failure(s), 0 parse failure(s)
2025-04-07 11:19:40,306:ERROR:certbot._internal.log:1 renew failure(s), 0 parse failure(s)

Again this is happening for all my virtualhosts, I have around 20. NSLOOKUP works fine, there are A records for all of these. Everything is a shared IPv4 address. The current host sdmx2024.adriacongrex.it has a port 80 configured.

Here is an output of apachectl -S

VirtualHost configuration:
*:12322                localhost (/etc/apache2/sites-enabled/adminer.conf:3)
*:80                   is a NameVirtualHost
         default server emos2024.adriacongrex.it (/etc/apache2/sites-enabled/emos2024.conf:4)
         port 80 namevhost emos2024.adriacongrex.it (/etc/apache2/sites-enabled/emos2024.conf:4)
         port 80 namevhost sdmx2024.adriacongrex.it (/etc/apache2/sites-enabled/sdmx2024.adriacongrex.it.conf:5)
         port 80 namevhost teddyconvention.adriacongrex.it (/etc/apache2/sites-enabled/teddyconvention.conf:1)
         port 80 namevhost tigullioaritmologia.it (/etc/apache2/sites-enabled/www.tigullioaritmologia.it.conf:4)
*:443                  is a NameVirtualHost
         default server eata2025.com (/etc/apache2/sites-enabled/eata2025.conf:11)
         port 443 namevhost eata2025.com (/etc/apache2/sites-enabled/eata2025.conf:11)
         port 443 namevhost edpd2024edpd.eu (/etc/apache2/sites-enabled/edpd2024edpd.conf:3)
         port 443 namevhost www.edpd2024edpd.eu (/etc/apache2/sites-enabled/edpd2024edpd.conf:14)
         port 443 namevhost emos2024.adriacongrex.it (/etc/apache2/sites-enabled/emos2024.conf:10)
         port 443 namevhost era-events.adriacongrex.it (/etc/apache2/sites-enabled/eraevents.conf:4)
         port 443 namevhost www.eventistampabmw.com (/etc/apache2/sites-enabled/eventistampabmw.conf:4)
         port 443 namevhost eventistampabmw.com (/etc/apache2/sites-enabled/eventistampabmw.conf:14)
                 alias www.eventistampabmw.com
         port 443 namevhost followup.adriacongrex.it (/etc/apache2/sites-enabled/followup.conf:4)
         port 443 namevhost form.adriacongrex.it (/etc/apache2/sites-enabled/form.adriacongrex.it.conf:4)
         port 443 namevhost frontex.adriacongrex.it (/etc/apache2/sites-enabled/frontex.conf:4)
         port 443 namevhost sdmx2024.adriacongrex.it (/etc/apache2/sites-enabled/sdmx2024.adriacongrex.it.conf:10)
         port 443 namevhost teddyconvention.adriacongrex.it (/etc/apache2/sites-enabled/teddyconvention.conf:10)
         port 443 namevhost ticketing.adriacongrex.it (/etc/apache2/sites-enabled/ticketing.conf:10)
         port 443 namevhost www.tigullioaritmologia.it (/etc/apache2/sites-enabled/www.tigullioaritmologia.it.conf:9)
ServerRoot: "/etc/apache2"
Main DocumentRoot: "/var/www/html"
Main ErrorLog: "/var/log/apache2/error.log"
Mutex mpm-accept: using_defaults
Mutex fcgid-pipe: using_defaults
Mutex watchdog-callback: using_defaults
Mutex rewrite-map: using_defaults
Mutex ssl-stapling-refresh: using_defaults
Mutex fcgid-proctbl: using_defaults
Mutex ssl-stapling: using_defaults
Mutex ssl-cache: using_defaults
Mutex default: dir="/var/run/apache2/" mechanism=default 
PidFile: "/var/run/apache2/apache2.pid"
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
Define: MODSEC_2.5
Define: MODSEC_2.9
User: name="www-data" id=33
Group: name="www-data" id=33

Not really sure how to proceed. I was able to get one certified by turning off Apache2 and having certbot spinup its own temporary webserver on port 80. At that point it renewed the cert, but a basic renew command with the apache plugin gives me the aforementioned error on everything.

1 Like

The error from your web server is

Your browser sent a request that this server could not understand.
Reason: You're speaking plain HTTP to an SSL-enabled server port.
Instead use the HTTPS scheme to access this URL, please.

That suggests you have enabled SSL on port 80 instead of just on port 443.

3 Likes

I can confirm. Using curl with protocol https:// on port :80 I get the expected "404 file not found" HTTP response while the http:// protocol gets the 400 bad request response.

OPs webserver is severely misconfigured or perhaps external port 80 gets mapped to an internal port used for HTTPS.

2 Likes

Appreciate the feedback. Unsure how this is happening. This is my current vhost config:

<IfModule mod_ssl.c>
    SSLStaplingCache shmcb:${APACHE_RUN_DIR}/ssl_stapling(32768)


<VirtualHost *:80>
    DocumentRoot "/var/www/sdmx2024"
    ServerName sdmx2024.adriacongrex.it
</VirtualHost>

<VirtualHost *:443>
RewriteEngine off
#Block HTTP 0.9 and HTTP/1.0
    RewriteCond %{SERVER_PROTOCOL} ^HTTP/1\.0$ [NC]
    RewriteRule .* - [F]
#Block TRACE TRACK and OPTIONS HTML type requests
    RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK|OPTIONS)
    RewriteRule .* - [F]
#Block Unwanted User Agents
    RewriteCond %{HTTP_USER_AGENT} wfuzz [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} python [NC,OR]
#    RewriteCond %{HTTP_USER_AGENT} go [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} Fuzz\ Faster\ U\ Fool\ v1\.1\.0-git [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} Mozilla/4\.0\ \(compatible [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} curl [NC]
    RewriteRule .* - [F]
#Block unwanted Referers
    RewriteCond %{HTTP_REFERER} blowfish|CatchBot|BecomeBot [NC]
    RewriteRule .* - [F,L]
#End of Rewrite Rules

    DocumentRoot "/var/www/sdmx2024"
    ServerName sdmx2024.adriacongrex.it
    ServerAdmin it@adriacongrex.it

    SSLEngine On
    SSLUseStapling On
    SSLStaplingResponseMaxAge 900
    SSLCACertificateFile /etc/letsencrypt/live/sdmx2024.adriacongrex.it/fullchain.pem
    SSLCertificateFile /etc/letsencrypt/live/sdmx2024.adriacongrex.it/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/sdmx2024.adriacongrex.it/privkey.pem
    Include /etc/letsencrypt/options-ssl-apache.conf
    SSLProtocol -ALL +TLSv1.3 +TLSv1.2
    SSLOpenSSLConfCmd Curves X25519:secp521r1:secp384r1:prime256v1
    SSLCipherSuite    TLSv1.3   TLS_AES_256_GCM_SHA384:TLS_AES_128_GCM_SHA256
    SSLCipherSuite    SSL       ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256
    SSLHonorCipherOrder On
    SSLSessionTickets Off
    SSLOptions +StrictRequire

    ErrorLog ${APACHE_LOG_DIR}/error_sdmx2024.log
    CustomLog ${APACHE_LOG_DIR}/access_sdmx2024.log combined
    LogLevel warn ssl:warn
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" combined
    LogFormat "%v %h %l %u %t \"%r\" %>s %b" combined

    <FilesMatch "\.(cgi|shtml|phtml|php)$">
        SSLOptions +StdEnvVars
    </FilesMatch>

    <Directory "/var/www/sdmx2024">
        Options FollowSymLinks
        AllowOverride All
        Order allow,deny
        Allow from all
    </Directory>
</VirtualHost>
</IfModule>

Did you check for that in your NAT device yet?

4 Likes