Certbot ACME challenge not attempted but reported ad 403 error

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: chat.felixtech.io and turn.felixtech.io

I ran this command:
docker run --rm --network="dmz-net" --ip="172.16.1.19" -v /opt/stacks/element/certbot:/etc/letsencrypt certbot/certbot --non-interactive --agree-tos -m office@felixtech.io certonly --standalone -d turn.felixtech.io -vvvv

It produced this output:

Root logging level set at -10
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Requested authenticator standalone and installer None
Single candidate plugin: * standalone
Description: Runs an HTTP server locally which serves the necessary validation files under the /.well-known/acme-challenge/ request path. Suitable if there is no HTTP server already running. HTTP challenge only (wildcards not supported).
Interfaces: Authenticator, Plugin
Entry point: EntryPoint(name='standalone', value='certbot._internal.plugins.standalone:Authenticator', group='certbot.plugins')
Initialized: <certbot._internal.plugins.standalone.Authenticator object at 0x7f3c7d2d7170>
Prep: True
Selected authenticator <certbot._internal.plugins.standalone.Authenticator object at 0x7f3c7d2d7170> and installer None
Plugins selected: Authenticator standalone, Installer None
Picked account: <Account(RegistrationResource(body=Registration(key=None, contact=(), agreement=None, status=None, terms_of_service_agreed=None, only_return_existing=None, external_account_binding=None), uri='https://acme-v02.api.letsencrypt.org/acme/acct/1645499597', new_authzr_uri=None, terms_of_service=None), 6cacff391b272d23f3f422d60a2d94f2, Meta(creation_dt=datetime.datetime(2024, 3, 30, 16, 23, 40, tzinfo=<UTC>), creation_host='28856469db47', register_to_eff=None))>
Sending GET request to https://acme-v02.api.letsencrypt.org/directory.
Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org:443
https://acme-v02.api.letsencrypt.org:443 "GET /directory HTTP/1.1" 200 752
Received response:
HTTP 200
Server: nginx
Date: Sun, 31 Mar 2024 07:26:47 GMT
Content-Type: application/json
Content-Length: 752
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "FJq3cv5FT1k": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
  "keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.3-September-21-2022.pdf",
    "website": "https://letsencrypt.org"
  },
  "newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
  "renewalInfo": "https://acme-v02.api.letsencrypt.org/draft-ietf-acme-ari-02/renewalInfo/",
  "revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert"
}
Notifying user: Requesting a certificate for turn.felixtech.io
Requesting a certificate for turn.felixtech.io
Requesting fresh nonce
Sending HEAD request to https://acme-v02.api.letsencrypt.org/acme/new-nonce.
https://acme-v02.api.letsencrypt.org:443 "HEAD /acme/new-nonce HTTP/1.1" 200 0
Received response:
HTTP 200
Server: nginx
Date: Sun, 31 Mar 2024 07:26:47 GMT
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: CwITDWGvqRrPUoKSB9uPopG4Y7hf9eugP0aJIlGT5e0tO09yJ0M
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800


Storing nonce: CwITDWGvqRrPUoKSB9uPopG4Y7hf9eugP0aJIlGT5e0tO09yJ0M
JWS payload:
b'{\n  "identifiers": [\n    {\n      "type": "dns",\n      "value": "turn.felixtech.io"\n    }\n  ]\n}'
Sending POST request to https://acme-v02.api.letsencrypt.org/acme/new-order:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2FjY3QvMTY0NTQ5OTU5NyIsICJub25jZSI6ICJDd0lURFdHdnFSclBVb0tTQjl1UG9wRzRZN2hmOWV1Z1AwYUpJbEdUNWUwdE8wOXlKME0iLCAidXJsIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL25ldy1vcmRlciJ9",
  "signature": "dbm2iOkx0QYZU3sWgT3wCqdFJKPL2Djo23_3Dqsvj_sCKRpdY1r1woF_1FSS2QeRFjRC6bezCI7Eyj9uOSsyqJJ-Wh7b-Qq8QFgAdlqJSq4nhHfcQHLX0QiF_x_EF1751uWCmAVZQxdAFIE983ylYU1RD9f-NEfcG1Zs80MP5dteI6afouLF8AywNKxgfxT8FgUmHr9j0itx-JSlfdrSMB8pd2LYoVKwZX-Nxl3iSr2F9l0rCLP3gceEP0UDe6mXXnZpAVXbusqQbQ6tHrI4o6uoGNxUQOancuwn1CC2yIt1SgDQDLfHthtYYf2XKekIKoYztxEwLf72uiNtvH6cyA",
  "payload": "ewogICJpZGVudGlmaWVycyI6IFsKICAgIHsKICAgICAgInR5cGUiOiAiZG5zIiwKICAgICAgInZhbHVlIjogInR1cm4uZmVsaXh0ZWNoLmlvIgogICAgfQogIF0KfQ"
}
https://acme-v02.api.letsencrypt.org:443 "POST /acme/new-order HTTP/1.1" 201 343
Received response:
HTTP 201
Server: nginx
Date: Sun, 31 Mar 2024 07:26:48 GMT
Content-Type: application/json
Content-Length: 343
Connection: keep-alive
Boulder-Requester: 1645499597
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
Location: https://acme-v02.api.letsencrypt.org/acme/order/1645499597/256888582837
Replay-Nonce: D7lnnWxw_BC5foXbHMXW9aSw1VOW0WFF6ifA-cow0eSjMkf-g8c
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "status": "pending",
  "expires": "2024-04-07T07:26:48Z",
  "identifiers": [
    {
      "type": "dns",
      "value": "turn.felixtech.io"
    }
  ],
  "authorizations": [
    "https://acme-v02.api.letsencrypt.org/acme/authz-v3/332797577547"
  ],
  "finalize": "https://acme-v02.api.letsencrypt.org/acme/finalize/1645499597/256888582837"
}
Storing nonce: D7lnnWxw_BC5foXbHMXW9aSw1VOW0WFF6ifA-cow0eSjMkf-g8c
JWS payload:
b''
Sending POST request to https://acme-v02.api.letsencrypt.org/acme/authz-v3/332797577547:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2FjY3QvMTY0NTQ5OTU5NyIsICJub25jZSI6ICJEN2xubld4d19CQzVmb1hiSE1YVzlhU3cxVk9XMFdGRjZpZkEtY293MGVTak1rZi1nOGMiLCAidXJsIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2F1dGh6LXYzLzMzMjc5NzU3NzU0NyJ9",
  "signature": "EuYBc4iOg3nZC7nQjd27AZ5akpWZErdn9ANTiJpGUflfnPbWYyAQvcYtB3YUQjJPPPXxbF7irmgkb2d_ZKANzp66Elu2f50soGQJuknhg83X1Z-9-l-ua3GWpJPqG9rdM6cJ6YnwZcrB5Wh3eJRCVsbqxJC76S91vBbgkJhoSbfAnVkASFbAsCJytfDesiXoJWqbN9dv5CGl1j-VI7pcWEmfA2pKp5ORjWHLozhGkKG6h3JnB2pKSIS3oVEa3KkDU_aWqs39uaio1z0wF1F26DQHqic6fPchqTwFnS7tQGOp7RmuK_YdZ_LwH0qaE5uQclr8EGPrh15LYTZe6m95iw",
  "payload": ""
}
https://acme-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/332797577547 HTTP/1.1" 200 801
Received response:
HTTP 200
Server: nginx
Date: Sun, 31 Mar 2024 07:26:48 GMT
Content-Type: application/json
Content-Length: 801
Connection: keep-alive
Boulder-Requester: 1645499597
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: CwITDWGv9MvkHZhwr9swvd1l6KfN6Kw1ECZeYBTkb76pUqJfVxU
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "turn.felixtech.io"
  },
  "status": "pending",
  "expires": "2024-04-07T07:26:48Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/332797577547/spdr5g",
      "token": "SsK-3hJ-YDPHBv4fgcg1BBQ4zUoW8XvwheaXG6TGBYw"
    },
    {
      "type": "dns-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/332797577547/mdSF9Q",
      "token": "SsK-3hJ-YDPHBv4fgcg1BBQ4zUoW8XvwheaXG6TGBYw"
    },
    {
      "type": "tls-alpn-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/332797577547/ffXWIg",
      "token": "SsK-3hJ-YDPHBv4fgcg1BBQ4zUoW8XvwheaXG6TGBYw"
    }
  ]
}
Storing nonce: CwITDWGv9MvkHZhwr9swvd1l6KfN6Kw1ECZeYBTkb76pUqJfVxU
Performing the following challenges:
http-01 challenge for turn.felixtech.io
Successfully bound to :80 using IPv6
Certbot wasn't able to bind to :80 using IPv4, this is often expected due to the dual stack nature of IPv6 socket implementations.
JWS payload:
b'{}'
Sending POST request to https://acme-v02.api.letsencrypt.org/acme/chall-v3/332797577547/spdr5g:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2FjY3QvMTY0NTQ5OTU5NyIsICJub25jZSI6ICJDd0lURFdHdjlNdmtIWmh3cjlzd3ZkMWw2S2ZONkt3MUVDWmVZQlRrYjc2cFVxSmZWeFUiLCAidXJsIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2NoYWxsLXYzLzMzMjc5NzU3NzU0Ny9zcGRyNWcifQ",
  "signature": "fVSMdAnlBEXA8o1huYowMKyz8CcvTcRBOc53zKiUHmZ6QOB4WlxpR0lSkCNMneynG99hFWbQIHRIv_mOrV4YkDph2zzi4SKbGb6sn1TQ_B0Bf6rem9Y8EDpIOzguPZg3WeWa--E2zcvYruRCaVjbX2_tX-1En9kuXZ0XUTqiQjesn5oS7yr78_FA-3WG29EluhiEBd2z8X6ssOWa0z5tzjE06qH7w4CWlg0kQv41IfnT1jTqRSGnEgji8EefbS0GzLHoj2FSRcqGtOenlgvBl84JcQSMXZC7ryCDHCCcwgJosykILVxubkF9H2JkDyomrp9vGtXsCtdnnxV8Gg2wEg",
  "payload": "e30"
}
https://acme-v02.api.letsencrypt.org:443 "POST /acme/chall-v3/332797577547/spdr5g HTTP/1.1" 200 187
Received response:
HTTP 200
Server: nginx
Date: Sun, 31 Mar 2024 07:26:48 GMT
Content-Type: application/json
Content-Length: 187
Connection: keep-alive
Boulder-Requester: 1645499597
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index", <https://acme-v02.api.letsencrypt.org/acme/authz-v3/332797577547>;rel="up"
Location: https://acme-v02.api.letsencrypt.org/acme/chall-v3/332797577547/spdr5g
Replay-Nonce: CwITDWGvPEaDTgRd7yA0QPLw5UtIJd0uIk2vst-0HVsXOganCew
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "type": "http-01",
  "status": "pending",
  "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/332797577547/spdr5g",
  "token": "SsK-3hJ-YDPHBv4fgcg1BBQ4zUoW8XvwheaXG6TGBYw"
}
Storing nonce: CwITDWGvPEaDTgRd7yA0QPLw5UtIJd0uIk2vst-0HVsXOganCew
Waiting for verification...
JWS payload:
b''
Sending POST request to https://acme-v02.api.letsencrypt.org/acme/authz-v3/332797577547:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2FjY3QvMTY0NTQ5OTU5NyIsICJub25jZSI6ICJDd0lURFdHdlBFYURUZ1JkN3lBMFFQTHc1VXRJSmQwdUlrMnZzdC0wSFZzWE9nYW5DZXciLCAidXJsIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2F1dGh6LXYzLzMzMjc5NzU3NzU0NyJ9",
  "signature": "V3BRYPTs8MwobbdgUcNmJxSS0b710P-rwVtR1aRt912TsFFdm7UcLAXQlw92Uylfi55hXZmWqpCJ1iotAkTToOmPxFgcW7Y3SJbFSJHepmb1WcaIbSCyKRHUFfyh2Kqfbpm0YCiMSJ0xjHmMp1vNEofF5b9ytDFsWicdPwnAUpnR4Fv_sQ1asq0nPi_F5hflGI29ILy_8o5sdX86a9jozYvYjhNdzCl06TTniac7T6aez7DYW-3WG8Vj_aw9YR5XmpHkyRk9SvTlNzk94lqMjo0UEI0jnL6Rwr-LVcrkqo8CZOG2bN4LB9fEf5AaJ62VXBUxQkP0CoN5efEA2tmShA",
  "payload": ""
}
https://acme-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/332797577547 HTTP/1.1" 200 1148
Received response:
HTTP 200
Server: nginx
Date: Sun, 31 Mar 2024 07:26:49 GMT
Content-Type: application/json
Content-Length: 1148
Connection: keep-alive
Boulder-Requester: 1645499597
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: D7lnnWxwDwskvk6lSkCWC3rE9Ey_ZA8KLqzimLNRfshwgP9pmnE
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "turn.felixtech.io"
  },
  "status": "invalid",
  "expires": "2024-04-07T07:26:48Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "invalid",
      "error": {
        "type": "urn:ietf:params:acme:error:unauthorized",
        "detail": "100.200.100.200: Invalid response from http://turn.felixtech.io/.well-known/acme-challenge/SsK-3hJ-YDPHBv4fgcg1BBQ4zUoW8XvwheaXG6TGBYw: 404",
        "status": 403
      },
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/332797577547/spdr5g",
      "token": "SsK-3hJ-YDPHBv4fgcg1BBQ4zUoW8XvwheaXG6TGBYw",
      "validationRecord": [
        {
          "url": "http://turn.felixtech.io/.well-known/acme-challenge/SsK-3hJ-YDPHBv4fgcg1BBQ4zUoW8XvwheaXG6TGBYw",
          "hostname": "turn.felixtech.io",
          "port": "80",
          "addressesResolved": [
            "100.200.100.200"
          ],
          "addressUsed": "100.200.100.200",
          "resolverAddrs": [
            "A:10.0.12.82:22757",
            "AAAA:10.0.12.83:29255"
          ]
        }
      ],
      "validated": "2024-03-31T07:26:48Z"
    }
  ]
}
Storing nonce: D7lnnWxwDwskvk6lSkCWC3rE9Ey_ZA8KLqzimLNRfshwgP9pmnE
Challenge failed for domain turn.felixtech.io
http-01 challenge for turn.felixtech.io

Certbot failed to authenticate some domains (authenticator: standalone). The Certificate Authority reported these problems:
  Domain: turn.felixtech.io
  Type:   unauthorized
  Detail: 100.200.100.200: Invalid response from http://turn.felixtech.io/.well-known/acme-challenge/SsK-3hJ-YDPHBv4fgcg1BBQ4zUoW8XvwheaXG6TGBYw: 404

Hint: The Certificate Authority failed to download the challenge files from the temporary standalone webserver started by Certbot on port 80. Ensure that the listed domains point to this machine and that it can accept inbound connections from the internet.

Notifying user:
Certbot failed to authenticate some domains (authenticator: standalone). The Certificate Authority reported these problems:
  Domain: turn.felixtech.io
  Type:   unauthorized
  Detail: 100.200.100.200: Invalid response from http://turn.felixtech.io/.well-known/acme-challenge/SsK-3hJ-YDPHBv4fgcg1BBQ4zUoW8XvwheaXG6TGBYw: 404

Hint: The Certificate Authority failed to download the challenge files from the temporary standalone webserver started by Certbot on port 80. Ensure that the listed domains point to this machine and that it can accept inbound connections from the internet.

Encountered exception:
Traceback (most recent call last):
  File "/opt/certbot/src/certbot/certbot/_internal/auth_handler.py", line 108, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, max_time_mins, best_effort)
  File "/opt/certbot/src/certbot/certbot/_internal/auth_handler.py", line 212, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.

Calling registered functions
Cleaning up challenges
Stopping server at :::80...
Exiting abnormally:
Traceback (most recent call last):
  File "/usr/local/bin/certbot", line 33, in <module>
    sys.exit(load_entry_point('certbot', 'console_scripts', 'certbot')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/certbot/src/certbot/certbot/main.py", line 19, in main
    return internal_main.main(cli_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/certbot/src/certbot/certbot/_internal/main.py", line 1894, in main
    return config.func(config, plugins)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/certbot/src/certbot/certbot/_internal/main.py", line 1600, in certonly
    lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/certbot/src/certbot/certbot/_internal/main.py", line 143, in _get_and_save_cert
    lineage = le_client.obtain_and_enroll_certificate(domains, certname)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/certbot/src/certbot/certbot/_internal/client.py", line 517, in obtain_and_enroll_certificate
    cert, chain, key, _ = self.obtain_certificate(domains)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/certbot/src/certbot/certbot/_internal/client.py", line 428, in obtain_certificate
    orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/certbot/src/certbot/certbot/_internal/client.py", line 496, in _get_order_and_authorizations
    authzr = self.auth_handler.handle_authorizations(orderr, self.config, best_effort)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/certbot/src/certbot/certbot/_internal/auth_handler.py", line 108, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, max_time_mins, best_effort)
  File "/opt/certbot/src/certbot/certbot/_internal/auth_handler.py", line 212, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.
Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

My web server is (include version): whatever is in the official Docker image

The operating system my web server runs on is (include version): VMware Photon OS

My hosting provider, if applicable, is: own datacenter

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 2.9.0

We have used this in an unchanged configuration since a few years and it has always worked. Since a few days Certbot doesn't even attempt to do the ACME challenge anymore and just invents a 403 error. There is no incoming traffic to this IP address on port 80 at all. This is confirmed on the edge firewall of the datacenter. It has been made sure by the firewall people that on the networking side everything is fine. There is just no attempt that anyone is ever trying to read the challenge from this webserver.

2

Hi @felixtech-msp, and welcome to the LE community forum :slight_smile:

If that IP information wasn't obfuscated, there may be an issue there; As that is NOT the IP resolved for that name:

felixtech.io nameserver = porter.ns.cloudflare.com
felixtech.io nameserver = karsyn.ns.cloudflare.com

Name:    turn.felixtech.io
Address: 91.192.101.38
3 Likes
3

That looks like a problem.

3 Likes
4

The IP is actually correct in the original output.

So if Certbot can't bind to port 80 that is then a problem within Certbot itself.
Within the original Docker container there shouldn't be anything blocking the webserver from binding to port 80.
Also as the container is run as root, there shouldn't be any issue with permissions.
The error says then "this is often expected due to the dual stack nature of IPv6 socket implementations" so I guess if it can bind to IPv6 then a IPv4 address should work as well.

This has worked since 2021 in unchanged form.

What I can't understand is that the IP is correct, but there is no actual attempt to connect to anything. Even if port 80 wouldn't be open, there should be at least incoming traffic on the edge firewall.
Also the 403 error is a complete lie as there can't be any HTTP response to a request that never happened.

When disabling IPv6 on the server completely, the message is the same. We don't use IPv6 internally at all.

5

The IP of 100.200.100.200 belongs to T-Mobile USA. Is T-mobile USA your ISP and have you been assigned that IP from them?

1 Like
6

I don't see why there are different IPs:

What shows?:

curl -4 ifconfig.me

3 Likes
7

The 403 in the log message status is the error response from the Let's Encrypt ACME Server sent to the Certbot ACME Client. This is just the category of failure why the LE Server rejected the cert request.

The specific reason for the failure is shown in the "Detail" portion of the error which is a 404 (Not Found). This means the Let's Encrypt server got a 404 response from something responding to the HTTP request to your domain. A failure to get any response is usually shown in the "detail" as a "timeout".

Would an openresty server have replied prior to it reaching your container? Or even your edge firewall? Because that is what is replying right now

The common "Server" response header is omitted but the data includes openresty

curl -i http://turn.felixtech.io/.well-known/acme-challenge/Test404
HTTP/1.1 404 Not Found
Date: Sun, 31 Mar 2024 13:38:20 GMT
Content-Type: text/html
Content-Length: 150
Connection: keep-alive

<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>openresty</center>
</body>
</html>
3 Likes
8

The IP in the log was redacted and therefore changed. As I said there is no networking problem. As far as our team was able to find out this isn't even a Certbot problem. The LE infrastructure just fantasizes around and is never attempting to check if the challenge is actually there. The actual application which is running on this IP does work normally, so a DNS, network or firewall issue is definitely not happening.
I have tried older builds of Certbot which resulted in the same issue which just confirms that this is not a problem with the Docker image or Certbot itself.
I may retry with this standalone solution in a few months and check if it has changed.
We have switched to DNS challenge now as we use anyways Cloudflare which was super easy to set up.
On our main infrastrcture we use DNS challenges since a year as the HTTP challenge seemed to be unreliable and just doesn't work every few months.

9

When the challenge is happening, the main application is shut down. The 404 response can only come from Certbot itself.
We don't have an openresty server in this infrastructure. Actually on the turn.felixtech.io subdomain there isn't actually any webserver running.

10

That is unlikely. LE issues around 4 million certs per day.

Do you have an explanation for me seeing an openresty server replying to HTTP requests to your domain?

I just issued this on my own test server. What is it if not yours?

curl -i http://turn.felixtech.io/.well-known/acme-challenge/Test404
HTTP/1.1 404 Not Found
Date: Sun, 31 Mar 2024 13:38:20 GMT
Content-Type: text/html
Content-Length: 150
Connection: keep-alive

<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>openresty</center>
</body>
</html>
3 Likes
11

I have no clue how this can happen as there is no openresty server running on this domain. Port 80 isn't even open when there is no challenge ongoing.

Running the same command as you has on my workstation a different output:

curl -i http://turn.felixtech.io/.well-known/acme-challenge/Test404
curl: (7) Failed to connect to turn.felixtech.io port 80 after 7 ms: Couldn't connect to server
12

Try it from the public internet. Use a mobile phone with wifi disabled to use your carrier's network.

3 Likes
13

I can see that 404 from my phone but it doesn't come from our infrastructure. Due to DNS challenge now being used, I have deleted the port 80 firewall rule. There is also not coming any traffic on port 80 from the internet to this IP (would see that as default block rule). On port 443 there is a nginx webserver.
This response is created well before it even gets routed to our internet connection.
I have also confirmed with the ISP guys in the datacenter that there is nothing in their stuff which would reply to a HTTP request from their side (which would be insane anyways).

14

But now you know it is not coming from Let's Encrypt.

A request from your phone / public internet gets a reply from an openresty server. Has nothing to do with LE.

I have seen similar problems caused by firewall inspection services (possibly ISP or comms equipment). Or URL/Domain forwarding services. It must be something related to your ISP, network config, or servers. There is nothing else "farther away" from you that would cause this pattern of failure (well, maybe if you lived in certain countries whose government controls internet traffic but not the case here).

3 Likes
15

You said you switched to DNS Challenge and I see a fresh cert issued. That's good. But, if you think your HTTPS should be working you should check out the below. Requests to turn.felixtech.io are failing (from the public internet)

https://www.ssllabs.com/ssltest/analyze.html?d=turn.felixtech.io&hideResults=on

image
image2208×732 57.1 KB

3 Likes
16

Yes, that is intended. The turn.felixtech.io domain has no HTTP or HTTPS running on it. This domain only runs a TURN server and this application has a TLS certificate. With this check you get the HTTP service which runs on this IP under the chat.felixtech.io domain.

1 Like
17

We found it to be a bug within the NAT function on the OPNsense edge firewall. It forwarded all the incoming HTTP requests to a random unrelated server. Cause of the bug is that when the firewall has lots of IP addresses on the WAN interface and there are entries of the first IP in the NAT table higher up than other entries of other IPs it will just choose the first rule with a matching port and non-matching IP address.
This is completely insane and as of our experience with Deciso they won't care at all about this bug and so I just reordered the NAT table so all the entries with the first IP of the WAN interface are on the bottom.

1 Like
18

So, in your infrastructure then yes :slight_smile:

4 Likes
19

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.