Renew error: Timeout during connect (using IPFire)

Help
1

Details of my setting below, followed by the standard questionnaire. Just a brief intro first.

After more than 2 years, suddenly I cannot certbot renew my home nextcloud server. Last successful renew in June. Port 80 is open (tested) and any access gets rewritten into https. I do not understand enough what I am doing to solve the problem by myself. Whatever I do, I cannot prevent the rewriting of the request into https. You can find Apache setting below, followed by the answers to the standard questionnaire.

apachectl -t -D DUMP_VHOSTS
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message
VirtualHost configuration:
*:443                  nextcloud.panaccio.family (/etc/apache2/sites-enabled/default-ssl.conf:2)
*:80                   nextcloud.panaccio.family (/etc/apache2/sites-enabled/nextcloud.conf:1)

cat /etc/apache2/sites-enabled/default-ssl.conf
	<VirtualHost _default_:443>
		ServerAdmin admin@panaccio.family
		DocumentRoot /var/www/nextcloud
		ErrorLog ${APACHE_LOG_DIR}/error.log
		CustomLog ${APACHE_LOG_DIR}/access.log combined
		SSLEngine on
		<FilesMatch "\.(cgi|shtml|phtml|php)$">
				SSLOptions +StdEnvVars
		</FilesMatch>
		<Directory /usr/lib/cgi-bin>
				SSLOptions +StdEnvVars
		</Directory>
		ServerName	nextcloud.panaccio.family
		Header always set Strict-Transport-Security "max-age=31536000"
		SSLCertificateFile /etc/letsencrypt/live/nextcloud.panaccio.family/fullchain.pem
		SSLCertificateKeyFile /etc/letsencrypt/live/nextcloud.panaccio.family/privkey.pem
                Include /etc/letsencrypt/options-ssl-apache.conf
		RewriteEngine On
		RewriteRule ^/sites/(.*)$ https://nextcloud.panaccio.family/index.php/apps/cms_pico/pico/$1 [QSA,L]
	</VirtualHost>
</IfModule>

 cat /etc/apache2/sites-enabled/nextcloud.conf
<VirtualHost *:80>
  ServerAdmin admin@panaccio.family
  DocumentRoot /var/www/nextcloud
  ServerName nextcloud.panaccio.family
  TransferLog /var/log/apache2/nextcloud_access.log
  ErrorLog /var/log/apache2/nextcloud_error.log
  RewriteEngine on
  RewriteCond %{SERVER_NAME} =nextcloud.panaccio.family
  RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>

I ran this command:
certbot -v renew --dry-run

It produced this output:

Root logging level set at 10
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Notifying user: Processing /etc/letsencrypt/renewal/nextcloud.panaccio.family.conf
Requested authenticator <certbot._internal.cli.cli_utils._Default object at 0x7ff9513e0190> and installer <certbot._internal.cli.cli_utils._Default object at 0x7ff9513e0190>
Var dry_run=True (set by user).
Var server={'dry_run', 'staging'} (set by user).
Var dry_run=True (set by user).
Var server={'dry_run', 'staging'} (set by user).
Var account={'server'} (set by user).
Starting new HTTP connection (1): r3.o.lencr.org:80
http://r3.o.lencr.org:80 "POST / HTTP/1.1" 200 503
OCSP response for certificate /etc/letsencrypt/archive/nextcloud.panaccio.family/cert19.pem is signed by the certificate's issuer.
OCSP certificate status for /etc/letsencrypt/archive/nextcloud.panaccio.family/cert19.pem is: OCSPCertStatus.GOOD
Should renew, less than 30 days before certificate expiry 2022-09-17 01:45:56 UTC.
Cert is due for renewal, auto-renewing...
Requested authenticator apache and installer apache
Apache version is 2.4.54
Single candidate plugin: * apache
Description: Apache Web Server plugin
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: apache = certbot_apache._internal.entrypoint:ENTRYPOINT
Initialized: <certbot_apache._internal.override_debian.DebianConfigurator object at 0x7ff951425790>
Prep: True
Single candidate plugin: * apache
Description: Apache Web Server plugin
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: apache = certbot_apache._internal.entrypoint:ENTRYPOINT
Initialized: <certbot_apache._internal.override_debian.DebianConfigurator object at 0x7ff951425790>
Prep: True
Selected authenticator <certbot_apache._internal.override_debian.DebianConfigurator object at 0x7ff951425790> and installer <certbot_apache._internal.override_debian.DebianConfigurator object at 0x7ff951425790>
Plugins selected: Authenticator apache, Installer apache
Picked account: <Account(RegistrationResource(body=Registration(key=None, contact=(), agreement=None, status=None, terms_of_service_agreed=None, only_return_existing=None, external_account_binding=None), uri='https://acme-staging-v02.api.letsencrypt.org/acme/acct/67892723', new_authzr_uri=None, terms_of_service=None), 126fadcc1afe917454a39b7f70610731, Meta(creation_dt=datetime.datetime(2022, 9, 9, 9, 29, 44, tzinfo=<UTC>), creation_host='webserver', register_to_eff=None))>
Sending GET request to https://acme-staging-v02.api.letsencrypt.org/directory.
Starting new HTTPS connection (1): acme-staging-v02.api.letsencrypt.org:443
https://acme-staging-v02.api.letsencrypt.org:443 "GET /directory HTTP/1.1" 200 843
Received response:
HTTP 200
Server: nginx
Date: Fri, 09 Sep 2022 09:43:26 GMT
Content-Type: application/json
Content-Length: 843
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "keyChange": "https://acme-staging-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017-w-v1.3-notice.pdf",
    "website": "https://letsencrypt.org/docs/staging-environment/"
  },
  "newAccount": "https://acme-staging-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-staging-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-staging-v02.api.letsencrypt.org/acme/new-order",
  "reVktW6Wv2A": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
  "renewalInfo": "https://acme-staging-v02.api.letsencrypt.org/get/draft-ietf-acme-ari-00/renewalInfo/",
  "revokeCert": "https://acme-staging-v02.api.letsencrypt.org/acme/revoke-cert"
}
Notifying user: Simulating renewal of an existing certificate for nextcloud.panaccio.family
Requesting fresh nonce
Sending HEAD request to https://acme-staging-v02.api.letsencrypt.org/acme/new-nonce.
https://acme-staging-v02.api.letsencrypt.org:443 "HEAD /acme/new-nonce HTTP/1.1" 200 0
Received response:
HTTP 200
Server: nginx
Date: Fri, 09 Sep 2022 09:43:27 GMT
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: 0001pIvXJyzVdARjgybjnv84ptix5drg48_6TVhi-Etcrts
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800


Storing nonce: 0001pIvXJyzVdARjgybjnv84ptix5drg48_6TVhi-Etcrts
JWS payload:
b'{\n  "identifiers": [\n    {\n      "type": "dns",\n      "value": "nextcloud.panaccio.family"\n    }\n  ]\n}'
Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/new-order:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS1zdGFnaW5nLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYWNjdC82Nzg5MjcyMyIsICJub25jZSI6ICIwMDAxcEl2WEp5elZkQVJqZ3liam52ODRwdGl4NWRyZzQ4XzZUVmhpLUV0Y3J0cyIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXN0YWdpbmctdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9uZXctb3JkZXIifQ",
  "signature": "EXVyk5lpjBjAUTYCNkuYwAieF62_v2yeXTUu3Wa8cKmWiRMNRpBTkAPYQLAadqtdfjcEKAjTHVTeQtxfvpPwLWUB78O3GWdOz6GMzCzfTvGGqUfyl2LIa1Pi8ZjANzjQBYzlVOuM_jHhPpCupMU17D-fuaZs7NGx-yu1J6jWquG1kfKgTG-1q3uQ9RRoHGBj58NhCu98zBzCKu6BuOFoz-P6v79hqiAAciU8cdcz4GMHzKpjqqd-W0t0fKEw4Pjf9Pm7BT9poHZit-1pRkqeTcesN4i-sX8RlZpZ7RV9Ifj54DvyA0mCDVUjPxncYdS0Iv-27QXN55QPA6HfMs6Rdw",
  "payload": "ewogICJpZGVudGlmaWVycyI6IFsKICAgIHsKICAgICAgInR5cGUiOiAiZG5zIiwKICAgICAgInZhbHVlIjogIm5leHRjbG91ZC5wYW5hY2Npby5mYW1pbHkiCiAgICB9CiAgXQp9"
}
https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/new-order HTTP/1.1" 201 361
Received response:
HTTP 201
Server: nginx
Date: Fri, 09 Sep 2022 09:43:27 GMT
Content-Type: application/json
Content-Length: 361
Connection: keep-alive
Boulder-Requester: 67892723
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
Location: https://acme-staging-v02.api.letsencrypt.org/acme/order/67892723/3990418093
Replay-Nonce: 0002lSFuSkANHyhDYvgm1Ogfz7OS1MVsDDIigRI906Cctlo
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "status": "pending",
  "expires": "2022-09-16T09:43:27Z",
  "identifiers": [
    {
      "type": "dns",
      "value": "nextcloud.panaccio.family"
    }
  ],
  "authorizations": [
    "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3580687073"
  ],
  "finalize": "https://acme-staging-v02.api.letsencrypt.org/acme/finalize/67892723/3990418093"
}
Storing nonce: 0002lSFuSkANHyhDYvgm1Ogfz7OS1MVsDDIigRI906Cctlo
JWS payload:
b''
Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3580687073:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS1zdGFnaW5nLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYWNjdC82Nzg5MjcyMyIsICJub25jZSI6ICIwMDAybFNGdVNrQU5IeWhEWXZnbTFPZ2Z6N09TMU1Wc0RESWlnUkk5MDZDY3RsbyIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXN0YWdpbmctdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9hdXRoei12My8zNTgwNjg3MDczIn0",
  "signature": "sjZCjT9s2H1044jSACdpbkvfP0BIXqAyuoAiYTqtkQvHLv8DdItErd3bvY_zBzjqETfPmAbcwfi-___aDJvqqLlUlUXAt3GntKsZpxMdgs2S5leP9Z7_NCXYwXZoG0NpDsZFUJZeD8VafhLYJPhjdUkHFTkEPloTVs09IQ2G-Owm7rTa5aeW-K4QsJhXyiSzZmVgLd7BrnS5lBf1gdGquhd-4ocXcdfiDz_eMAGfigfXgsH_3BhmrEbhn2mwUifuibSmyypVyd3VXctsr7D4KG41c5JstBNKI_BxuPaVZvyqvDRvmRKti5dY0AzAPtKEq_bjesMtm4rQRlouQOAwtg",
  "payload": ""
}
https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/3580687073 HTTP/1.1" 200 827
Received response:
HTTP 200
Server: nginx
Date: Fri, 09 Sep 2022 09:43:27 GMT
Content-Type: application/json
Content-Length: 827
Connection: keep-alive
Boulder-Requester: 67892723
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: 00026xQ63rNZix7zVTNzWGQiLeGNLtOHeH1R_J6Y79DrF5Q
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "nextcloud.panaccio.family"
  },
  "status": "pending",
  "expires": "2022-09-16T09:43:27Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "pending",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/3580687073/h4VMjw",
      "token": "zJy_d4Ck_jT-5kRtc4-TVW6_EfmNcBlPBWAhZ7kC4Pk"
    },
    {
      "type": "dns-01",
      "status": "pending",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/3580687073/srvmwA",
      "token": "zJy_d4Ck_jT-5kRtc4-TVW6_EfmNcBlPBWAhZ7kC4Pk"
    },
    {
      "type": "tls-alpn-01",
      "status": "pending",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/3580687073/Ek_9Ug",
      "token": "zJy_d4Ck_jT-5kRtc4-TVW6_EfmNcBlPBWAhZ7kC4Pk"
    }
  ]
}
Storing nonce: 00026xQ63rNZix7zVTNzWGQiLeGNLtOHeH1R_J6Y79DrF5Q
Performing the following challenges:
http-01 challenge for nextcloud.panaccio.family
Adding a temporary challenge validation Include for name: nextcloud.panaccio.family in: /etc/apache2/sites-enabled/default-ssl.conf
Adding a temporary challenge validation Include for name: nextcloud.panaccio.family in: /etc/apache2/sites-enabled/nextcloud.conf
writing a pre config file with text:
         RewriteEngine on
        RewriteRule ^/\.well-known/acme-challenge/([A-Za-z0-9-_=]+)$ /var/lib/letsencrypt/http_challenges/$1 [END]
    
writing a post config file with text:
         <Directory /var/lib/letsencrypt/http_challenges>
            Require all granted
        </Directory>
        <Location /.well-known/acme-challenge>
            Require all granted
        </Location>
    
Creating backup of /etc/apache2/sites-enabled/default-ssl.conf
Creating backup of /etc/apache2/sites-enabled/nextcloud.conf
Waiting for verification...
JWS payload:
b'{}'
Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/3580687073/h4VMjw:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS1zdGFnaW5nLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYWNjdC82Nzg5MjcyMyIsICJub25jZSI6ICIwMDAyNnhRNjNyTlppeDd6VlROeldHUWlMZUdOTHRPSGVIMVJfSjZZNzlEckY1USIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXN0YWdpbmctdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9jaGFsbC12My8zNTgwNjg3MDczL2g0Vk1qdyJ9",
  "signature": "0CIwuXQx7f75Ni6zgQ8oNfFJvvSJ61AnlY6xvJclGoiGzXMEDLVGyxQtqXUr4NqT86yR9Zf-kEtltLnsGHqKiK0qmgkCKPnb_pfLBsDI96Af01MRszk9y4N4ICFH8nTnDXvwHdTF1OQlQubTJ8Ascg2FRz63HkOr0doKQkl7JCRbrRlftMbXgPjIKVcWgtgxnNTw8yH9BA3cCRvJrlKK0jgIn4fSKwjKeqL4EOMqKB3tctdpQsc7c0JCnB-1wrosDrpSd5ZvdLf6blcekpX1mAhkq7QmvK_4ARfTeZwJiqORFeQs6AkjpfshxLYfAnDlet9Y76xbOmvu2uw8k9Hg2Q",
  "payload": "e30"
}
https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/chall-v3/3580687073/h4VMjw HTTP/1.1" 200 193
Received response:
HTTP 200
Server: nginx
Date: Fri, 09 Sep 2022 09:43:31 GMT
Content-Type: application/json
Content-Length: 193
Connection: keep-alive
Boulder-Requester: 67892723
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index", <https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3580687073>;rel="up"
Location: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/3580687073/h4VMjw
Replay-Nonce: 00020GGpz65NlPbpS4QJSJ8NDjhMXdnmaoSL0MAZjFm-ODs
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "type": "http-01",
  "status": "pending",
  "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/3580687073/h4VMjw",
  "token": "zJy_d4Ck_jT-5kRtc4-TVW6_EfmNcBlPBWAhZ7kC4Pk"
}
Storing nonce: 00020GGpz65NlPbpS4QJSJ8NDjhMXdnmaoSL0MAZjFm-ODs
JWS payload:
b''
Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3580687073:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS1zdGFnaW5nLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYWNjdC82Nzg5MjcyMyIsICJub25jZSI6ICIwMDAyMEdHcHo2NU5sUGJwUzRRSlNKOE5EamhNWGRubWFvU0wwTUFaakZtLU9EcyIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXN0YWdpbmctdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9hdXRoei12My8zNTgwNjg3MDczIn0",
  "signature": "eJ5ec5mTLtq_lDghaKLB0CDP_hw-CWj_E7s-VKRik08C7Nrl3dCfvt8MoM6m5Hfgq9oZl18lFQJUxIOxRIbbNOJpE9zDo5NU_xPp2ZttIc5BpInCofkZdFDCwyHeRkHIo24iMysuVGSKvS3o8ufXvdUPIlmmGf0ykb9vBXIuwGIfQ5xir5fMtiSccRHlorsP96KaWEkIvSQJrjtKHmcEUyk_FoAUeUQXcKGjvJTH2KnNynZQ5zGtrTawXlW9VOD05dGW-0NKRMlDgPDD8UeqD3cu56ezjOuSgfejPYIXhhqcPXAzxlpb7jjhttiBto7ULAwyoVzJxmFp3rHcPo-bXg",
  "payload": ""
}
https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/3580687073 HTTP/1.1" 200 827
Received response:
HTTP 200
Server: nginx
Date: Fri, 09 Sep 2022 09:43:33 GMT
Content-Type: application/json
Content-Length: 827
Connection: keep-alive
Boulder-Requester: 67892723
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: 00023GzfbfEOM81LiMxh1VRIRHh__rr0pFIZxQfK49icqU4
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "nextcloud.panaccio.family"
  },
  "status": "pending",
  "expires": "2022-09-16T09:43:27Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "pending",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/3580687073/h4VMjw",
      "token": "zJy_d4Ck_jT-5kRtc4-TVW6_EfmNcBlPBWAhZ7kC4Pk"
    },
    {
      "type": "dns-01",
      "status": "pending",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/3580687073/srvmwA",
      "token": "zJy_d4Ck_jT-5kRtc4-TVW6_EfmNcBlPBWAhZ7kC4Pk"
    },
    {
      "type": "tls-alpn-01",
      "status": "pending",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/3580687073/Ek_9Ug",
      "token": "zJy_d4Ck_jT-5kRtc4-TVW6_EfmNcBlPBWAhZ7kC4Pk"
    }
  ]
}
Storing nonce: 00023GzfbfEOM81LiMxh1VRIRHh__rr0pFIZxQfK49icqU4
JWS payload:
b''
Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3580687073:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS1zdGFnaW5nLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYWNjdC82Nzg5MjcyMyIsICJub25jZSI6ICIwMDAyM0d6ZmJmRU9NODFMaU14aDFWUklSSGhfX3JyMHBGSVp4UWZLNDlpY3FVNCIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXN0YWdpbmctdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9hdXRoei12My8zNTgwNjg3MDczIn0",
  "signature": "HQ06xHcDuPSCpd8wJR8sxGElWdtwJAo2mKwKM_OmwmfY5F2QbaL_GpWI6kcafQvvhV2AaOC3wRAsGv4MEn45OGCNcmdleTvaTSsIywp7s1xxpNSv3Apo9p_Zj7oS5zv9c3pg_CqVRyelwbGS-FQ6EWDvMmDht_aEJi3dcXd8xsyMIHpAeW4Ir5vLNt6Qx0GQKiuKsseiZ9gs7oixa2ZJ98GYfK08FRstxQxO7E-D2TiJRGycKSgkEUAv0T7DDPN3NFgvIsOGxCWw2fmeDOUARWy_YqRQj7aLbtsazFiLhQtbEhhylsAKtHkFwmRFTANGf_7ZuTh2sNGSeZS57oqurQ",
  "payload": ""
}
https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/3580687073 HTTP/1.1" 200 827
Received response:
HTTP 200
Server: nginx
Date: Fri, 09 Sep 2022 09:43:36 GMT
Content-Type: application/json
Content-Length: 827
Connection: keep-alive
Boulder-Requester: 67892723
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: 0002bzsic3oGPzS8vk7Fa8ucFz_ACVtkmI3XMaqzeZ72Kzk
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "nextcloud.panaccio.family"
  },
  "status": "pending",
  "expires": "2022-09-16T09:43:27Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "pending",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/3580687073/h4VMjw",
      "token": "zJy_d4Ck_jT-5kRtc4-TVW6_EfmNcBlPBWAhZ7kC4Pk"
    },
    {
      "type": "dns-01",
      "status": "pending",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/3580687073/srvmwA",
      "token": "zJy_d4Ck_jT-5kRtc4-TVW6_EfmNcBlPBWAhZ7kC4Pk"
    },
    {
      "type": "tls-alpn-01",
      "status": "pending",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/3580687073/Ek_9Ug",
      "token": "zJy_d4Ck_jT-5kRtc4-TVW6_EfmNcBlPBWAhZ7kC4Pk"
    }
  ]
}
Storing nonce: 0002bzsic3oGPzS8vk7Fa8ucFz_ACVtkmI3XMaqzeZ72Kzk
JWS payload:
b''
Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3580687073:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS1zdGFnaW5nLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYWNjdC82Nzg5MjcyMyIsICJub25jZSI6ICIwMDAyYnpzaWMzb0dQelM4dms3RmE4dWNGel9BQ1Z0a21JM1hNYXF6ZVo3Mkt6ayIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXN0YWdpbmctdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9hdXRoei12My8zNTgwNjg3MDczIn0",
  "signature": "X5aCX9u5e1GmpncYT-3uOu_xeekEc1IoPDeVuJR3S65EJYXTdGzW-nqFKdHWqzcuJNagrBzP1zGVg40H790CW--6hgGYXTdxFcRd2S3Rjr587iEKX1TKF8VOQHS8N42ioiL7pEKnoAte-Wg8tXcF21nON56CKAg3Z0fg-VaxbaQFPkvWMMdmOVHtWG06PYs50x-StZa6R1w6jsiDn3ZbVzyHghPKib8XyqwNson_VTNr_ZyH8H_41UGuy7ANFywSKADQOCX3z3w4D_xlvIokQUFz94nlHeCyhhG9oHNrsqVcJelq5St8TRAgz6rETnanMcPjC5Wx9AEjQHhkTVqbhg",
  "payload": ""
}
https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/3580687073 HTTP/1.1" 200 827
Received response:
HTTP 200
Server: nginx
Date: Fri, 09 Sep 2022 09:43:39 GMT
Content-Type: application/json
Content-Length: 827
Connection: keep-alive
Boulder-Requester: 67892723
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: 0002tlC_lb_yp7cDOwFzz9aTy4i4aj2KeRjJu7T9xaeJXdc
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "nextcloud.panaccio.family"
  },
  "status": "pending",
  "expires": "2022-09-16T09:43:27Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "pending",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/3580687073/h4VMjw",
      "token": "zJy_d4Ck_jT-5kRtc4-TVW6_EfmNcBlPBWAhZ7kC4Pk"
    },
    {
      "type": "dns-01",
      "status": "pending",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/3580687073/srvmwA",
      "token": "zJy_d4Ck_jT-5kRtc4-TVW6_EfmNcBlPBWAhZ7kC4Pk"
    },
    {
      "type": "tls-alpn-01",
      "status": "pending",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/3580687073/Ek_9Ug",
      "token": "zJy_d4Ck_jT-5kRtc4-TVW6_EfmNcBlPBWAhZ7kC4Pk"
    }
  ]
}
Storing nonce: 0002tlC_lb_yp7cDOwFzz9aTy4i4aj2KeRjJu7T9xaeJXdc
JWS payload:
b''
Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3580687073:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS1zdGFnaW5nLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYWNjdC82Nzg5MjcyMyIsICJub25jZSI6ICIwMDAydGxDX2xiX3lwN2NET3dGeno5YVR5NGk0YWoyS2VSakp1N1Q5eGFlSlhkYyIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXN0YWdpbmctdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9hdXRoei12My8zNTgwNjg3MDczIn0",
  "signature": "GNB2Nif_zx2mqcJJBWklAnZEcEGokXSYg-hRy93sy-QWgWktxUz2Fas02XhNXaCyGhhv8e1FkDN2KGDXpE9CuT5a1S-ndY5BQcCzzYTTJ9D5ZW6JFPEES9oBnWHsa3EPy5YXayY52bakLCR4pNwytf8rrVLy2rQTLLvx9tw5ZhACBkb2z5cg-jsIvTa5S8F2JpmftPxpLBBVv8lT7rEP711PFRTRnP_-10YNX4IkfJ0TZ2DxFhS5SDel0DttzYvS2kn3Ftxd9pVz1K0gEzOPUS7Zk_efXgA8kpx6p7Zzx6CwTd_czAAPoookRocOs120Ixcn9Btl5QFliMe0wcTiNQ",
  "payload": ""
}
https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/3580687073 HTTP/1.1" 200 1104
Received response:
HTTP 200
Server: nginx
Date: Fri, 09 Sep 2022 09:43:42 GMT
Content-Type: application/json
Content-Length: 1104
Connection: keep-alive
Boulder-Requester: 67892723
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: 0002WWp_XB_eNxjx5kE7CLq1k5zdWWcChYxKuevYx9UuDP4
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "nextcloud.panaccio.family"
  },
  "status": "invalid",
  "expires": "2022-09-16T09:43:27Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "invalid",
      "error": {
        "type": "urn:ietf:params:acme:error:connection",
        "detail": "80.253.88.254: Fetching http://nextcloud.panaccio.family/.well-known/acme-challenge/zJy_d4Ck_jT-5kRtc4-TVW6_EfmNcBlPBWAhZ7kC4Pk: Timeout during connect (likely firewall problem)",
        "status": 400
      },
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/3580687073/h4VMjw",
      "token": "zJy_d4Ck_jT-5kRtc4-TVW6_EfmNcBlPBWAhZ7kC4Pk",
      "validationRecord": [
        {
          "url": "http://nextcloud.panaccio.family/.well-known/acme-challenge/zJy_d4Ck_jT-5kRtc4-TVW6_EfmNcBlPBWAhZ7kC4Pk",
          "hostname": "nextcloud.panaccio.family",
          "port": "80",
          "addressesResolved": [
            "80.253.88.254"
          ],
          "addressUsed": "80.253.88.254"
        }
      ],
      "validated": "2022-09-09T09:43:31Z"
    }
  ]
}
Storing nonce: 0002WWp_XB_eNxjx5kE7CLq1k5zdWWcChYxKuevYx9UuDP4
Challenge failed for domain nextcloud.panaccio.family
http-01 challenge for nextcloud.panaccio.family

Domain: nextcloud.panaccio.family
My web server is (include version):

Server version: Apache/2.4.54 (Debian)
Server built:   2022-06-09T04:26:43

The operating system my web server runs on is (include version):
Debian GNU/Linux 11 (bullseye) KVM guest, on a Debian GNU/Linux 11 (bullseye) host machine

My hosting provider, if applicable, is:
Home server
I can login to a root shell on my machine (yes or no, or I don't know):
yes
I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
No
The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

certbot 1.12.0
1 Like
2

Welcome to the community @wilya7

Thanks for the great first report. Very helpful to see all that right away.

It looks to me like a firewall is blocking the IP address(es) used by the Let's Encrypt servers. I can connect to your site from my own test server. And, the Let's Debug test site does two test connections. The first from its own server works but the one using the Let's Encrypt staging system times out.

Have you enabled a firewall for IP addresses since you last got a good cert?

PS: As for the redirect, certbot inserts temp changes to your Apache config which will control how the cert challenge is handled. You don't need to worry about that.

5 Likes
3

I appreciate your feedback and your help. I understand so little of all this that I was not sure what to report.

I have a fiber coming to my home from my service provider, which goes to an home made firewall running IPFire a specialized linux distribution made for firewalls. I run few rules that are not in anyway involved, plus a filter based on Spamhaus drop and some geolocation filter. I disabled everything now, but it seems that the test keeps failing. Either IPFire is malfunctioning or there is a firewall filtering upstream coming from my provider. Do you have any suggestion? Can I use another server with a different IP for the renew process?

As an alternative hypothesis, I cannot find any folder .well-known/ in the root directory of my server. Is it possible that I removed it (I do not recollect but it is possible) and now the script is malfunctioning?

4

Some routers have firewalls too. Maybe something there?

Maybe just restart after disabling the firewalls? Maybe something is just stuck

No, certbot apache plug-in will create that if/as needed. But, even if it could that would not be a timeout failure affecting only certain IP

Sure, as long as that server is pointed to by your DNS. Or, if you use port forwarding on your router

5 Likes
5

I my firewall is also my router, nothing else between my server and the provider gateway.

I did reboot the firewall. I am considering creating another virtual machine with a completely new Nextcloud install and test certbot there. If the problem is an IP filter, it should fail too. Of course I will report here.

As a work around, can I renew using a the DNS based approach or do I have to install a new certificate from scratch?

6

Possibly. If you can add/delete TXT records in your DNS you can manually get a cert. But, your current cert is good until Sep17 so you have time to sort this out.

If your DNS provider supports an API, and if certbot supports that DNS, you could automate the DNS challenge with it. The DNS Challenge is often more difficult to setup but if you can't find the IP block that may be the only way forward. (another acme client acme.sh supports many more DNS providers

5 Likes
7

Found the problem. IPFire in June introduced a security feature called "Reverse Path Filtering". I do not understand much what it means, however I believe it is a route that is dropped by the firewall kernel because it is considered unroutable.

I disabled the RPF by issuing these commands on the console of my Linux based firewall:

sysctl net.ipv4.conf.default.rp_filter=2
sysctl net.ipv4.conf.all.rp_filter=2

and the renewal went thorough beautifully.

@MikeMcQ thank you so much, you basically solved the problem for me.

Should I open a bug report with the certbot people?

1 Like
8

Glad I could help but I don't see any bug in certbot. You must allow inbound traffic from the Let's Encrypt servers for the HTTP challenge to succeed.

LE does not publish that list and the IP's often change. If you can allow inbound URL's that contain /.well-known/acme-challenge from any IP that would be best.

I don't think the Reverse Path Filtering will work with Let's Encrypt. If I understand it right, it will block inbound requests that it doesn't see as result of a prior outbound request. The inbound requests from Let's Encrypt servers intentionally come from previously unknown IP's to better ensure you, and only you, control that domain.

PS: thanks for letting us know exactly what you found

5 Likes
9

I don't think this is what causes the filtering, otherwise you could not connect to my home server to check it, as you did. I think it is more likely that between the server that issues the acme-challenge and its target there are forwarding hops in one direction and not in the other direction so the path is asymmetrical. That is why I was thinking to open a bug report, maybe they could make it symmetrical.

Edit: this is a better explanation of RPF.

Edit n.2: I opened this bug report. I believe this should be documented somewhere. The certbot people will decide if this is inappropriate and close the bug. Hopefully, now it will be easier to find this information using search engines.

Thank you again for your assistance.

2 Likes
10

If the block was triggered by a RPF check, it might be related to a routing misconfiguration on your equipment.

If you need any help reviewing that, please show the routing table in use by the router/firewall.

7 Likes
11

Yes, of course. My brain cramp. rg305 knows routing better than I do so maybe better progress on that with him.

Note that if there is a "bug" or something worth documenting it is a Let's Encrypt issue - not certbot. Certbot is just one of many ACME clients but it is the LE ACME Server behavior involved here.

5 Likes
12

I do not think the router is misconfigured. It is simply doing a DNAT on the web ports toward my web server in the DMZ (10.1.2.100).

Background: in the firewall/router, Orange0 is the DMZ. Blue0 is the wifi, green0 is the internal LAN. There is also a tun0 interface due to OpenVPN. I have a rule that does a DNAT on ports 80/443 from red0 (the interface to the WAN, basically the fiber coming from the provider) to orange0, specifically 10.1.2.100 (web server). This is the routing table on the firewall:

route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         80.253.88.1     0.0.0.0         UG    1004   0        0 red0
10.1.1.0        0.0.0.0         255.255.255.0   U     0      0        0 green0
10.1.2.0        0.0.0.0         255.255.255.0   U     0      0        0 orange0
10.1.3.0        0.0.0.0         255.255.255.0   U     0      0        0 blue0
10.1.4.0        10.1.4.2        255.255.255.0   UG    0      0        0 tun0
10.1.4.2        0.0.0.0         255.255.255.255 UH    0      0        0 tun0
80.253.88.0     0.0.0.0         255.255.255.0   U     1004   0        0 red0

this is the routing table of the server in the DMZ:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.1.2.1        0.0.0.0         UG    0      0        0 eth0
10.1.2.0        0.0.0.0         255.255.255.0   U     0      0        0 eth0

These are the kernel logs during a failed attempt at certbot renew:

Sep  9 14:52:26 ipfire kernel: DNAT IN= OUT=lo SRC=80.253.88.254 DST=80.253.88.254 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=34887 DF PROTO=TCP SPT=57720 DPT=443 WINDOW=65495 RES=0x00 SYN URGP=0 
Sep  9 14:52:27 ipfire kernel: DNAT IN= OUT=lo SRC=80.253.88.254 DST=80.253.88.254 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=49412 DF PROTO=TCP SPT=57732 DPT=443 WINDOW=65495 RES=0x00 SYN URGP=0 
Sep  9 14:52:27 ipfire kernel: DNAT IN= OUT=lo SRC=80.253.88.254 DST=80.253.88.254 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=40792 DF PROTO=TCP SPT=57742 DPT=443 WINDOW=65495 RES=0x00 SYN URGP=0 
Sep  9 14:52:27 ipfire kernel: DNAT IN= OUT=lo SRC=80.253.88.254 DST=80.253.88.254 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=57921 DF PROTO=TCP SPT=57752 DPT=443 WINDOW=65495 RES=0x00 SYN URGP=0 
Sep  9 14:52:29 ipfire kernel: DROP_INPUT IN=red0 OUT= MAC=00:0d:b9:42:68:92:00:00:f7:f7:02:56:08:00 SRC=89.248.163.237 DST=80.253.88.254 LEN=40 TOS=0x00 PREC=0x00 TTL=240 ID=51379 PROTO=TCP SPT=40441 DPT=1067 WINDOW=1024 RES=0x00 SYN URGP=0 
Sep  9 14:52:30 ipfire kernel: DROP_INPUT IN=red0 OUT= MAC=00:0d:b9:42:68:92:00:00:f7:f7:02:56:08:00 SRC=87.246.7.198 DST=80.253.88.254 LEN=40 TOS=0x00 PREC=0x00 TTL=241 ID=54594 PROTO=TCP SPT=48828 DPT=851 WINDOW=1024 RES=0x00 SYN URGP=0 
Sep  9 14:52:49 ipfire kernel: DROP_INPUT IN=red0 OUT= MAC=00:0d:b9:42:68:92:00:00:f7:f7:02:56:08:00 SRC=89.248.165.20 DST=80.253.88.254 LEN=40 TOS=0x00 PREC=0x00 TTL=240 ID=21813 PROTO=TCP SPT=44274 DPT=8153 WINDOW=1024 RES=0x00 SYN URGP=0 
Sep  9 14:52:53 ipfire kernel: DNAT IN=red0 OUT= MAC=00:0d:b9:42:68:92:00:00:f7:f7:02:56:08:00 SRC=18.119.10.60 DST=80.253.88.254 LEN=60 TOS=0x00 PREC=0x00 TTL=39 ID=57653 DF PROTO=TCP SPT=55690 DPT=80 WINDOW=62727 RES=0x00 SYN URGP=0 
Sep  9 14:52:53 ipfire kernel: FORWARDFW IN=red0 OUT=orange0 MAC=00:0d:b9:42:68:92:00:00:f7:f7:02:56:08:00 SRC=18.119.10.60 DST=10.1.2.100 LEN=60 TOS=0x00 PREC=0x00 TTL=38 ID=57653 DF PROTO=TCP SPT=55690 DPT=80 WINDOW=62727 RES=0x00 SYN URGP=0 
Sep  9 14:52:53 ipfire kernel: DNAT IN=red0 OUT= MAC=00:0d:b9:42:68:92:00:00:f7:f7:02:56:08:00 SRC=35.90.117.55 DST=80.253.88.254 LEN=60 TOS=0x00 PREC=0x00 TTL=39 ID=560 DF PROTO=TCP SPT=62832 DPT=80 WINDOW=62727 RES=0x00 SYN URGP=0 
Sep  9 14:52:53 ipfire kernel: FORWARDFW IN=red0 OUT=orange0 MAC=00:0d:b9:42:68:92:00:00:f7:f7:02:56:08:00 SRC=35.90.117.55 DST=10.1.2.100 LEN=60 TOS=0x00 PREC=0x00 TTL=38 ID=560 DF PROTO=TCP SPT=62832 DPT=80 WINDOW=62727 RES=0x00 SYN URGP=0 
Sep  9 14:52:53 ipfire kernel: DNAT IN=red0 OUT= MAC=00:0d:b9:42:68:92:00:00:f7:f7:02:56:08:00 SRC=23.178.112.107 DST=80.253.88.254 LEN=60 TOS=0x00 PREC=0x00 TTL=44 ID=439 DF PROTO=TCP SPT=17448 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0 
Sep  9 14:52:53 ipfire kernel: FORWARDFW IN=red0 OUT=orange0 MAC=00:0d:b9:42:68:92:00:00:f7:f7:02:56:08:00 SRC=23.178.112.107 DST=10.1.2.100 LEN=60 TOS=0x00 PREC=0x00 TTL=43 ID=439 DF PROTO=TCP SPT=17448 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0 
Sep  9 14:52:56 ipfire kernel: DNAT IN=red0 OUT= MAC=00:0d:b9:42:68:92:00:00:f7:f7:02:56:08:00 SRC=3.73.48.232 DST=80.253.88.254 LEN=60 TOS=0x00 PREC=0x00 TTL=44 ID=39482 DF PROTO=TCP SPT=23434 DPT=80 WINDOW=62727 RES=0x00 SYN URGP=0 
Sep  9 14:52:56 ipfire kernel: FORWARDFW IN=red0 OUT=orange0 MAC=00:0d:b9:42:68:92:00:00:f7:f7:02:56:08:00 SRC=3.73.48.232 DST=10.1.2.100 LEN=60 TOS=0x00 PREC=0x00 TTL=43 ID=39482 DF PROTO=TCP SPT=23434 DPT=80 WINDOW=62727 RES=0x00 SYN URGP=0 
Sep  9 14:52:56 ipfire kernel: DNAT IN= OUT=lo SRC=80.253.88.254 DST=80.253.88.254 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=28626 DF PROTO=TCP SPT=42700 DPT=443 WINDOW=65495 RES=0x00 SYN URGP=0 
Sep  9 14:53:21 ipfire kernel: DROP_INPUT IN=red0 OUT= MAC=00:0d:b9:42:68:92:00:00:f7:f7:02:56:08:00 SRC=122.202.54.44 DST=80.253.88.254 LEN=40 TOS=0x00 PREC=0x00 TTL=46 ID=39894 PROTO=TCP SPT=12289 DPT=23 WINDOW=14601 RES=0x00 SYN URGP=0 
Sep  9 14:53:22 ipfire kernel: DROP_INPUT IN=red0 OUT= MAC=00:0d:b9:42:68:92:00:00:f7:f7:02:56:08:00 SRC=167.248.133.141 DST=80.253.88.254 LEN=44 TOS=0x00 PREC=0x00 TTL=34 ID=48989 PROTO=TCP SPT=40600 DPT=5000 WINDOW=1024 RES=0x00 SYN URGP=0 
Sep  9 14:53:26 ipfire kernel: DNAT IN= OUT=lo SRC=80.253.88.254 DST=80.253.88.254 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=22818 DF PROTO=TCP SPT=60202 DPT=443 WINDOW=65495 RES=0x00 SYN URGP=0 
Sep  9 14:53:27 ipfire kernel: DNAT IN= OUT=lo SRC=80.253.88.254 DST=80.253.88.254 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=57316 DF PROTO=TCP SPT=60212 DPT=443 WINDOW=65495 RES=0x00 SYN URGP=0 
Sep  9 14:53:27 ipfire kernel: DNAT IN= OUT=lo SRC=80.253.88.254 DST=80.253.88.254 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=43527 DF PROTO=TCP SPT=60216 DPT=443 WINDOW=65495 RES=0x00 SYN URGP=0 
Sep  9 14:53:35 ipfire kernel: DROP_CTINVALID IN=red0 OUT= MAC=00:0d:b9:42:68:92:00:00:f7:f7:02:56:08:00 SRC=103.141.158.237 DST=80.253.88.254 LEN=40 TOS=0x00 PREC=0x00 TTL=45 ID=0 DF PROTO=TCP SPT=1812 DPT=44714 WINDOW=0 RES=0x00 ACK RST URGP=0 
Sep  9 14:53:36 ipfire kernel: DNAT IN=red0 OUT= MAC=00:0d:b9:42:68:92:00:00:f7:f7:02:56:08:00 SRC=51.81.167.146 DST=80.253.88.254 LEN=60 TOS=0x00 PREC=0x00 TTL=45 ID=7154 DF PROTO=TCP SPT=50720 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0 
Sep  9 14:53:36 ipfire kernel: FORWARDFW IN=red0 OUT=orange0 MAC=00:0d:b9:42:68:92:00:00:f7:f7:02:56:08:00 SRC=51.81.167.146 DST=10.1.2.100 LEN=60 TOS=0x00 PREC=0x00 TTL=44 ID=7154 DF PROTO=TCP SPT=50720 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0 
Sep  9 14:53:36 ipfire kernel: DNAT IN=red0 OUT= MAC=00:0d:b9:42:68:92:00:00:f7:f7:02:56:08:00 SRC=51.81.167.146 DST=80.253.88.254 LEN=60 TOS=0x00 PREC=0x00 TTL=45 ID=52310 DF PROTO=TCP SPT=46098 DPT=443 WINDOW=64240 RES=0x00 SYN URGP=0 
Sep  9 14:53:36 ipfire kernel: FORWARDFW IN=red0 OUT=orange0 MAC=00:0d:b9:42:68:92:00:00:f7:f7:02:56:08:00 SRC=51.81.167.146 DST=10.1.2.100 LEN=60 TOS=0x00 PREC=0x00 TTL=44 ID=52310 DF PROTO=TCP SPT=46098 DPT=443 WINDOW=64240 RES=0x00 SYN URGP=0 
Sep  9 14:53:42 ipfire kernel: DROP_INPUT IN=red0 OUT= MAC=00:0d:b9:42:68:92:00:00:f7:f7:02:56:08:00 SRC=121.231.79.51 DST=80.253.88.254 LEN=40 TOS=0x00 PREC=0x00 TTL=49 ID=64974 PROTO=TCP SPT=3917 DPT=23 WINDOW=12405 RES=0x00 SYN URGP=0 
Sep  9 14:53:56 ipfire kernel: DNAT IN= OUT=lo SRC=80.253.88.254 DST=80.253.88.254 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=22934 DF PROTO=TCP SPT=41250 DPT=443 WINDOW=65495 RES=0x00 SYN URGP=0

after disabling the RPF, everything worked as before IPFire people (the developers of my firewall) introduced by default RPF in the kernel setting.

1 Like
13

Thanks, I will close the bug report there. Do you know how to reach the right party?

2 Likes
14

Let's Encrypt staff routinely monitor this support forum.

I appreciate your frustration but I'm not sure there's much to do. I think firewalls are the most common failure we see and there are large varieties of things which cause problems (broad IP blocks, geo blocking, acme protocol blocks, ...). It is impractical to document every problematic firewall setting (and then keep it up to date).

If it becomes a more frequent problem we might create a sticky thread about it. For example, the Palo Alto Networks company added a new setting on their gear earlier this year which blocked ACME challenges. Different models of their gear block it differently and it's been a challenge. I've closed more than a dozen threads from that alone.

You might also want to post in any IPFire forum. Alerting people there of RPF interfering with Let's Encrypt might be helpful. It's possible other ACME CA's will be similarly affected.

I appreciate that you discovered this and explained it so well. It will help should we see IPFire mentioned in the future.

5 Likes
15

Great!

Thanks, but I am not frustrated. I just wanted to do the right thing. I understand enough of these things to appreciate how complicated these interconnected and layered systems are, and how easy it's for something to go wrong. Thank you for assisting clueless users like me, so well.

As you suggested, I opened a thread in IPFire forum. I think this behavior might be relevant also in other systems where RPF strict becomes the default setting.

2 Likes
16

I like that! I changed your topic heading to be more descriptive for future Searchers

6 Likes
17

That looks correct.
I think it might be something more in the default definitions of RFC 1918 networks that could be incorrectly defined.

4 Likes
18

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.