Problem with certbot with ubuntu server 22.04.3 LTS

My domain is: sesstim4.timone.univ-amu.fr

I ran this command: sudo certbot -v --apache

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Enter email address (used for urgent renewal and security notices)
 (Enter 'c' to cancel): jean-charles.dufour@univ-amu.fr

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Please read the Terms of Service at
https://letsencrypt.org/documents/LE-SA-v1.3-September-21-2022.pdf. You must
agree in order to register with the ACME server. Do you agree?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: Y

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Would you be willing, once your first certificate is successfully issued, to
share your email address with the Electronic Frontier Foundation, a founding
partner of the Let's Encrypt project and the non-profit organization that
develops Certbot? We'd like to send you email about our work encrypting the web,
EFF news, campaigns, and ways to support digital freedom.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: Y
Account registered.

Which names would you like to activate HTTPS for?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: sesstim4.timone.univ-amu.fr
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel):
Requesting a certificate for sesstim4.timone.univ-amu.fr
Performing the following challenges:
http-01 challenge for sesstim4.timone.univ-amu.fr
Waiting for verification...
Challenge failed for domain sesstim4.timone.univ-amu.fr
http-01 challenge for sesstim4.timone.univ-amu.fr

Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:
  Domain: sesstim4.timone.univ-amu.fr
  Type:   connection
  Detail: 139.124.158.220: Fetching http://sesstim4.timone.univ-amu.fr/.well-known/acme-challenge/IsMkt-drXl9s1dch_25Ca6ESRQJEtOHiCu2wgv4gscI: Connection reset by peer

Hint: The Certificate Authority failed to verify the temporary Apache configuration changes made by Certbot. Ensure that the listed domains point to this Apache server and that it is accessible from the internet.

Cleaning up challenges
Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

My web server is (include version): Apache/2.4.52 (Ubuntu) buitl: 2023-05-03T20:02:51

The operating system my web server runs on is (include version): ubuntu server 22.04.3 LTS

My hosting provider, if applicable, is: Aix Marseille Université

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 1.21.0

I suspect a problem with WSGI because I have Flask applications served by apache. However my virtual-host enabled should handle this. I don't understand what's wrong. Perhaps a legal problem? Because the "acme-challenge" folder is not created after the sudo certbot -v --apache command.

Here are my virtual-host.conf and virtual-host-ssl.conf files (for the moment I'm temporarily using ZeroSSL, but I'd like to switch to certbot/let's encrypt)

<VirtualHost *:80>
	ServerName sesstim4.timone.univ-amu.fr
#	ServerAlias sesstim4.timone.univ-amu.fr

	ServerAdmin jean-charles.dufour@univ-amu.fr
	DocumentRoot /var/www/html

	# pour application agendaradar
        # Utilisation de IfDefine pour gérer le bug de certbot cf. ici https://github.com/certbot/certbot/issues/8373
        <IfDefine !wsgi_init>
	    WSGIDaemonProcess agendaradar python-path=/var/www/jcd_flask_apps/agendaradar:/var/www/jcd_flask_apps/agendaradar/venv/lib/python3.10/site-packages
	    WSGIProcessGroup agendaradar
	    WSGIScriptAlias /agendaradar /var/www/jcd_flask_apps/agendaradar/agendaradar.wsgi
            Define wsgi_init 1
        </IfDefine>	
	<Directory /var/www/jcd_flask_apps/agendaradar>
		Require all granted
	</Directory>

        # pour application agendareconduction
        # Utilisation de IfDefine pour gérer le bug de certbot cf. ici https://github.com/certbot/certbot/issues/8373
	<IfDefine !wsgi_init_agendareconduction>
	    WSGIDaemonProcess agendareconduction python-path=/var/www/jcd_flask_apps/agendareconduction:/var/www/jcd_flask_apps/agendareconduction/venv/lib/python3.10/site-packages
	    WSGIProcessGroup agendareconduction
	    WSGIScriptAlias /agendareconduction /var/www/jcd_flask_apps/agendareconduction/agendareconduction.wsgi
	    Define wsgi_init_agendareconduction 1
	</IfDefine>

	<Directory /var/www/jcd_flask_apps/agendareconduction>
	    Require all granted
	</Directory>

	# Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
	# error, crit, alert, emerg.
	# It is also possible to configure the loglevel for particular
	# modules, e.g.
	#LogLevel info ssl:warn

	ErrorLog ${APACHE_LOG_DIR}/error.log
	CustomLog ${APACHE_LOG_DIR}/access.log combined

	# For most configuration files from conf-available/, which are
	# enabled or disabled at a global level, it is possible to
	# include a line for only one particular virtual host. For example the
	# following line enables the CGI configuration for this host only
	# after it has been globally disabled with "a2disconf".
	#Include conf-available/serve-cgi-bin.conf
#RewriteEngine on
#RewriteCond %{SERVER_NAME} =sesstim4.timone.univ-amu.fr
#RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>

<IfModule mod_ssl.c>
<VirtualHost *:443>
	# The ServerName directive sets the request scheme, hostname and port that
	# the server uses to identify itself. This is used when creating
	# redirection URLs. In the context of virtual hosts, the ServerName
	# specifies what hostname must appear in the request's Host: header to
	# match this virtual host. For the default virtual host (this file) this
	# value is not decisive as it is used as a last resort host regardless.
	# However, you must set it for any further virtual host explicitly.
	ServerName sesstim4.timone.univ-amu.fr
	ServerAdmin jean-charles.dufour@univ-amu.fr
	DocumentRoot /var/www/html

	SSLEngine on
	SSLCertificateFile /etc/ssl/certificate.crt
	SSLCertificateKeyFile /etc/ssl/private/private.key
	SSLCertificateChainFile /etc/ssl/ca_bundle.crt

	# pour application agendaradar
        
        # Utilisation de IfDefine pour gérer le bug de certbot cf. ici https://github.com/certbot/certbot/issues/8373
        <IfDefine !wsgi_init>
		WSGIDaemonProcess agendaradar python-path=/var/www/jcd_flask_apps/agendaradar:/var/www/jcd_flask_apps/agendaradar/venv/lib/python3.10/site-packages
		#WSGIProcessGroup agendaradar
		<Location /agendaradar>
		    WSGIProcessGroup agendaradar
		</Location>	    
		WSGIScriptAlias /agendaradar /var/www/jcd_flask_apps/agendaradar/agendaradar.wsgi
		Define wsgi_init 1
        </IfDefine>	

	<Directory /var/www/jcd_flask_apps/agendaradar>
		Require all granted
	</Directory>

        # pour application agendareconduction
        # Utilisation de IfDefine pour gérer le bug de certbot cf. ici https://github.com/certbot/certbot/issues/8373
        <IfDefine !wsgi_init_agendareconduction>
		WSGIDaemonProcess agendareconduction python-path=/var/www/jcd_flask_apps/agendareconduction:/var/www/jcd_flask_apps/agendareconduction/venv/lib/python3.10/site-packages
		#WSGIProcessGroup agendareconduction
		<Location /agendareconduction>
			WSGIProcessGroup agendareconduction
		</Location>
		WSGIScriptAlias /agendareconduction /var/www/jcd_flask_apps/agendareconduction/agendareconduction.wsgi
		Define wsgi_init_agendareconduction 1
        </IfDefine>

        <Directory /var/www/jcd_flask_apps/agendareconduction>
            Require all granted
        </Directory>

	# Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
	# error, crit, alert, emerg.
	# It is also possible to configure the loglevel for particular
	# modules, e.g.
	#LogLevel info ssl:warn

	ErrorLog ${APACHE_LOG_DIR}/error.log
	CustomLog ${APACHE_LOG_DIR}/access.log combined

	# For most configuration files from conf-available/, which are
	# enabled or disabled at a global level, it is possible to
	# include a line for only one particular virtual host. For example the
	# following line enables the CGI configuration for this host only
	# after it has been globally disabled with "a2disconf".
	#Include conf-available/serve-cgi-bin.conf
</VirtualHost>

</IfModule>

Thank you for your help.

1 Like

Hi @jcdufour, and welcome to the LE community forum :slight_smile:

It seems that LE can't reach your site:

3 Likes

Using the online tool Let's Debug yields these results https://letsdebug.net/sesstim4.timone.univ-amu.fr/1676006

ANotWorking
ERROR
sesstim4.timone.univ-amu.fr has an A (IPv4) record (139.124.158.220) but a request to this address over port 80 did not succeed. Your web server must have at least one working IPv4 or IPv6 address.
Get "http://sesstim4.timone.univ-amu.fr/.well-known/acme-challenge/letsdebug-test": read tcp 172.104.24.29:36006->139.124.158.220:80: read: connection reset by peer

Trace:
@0ms: Making a request to http://sesstim4.timone.univ-amu.fr/.well-known/acme-challenge/letsdebug-test (using initial IP 139.124.158.220)
@0ms: Dialing 139.124.158.220
@1198ms: Experienced error: read tcp 172.104.24.29:36006->139.124.158.220:80: read: connection reset by peer
IssueFromLetsEncrypt
ERROR
A test authorization for sesstim4.timone.univ-amu.fr to the Let's Encrypt staging service has revealed issues that may prevent any certificate for this domain being issued.
139.124.158.220: Fetching http://sesstim4.timone.univ-amu.fr/.well-known/acme-challenge/UUOPrMZMnLilOvDrVIoVP8svITn_-hAhe87Q9xIdQqg: Connection reset by peer

Do per chance have a Palo Alto Firewall?

$ curl -Ii http://sesstim4.timone.univ-amu.fr/.well-known/acme-challenge/sometestfile
HTTP/1.1 404 Not Found
Date: Wed, 15 Nov 2023 03:12:09 GMT
Server: Apache/2.4.52 (Ubuntu)
Content-Type: text/html; charset=iso-8859-1
$ curl -Ii http://sesstim4.timone.univ-amu.fr/.well-known/acme-challenge/sometestfile -A "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
curl: (56) Recv failure: Connection reset by peer
2 Likes

Hi, nice to have your help. Thank you

As mentioned, the .well-known/acme-challenge folder was not created by certbot which explains why my server seems inaccessible. I thought certbot created this file automatically. Is this not the case?

Anyway, I've now created this folder "by hand" and placed a testfile in it, which can be accessed via the URL http://sesstim4.timone.univ-amu.fr/.well-known/acme-challenge/testfile.

It can be accessed from a browser. It seems to me that this proves that the server is giving access to this folder and is not blocking port 80.

As for the firewall: I'm actually in a university network that uses Palo Alto. I don't have control over the firewall, but I've contacted the network team, who assure me that there's no problem there (there are several hundred servers in my university behind Palo Alto that use certbot without any problem).

Here after what the sudo certbot -vvv --apache command produce
do you have any other suggestions?
Thanks

jcdufour@sesstim4:/var/www/html/.well-known/acme-challenge$ sudo certbot -vvv --apache

Root logging level set at 0
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Requested authenticator apache and installer apache
Apache version is 2.4.52
Single candidate plugin: * apache
Description: Apache Web Server plugin
Interfaces: Installer, Authenticator, Plugin
Entry point: apache = certbot_apache._internal.entrypoint:ENTRYPOINT
Initialized: <certbot_apache._internal.override_debian.DebianConfigurator object at 0x7f85d78f90c0>
Prep: True
Selected authenticator <certbot_apache._internal.override_debian.DebianConfigurator object at 0x7f85d78f90c0> and installer <certbot_apache._internal.override_debian.DebianConfigurator object at 0x7f85d78f90c0>
Plugins selected: Authenticator apache, Installer apache
Picked account: <Account(RegistrationResource(body=Registration(key=None, contact=(), agreement=None, status=None, terms_of_service_agreed=None, only_return_existing=None, external_account_binding=None), uri='https://acme-v02.api.letsencrypt.org/acme/acct/1413788386', new_authzr_uri=None, terms_of_service=None), a5b172f1665b034c979b810373a72736, Meta(creation_dt=datetime.datetime(2023, 11, 14, 16, 18, 8, tzinfo=<UTC>), creation_host='sesstim4', register_to_eff='jean-charles.dufour@univ-amu.fr'))>
Sending GET request to https://acme-v02.api.letsencrypt.org/directory.
Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org:443
https://acme-v02.api.letsencrypt.org:443 "GET /directory HTTP/1.1" 200 752
Received response:
HTTP 200
Server: nginx
Date: Wed, 15 Nov 2023 10:03:08 GMT
Content-Type: application/json
Content-Length: 752
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.3-September-21-2022.pdf",
    "website": "https://letsencrypt.org"
  },
  "newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
  "rdPxN8zJXnQ": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
  "renewalInfo": "https://acme-v02.api.letsencrypt.org/draft-ietf-acme-ari-01/renewalInfo/",
  "revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert"
}

Which names would you like to activate HTTPS for?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: sesstim4.timone.univ-amu.fr
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel):
Notifying user: Requesting a certificate for sesstim4.timone.univ-amu.fr
Requesting a certificate for sesstim4.timone.univ-amu.fr
Generating RSA key (2048 bits): /etc/letsencrypt/keys/0002_key-certbot.pem
Creating CSR: /etc/letsencrypt/csr/0002_csr-certbot.pem
Requesting fresh nonce
Sending HEAD request to https://acme-v02.api.letsencrypt.org/acme/new-nonce.
https://acme-v02.api.letsencrypt.org:443 "HEAD /acme/new-nonce HTTP/1.1" 200 0
Received response:
HTTP 200
Server: nginx
Date: Wed, 15 Nov 2023 10:03:10 GMT
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: GEqhmX18HCFzAX42hZcKkX5YYJT4eh-GMZKpO24JiTLQKE73pSs
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800


Storing nonce: GEqhmX18HCFzAX42hZcKkX5YYJT4eh-GMZKpO24JiTLQKE73pSs
JWS payload:
b'{\n  "identifiers": [\n    {\n      "type": "dns",\n      "value": "sesstim4.timone.univ-amu.fr"\n    }\n  ]\n}'
Sending POST request to https://acme-v02.api.letsencrypt.org/acme/new-order:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2FjY3QvMTQxMzc4ODM4NiIsICJub25jZSI6ICJHRXFobVgxOEhDRnpBWDQyaFpjS2tYNVlZSlQ0ZWgtR01aS3BPMjRKaVRMUUtFNzNwU3MiLCAidXJsIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL25ldy1vcmRlciJ9",
  "signature": "LoSdtxWT6Vamm68u3WLZkKANdrMRZpDBShnl_Dm9aBnCIs5CuvFHrr8tyozy-o0j2To_k8PM9Bhp8-LnfMTs6NoXKD7R4pEIYYq_0ApmglOS0rnLpfFGU89Y6i4NWXFqDR6E4cwctkuyDhhB0KF6FE1j0M_WTlpfbUHUGPs3R7VDPXD3okO-m4lWAQhl8U5SNrX9dbbElDezcbFv3AdXVY6DtPrs48MTCfI4iuGdHsE_5bEfXYHz1XBoTEKeoftetiieEKrD2G8Q6AeAmKQBvMi-geBAzmFwQiXlbgZ62r2CFQEacsavvbPqgze0kdwumBBLuxMq_S5wye-ZOYhJKA",
  "payload": "ewogICJpZGVudGlmaWVycyI6IFsKICAgIHsKICAgICAgInR5cGUiOiAiZG5zIiwKICAgICAgInZhbHVlIjogInNlc3N0aW00LnRpbW9uZS51bml2LWFtdS5mciIKICAgIH0KICBdCn0"
}
https://acme-v02.api.letsencrypt.org:443 "POST /acme/new-order HTTP/1.1" 201 353
Received response:
HTTP 201
Server: nginx
Date: Wed, 15 Nov 2023 10:03:10 GMT
Content-Type: application/json
Content-Length: 353
Connection: keep-alive
Boulder-Requester: 1413788386
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
Location: https://acme-v02.api.letsencrypt.org/acme/order/1413788386/222349468736
Replay-Nonce: fTELRHZhymoYJ2ViP2I6p9v5RhN8v8lKHQHAieP7zOJboYEi7bg
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "status": "pending",
  "expires": "2023-11-22T10:03:10Z",
  "identifiers": [
    {
      "type": "dns",
      "value": "sesstim4.timone.univ-amu.fr"
    }
  ],
  "authorizations": [
    "https://acme-v02.api.letsencrypt.org/acme/authz-v3/284026121066"
  ],
  "finalize": "https://acme-v02.api.letsencrypt.org/acme/finalize/1413788386/222349468736"
}
Storing nonce: fTELRHZhymoYJ2ViP2I6p9v5RhN8v8lKHQHAieP7zOJboYEi7bg
JWS payload:
b''
Sending POST request to https://acme-v02.api.letsencrypt.org/acme/authz-v3/284026121066:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2FjY3QvMTQxMzc4ODM4NiIsICJub25jZSI6ICJmVEVMUkhaaHltb1lKMlZpUDJJNnA5djVSaE44djhsS0hRSEFpZVA3ek9KYm9ZRWk3YmciLCAidXJsIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2F1dGh6LXYzLzI4NDAyNjEyMTA2NiJ9",
  "signature": "MsvneTQAn0KP35EIOKu6iSMtU_2wTFLGzaLit5aaVlv86G2qctZiRhXWxzuTNeTQp8mfPzCIp1xEiLg8cISfuSNkwGj9WWsK8NYR2U8SaytCcOK9e4W0i5Gz6gPD4a6y7X1RyVMBVh3DAy6NMSFF-Xbf0ul7rfrXHqZ_BLkTNXqaLB2mqenziGbFuFQzLRYZW3r3dz9PkSbLzxvvtqfbv45ociGByLTnowSQ1hDl9v9vMrt5eXQ_O7q4IUuHV2GeTrKS6f3VVyh2QsWUTr9seYyaixTnwZUmRLeFg7lqIGuW7DHzilOQfAY0ySvKNc9gcrtcFupYfHOHTyymO4c-ow",
  "payload": ""
}
https://acme-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/284026121066 HTTP/1.1" 200 811
Received response:
HTTP 200
Server: nginx
Date: Wed, 15 Nov 2023 10:03:10 GMT
Content-Type: application/json
Content-Length: 811
Connection: keep-alive
Boulder-Requester: 1413788386
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: fTELRHZhNtRTkrEobZVbwWUoBSfO3zxdhdBEa2RCsg0aFzG1g24
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "sesstim4.timone.univ-amu.fr"
  },
  "status": "pending",
  "expires": "2023-11-22T10:03:10Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/284026121066/yYDgcw",
      "token": "bD6XBucdAWhnvDr4qKq9asHMzI6vHOmtWK6T9VJ5cTM"
    },
    {
      "type": "dns-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/284026121066/n5iRKw",
      "token": "bD6XBucdAWhnvDr4qKq9asHMzI6vHOmtWK6T9VJ5cTM"
    },
    {
      "type": "tls-alpn-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/284026121066/3Ij10w",
      "token": "bD6XBucdAWhnvDr4qKq9asHMzI6vHOmtWK6T9VJ5cTM"
    }
  ]
}
Storing nonce: fTELRHZhNtRTkrEobZVbwWUoBSfO3zxdhdBEa2RCsg0aFzG1g24
Performing the following challenges:
http-01 challenge for sesstim4.timone.univ-amu.fr
Adding a temporary challenge validation Include for name: sesstim4.timone.univ-amu.fr in: /etc/apache2/sites-enabled/virtual-host.conf
Adding a temporary challenge validation Include for name: sesstim4.timone.univ-amu.fr in: /etc/apache2/sites-enabled/virtual-host-ssl.conf
writing a pre config file with text:
         RewriteEngine on
        RewriteRule ^/\.well-known/acme-challenge/([A-Za-z0-9-_=]+)$ /var/lib/letsencrypt/http_challenges/$1 [END]

writing a post config file with text:
         <Directory /var/lib/letsencrypt/http_challenges>
            Require all granted
        </Directory>
        <Location /.well-known/acme-challenge>
            Require all granted
        </Location>

Creating backup of /etc/apache2/sites-enabled/virtual-host-ssl.conf
Creating backup of /etc/apache2/sites-enabled/virtual-host.conf
JWS payload:
b'{}'
Sending POST request to https://acme-v02.api.letsencrypt.org/acme/chall-v3/284026121066/yYDgcw:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2FjY3QvMTQxMzc4ODM4NiIsICJub25jZSI6ICJmVEVMUkhaaE50UlRrckVvYlpWYndXVW9CU2ZPM3p4ZGhkQkVhMlJDc2cwYUZ6RzFnMjQiLCAidXJsIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2NoYWxsLXYzLzI4NDAyNjEyMTA2Ni95WURnY3cifQ",
  "signature": "APtdMAFue8fA9PYUEdzWJ-MCgew99PCdc0vimbArDbXn8fKI66_KgGc5iuEJydkYHqjN8TstS0tnJfcIfk22WL6UyAT2aYjQOrJI_CaaKIkBbJx9sZDsp7SJlNnegRVVcJ55t1lMyqUMrkTQnMhEw2tf4PO4rOpJVKMinVs00H_q43bfkBolgo8bkzReVY2uoAOSqL2Oh-Ck-ga0YZ8ifQzyWFXubSLCz9D9Wln9ToWCUIQKu6WxlGIhgCcajkN8yVkJqhngnoCC_uYKpY2PDLbYewywIZuylt1ndG60TCW89vxIwKeosImM_3DFn6M3u6MHvt3EgTeCQwoDyfiL2A",
  "payload": "e30"
}
https://acme-v02.api.letsencrypt.org:443 "POST /acme/chall-v3/284026121066/yYDgcw HTTP/1.1" 200 187
Received response:
HTTP 200
Server: nginx
Date: Wed, 15 Nov 2023 10:03:14 GMT
Content-Type: application/json
Content-Length: 187
Connection: keep-alive
Boulder-Requester: 1413788386
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index", <https://acme-v02.api.letsencrypt.org/acme/authz-v3/284026121066>;rel="up"
Location: https://acme-v02.api.letsencrypt.org/acme/chall-v3/284026121066/yYDgcw
Replay-Nonce: fTELRHZhuPeBGP8X1NPkvcKSnZxO2niIdxcFwUNweDsHvqEtjQs
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "type": "http-01",
  "status": "pending",
  "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/284026121066/yYDgcw",
  "token": "bD6XBucdAWhnvDr4qKq9asHMzI6vHOmtWK6T9VJ5cTM"
}
Storing nonce: fTELRHZhuPeBGP8X1NPkvcKSnZxO2niIdxcFwUNweDsHvqEtjQs
Waiting for verification...
JWS payload:
b''
Sending POST request to https://acme-v02.api.letsencrypt.org/acme/authz-v3/284026121066:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2FjY3QvMTQxMzc4ODM4NiIsICJub25jZSI6ICJmVEVMUkhaaHVQZUJHUDhYMU5Qa3ZjS1NuWnhPMm5pSWR4Y0Z3VU53ZURzSHZxRXRqUXMiLCAidXJsIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2F1dGh6LXYzLzI4NDAyNjEyMTA2NiJ9",
  "signature": "UaioYdoIKiDYDfjbzpA0nzsEBmhVK38QWehHf16-JUQvkWQRRByz29kIv5G7Jd09ABqbFCGGxyWlUoVliwgMwadTMkCY6D0-M_QlF7nLqz3yi9niPQh0UEZE7uQfdfC3CaTkBmUgt5RAeZ9YlhL_llJ2RynvDCEVmfNCGqcALU1XnfVYB3WK7yxYlERmCWNJ32YCTHvDoNjqvbK7R4Wq9twrCWSGm1O8l3PaakSAC8zcLB3G5_3SN3dpOnDYTdlsf_UZhkscUppq56GLVzilScOxhIErnAEuit3NttYYAakHBXTMHoW6hp8NREJULsYYBk5cJ3CnAS-ocm476OuHgg",
  "payload": ""
}
https://acme-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/284026121066 HTTP/1.1" 200 1088
Received response:
HTTP 200
Server: nginx
Date: Wed, 15 Nov 2023 10:03:15 GMT
Content-Type: application/json
Content-Length: 1088
Connection: keep-alive
Boulder-Requester: 1413788386
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: fTELRHZhwqWQ_X06Gy9Zny9vEu_4cowCyOWezsQFzOAYRDlb644
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "sesstim4.timone.univ-amu.fr"
  },
  "status": "invalid",
  "expires": "2023-11-22T10:03:10Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "invalid",
      "error": {
        "type": "urn:ietf:params:acme:error:connection",
        "detail": "139.124.158.220: Fetching http://sesstim4.timone.univ-amu.fr/.well-known/acme-challenge/bD6XBucdAWhnvDr4qKq9asHMzI6vHOmtWK6T9VJ5cTM: Connection reset by peer",
        "status": 400
      },
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/284026121066/yYDgcw",
      "token": "bD6XBucdAWhnvDr4qKq9asHMzI6vHOmtWK6T9VJ5cTM",
      "validationRecord": [
        {
          "url": "http://sesstim4.timone.univ-amu.fr/.well-known/acme-challenge/bD6XBucdAWhnvDr4qKq9asHMzI6vHOmtWK6T9VJ5cTM",
          "hostname": "sesstim4.timone.univ-amu.fr",
          "port": "80",
          "addressesResolved": [
            "139.124.158.220"
          ],
          "addressUsed": "139.124.158.220"
        }
      ],
      "validated": "2023-11-15T10:03:14Z"
    }
  ]
}
Storing nonce: fTELRHZhwqWQ_X06Gy9Zny9vEu_4cowCyOWezsQFzOAYRDlb644
Challenge failed for domain sesstim4.timone.univ-amu.fr
http-01 challenge for sesstim4.timone.univ-amu.fr
Notifying user:
Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:
  Domain: sesstim4.timone.univ-amu.fr
  Type:   connection
  Detail: 139.124.158.220: Fetching http://sesstim4.timone.univ-amu.fr/.well-known/acme-challenge/bD6XBucdAWhnvDr4qKq9asHMzI6vHOmtWK6T9VJ5cTM: Connection reset by peer

Hint: The Certificate Authority failed to verify the temporary Apache configuration changes made by Certbot. Ensure that the listed domains point to this Apache server and that it is accessible from the internet.


Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:
  Domain: sesstim4.timone.univ-amu.fr
  Type:   connection
  Detail: 139.124.158.220: Fetching http://sesstim4.timone.univ-amu.fr/.well-known/acme-challenge/bD6XBucdAWhnvDr4qKq9asHMzI6vHOmtWK6T9VJ5cTM: Connection reset by peer

Hint: The Certificate Authority failed to verify the temporary Apache configuration changes made by Certbot. Ensure that the listed domains point to this Apache server and that it is accessible from the internet.

Encountered exception:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/certbot/_internal/auth_handler.py", line 90, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, best_effort)
  File "/usr/lib/python3/dist-packages/certbot/_internal/auth_handler.py", line 178, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.

Calling registered functions
Cleaning up challenges
Exiting abnormally:
Traceback (most recent call last):
  File "/usr/bin/certbot", line 33, in <module>
    sys.exit(load_entry_point('certbot==1.21.0', 'console_scripts', 'certbot')())
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 15, in main
    return internal_main.main(cli_args)
  File "/usr/lib/python3/dist-packages/certbot/_internal/main.py", line 1574, in main
    return config.func(config, plugins)
  File "/usr/lib/python3/dist-packages/certbot/_internal/main.py", line 1287, in run
    new_lineage = _get_and_save_cert(le_client, config, domains,
  File "/usr/lib/python3/dist-packages/certbot/_internal/main.py", line 133, in _get_and_save_cert
    lineage = le_client.obtain_and_enroll_certificate(domains, certname)
  File "/usr/lib/python3/dist-packages/certbot/_internal/client.py", line 459, in obtain_and_enroll_certificate
    cert, chain, key, _ = self.obtain_certificate(domains)
  File "/usr/lib/python3/dist-packages/certbot/_internal/client.py", line 389, in obtain_certificate
    orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
  File "/usr/lib/python3/dist-packages/certbot/_internal/client.py", line 439, in _get_order_and_authorizations
    authzr = self.auth_handler.handle_authorizations(orderr, self.config, best_effort)
  File "/usr/lib/python3/dist-packages/certbot/_internal/auth_handler.py", line 90, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, best_effort)
  File "/usr/lib/python3/dist-packages/certbot/_internal/auth_handler.py", line 178, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.
Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

certbot will only create folders when they don't already exist.
If any folders were created previously and they don't have the right permissions, this could cause trouble.

As shown by @Bruce5051, that doesn't look like the trouble you are currently facing.

Does the school use Palo-Alto firewall equipment?

2 Likes

OP have already answered this question:

1 Like

hmm...

Then this is a new problem!

Which is made evident by these two requests:

curl -Ii http://sesstim4.timone.univ-amu.fr/.well-known/acme-challenge/sometestfile -A "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
curl: (52) Empty reply from server

curl -Ii http://sesstim4.timone.univ-amu.fr/.well-known/acme-challenge/sometestfile
HTTP/1.1 404 Not Found
Date: Wed, 15 Nov 2023 14:25:43 GMT
Server: Apache/2.4.52 (Ubuntu)
Content-Type: text/html; charset=iso-8859-1
2 Likes

the test file I have created is nammed "testfile" (not "sometestfile")

That is not relevant.
The fact that when the user agent is added the request is blocked is enough proof.

But to satisfy your request, here are the "testfile" requests [and exact same result]:

PASS [with 7 bytes]:

curl -Ii http://sesstim4.timone.univ-amu.fr/.well-known/acme-challenge/testfile
HTTP/1.1 200 OK
Date: Wed, 15 Nov 2023 15:05:29 GMT
Server: Apache/2.4.52 (Ubuntu)
Last-Modified: Wed, 15 Nov 2023 09:48:51 GMT
ETag: "7-60a2dcf879284"
Accept-Ranges: bytes
Content-Length: 7

FAIL [with zero bytes]:

curl -Ii http://sesstim4.timone.univ-amu.fr/.well-known/acme-challenge/testfile -A "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
curl: (52) Empty reply from server
3 Likes

But certbot have not created such folders. I have to done it manually...strange if you confirm tant certbot create these folders if they are not previously created.

The folder is not relevant either currently.

Only the fact that the response is completely different when adding the Let's Encrypt validation server user agent header matters.

TL;DR: without the LE validation server user agent header, the response is an expected 404 file not found or expected 200 OK (depending on which file is requested), but WITH the LE validation server user agent header, the response is "connection reset by peer", which is BAD (and most likely a firewall).

To be exact: the string "Let's Encrypt validation server;" (including the ;) results in the connection reset by peer result. So something is triggering on that string and most likely the Palo Alto firewall. As this is exactly what we've seen before. (And it wouldn't be the first time some firewall operator claims everything should be good to go, while it actually wasn't.)

4 Likes

Monitoring with tcpdump, here's what I get when I run sudo certbot -vvv --apache . There is no communication using port 80, even at first. Is this normal?

jcdufour@sesstim4:~$ sudo tcpdump -i any 'tcp port 80 or tcp port 443' and host acme-v02.api.letsencrypt.org
[sudo] Mot de passe de jcdufour :

tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
13:08:01.529188 eno1  Out IP sesstim4.timone.univ-amu.fr.45650 > 172.65.32.248.https: Flags [S], seq 163416455, win 64240, options [mss 1460,sackOK,TS val 2146305057 ecr 0,nop,wscale 7], length 0
13:08:01.545351 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.45650: Flags [S.], seq 1659856072, ack 163416456, win 65160, options [mss 1400,sackOK,TS val 268246586 ecr 2146305057,nop,wscale 13], length 0
13:08:01.545394 eno1  Out IP sesstim4.timone.univ-amu.fr.45650 > 172.65.32.248.https: Flags [.], ack 1, win 502, options [nop,nop,TS val 2146305073 ecr 268246586], length 0
13:08:01.638521 eno1  Out IP sesstim4.timone.univ-amu.fr.45650 > 172.65.32.248.https: Flags [P.], seq 1:518, ack 1, win 502, options [nop,nop,TS val 2146305166 ecr 268246586], length 517
13:08:01.655277 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.45650: Flags [.], ack 518, win 7, options [nop,nop,TS val 268246696 ecr 2146305166], length 0
13:08:01.842308 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.45650: Flags [P.], seq 1:2049, ack 518, win 8, options [nop,nop,TS val 268246882 ecr 2146305166], length 2048
13:08:01.842308 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.45650: Flags [P.], seq 2049:3365, ack 518, win 8, options [nop,nop,TS val 268246882 ecr 2146305166], length 1316
13:08:01.842347 eno1  Out IP sesstim4.timone.univ-amu.fr.45650 > 172.65.32.248.https: Flags [.], ack 2049, win 494, options [nop,nop,TS val 2146305370 ecr 268246882], length 0
13:08:01.842362 eno1  Out IP sesstim4.timone.univ-amu.fr.45650 > 172.65.32.248.https: Flags [.], ack 3365, win 485, options [nop,nop,TS val 2146305370 ecr 268246882], length 0
13:08:01.844937 eno1  Out IP sesstim4.timone.univ-amu.fr.45650 > 172.65.32.248.https: Flags [P.], seq 518:598, ack 3365, win 501, options [nop,nop,TS val 2146305372 ecr 268246882], length 80
13:08:01.845279 eno1  Out IP sesstim4.timone.univ-amu.fr.45650 > 172.65.32.248.https: Flags [P.], seq 598:884, ack 3365, win 501, options [nop,nop,TS val 2146305373 ecr 268246882], length 286
13:08:01.860895 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.45650: Flags [.], ack 598, win 7, options [nop,nop,TS val 268246901 ecr 2146305372], length 0
13:08:01.861212 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.45650: Flags [.], ack 884, win 7, options [nop,nop,TS val 268246902 ecr 2146305373], length 0
13:08:02.000125 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.45650: Flags [P.], seq 3365:3444, ack 884, win 8, options [nop,nop,TS val 268247040 ecr 2146305373], length 79
13:08:02.000125 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.45650: Flags [P.], seq 3444:3523, ack 884, win 8, options [nop,nop,TS val 268247040 ecr 2146305373], length 79
13:08:02.000260 eno1  Out IP sesstim4.timone.univ-amu.fr.45650 > 172.65.32.248.https: Flags [.], ack 3523, win 501, options [nop,nop,TS val 2146305528 ecr 268247040], length 0
13:08:02.000682 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.45650: Flags [P.], seq 3523:4555, ack 884, win 8, options [nop,nop,TS val 268247041 ecr 2146305373], length 1032
13:08:02.044187 eno1  Out IP sesstim4.timone.univ-amu.fr.45650 > 172.65.32.248.https: Flags [.], ack 4555, win 501, options [nop,nop,TS val 2146305572 ecr 268247041], length 0
13:08:12.765820 eno1  Out IP sesstim4.timone.univ-amu.fr.45650 > 172.65.32.248.https: Flags [P.], seq 884:1176, ack 4555, win 501, options [nop,nop,TS val 2146316293 ecr 268247041], length 292
13:08:12.781755 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.45650: Flags [.], ack 1176, win 7, options [nop,nop,TS val 268257822 ecr 2146316293], length 0
13:08:12.921892 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.45650: Flags [P.], seq 4555:4917, ack 1176, win 8, options [nop,nop,TS val 268257962 ecr 2146316293], length 362
13:08:12.921925 eno1  Out IP sesstim4.timone.univ-amu.fr.45650 > 172.65.32.248.https: Flags [.], ack 4917, win 501, options [nop,nop,TS val 2146316449 ecr 268257962], length 0
13:08:12.928293 eno1  Out IP sesstim4.timone.univ-amu.fr.45650 > 172.65.32.248.https: Flags [P.], seq 1176:1526, ack 4917, win 501, options [nop,nop,TS val 2146316456 ecr 268257962], length 350
13:08:12.928327 eno1  Out IP sesstim4.timone.univ-amu.fr.45650 > 172.65.32.248.https: Flags [P.], seq 1526:2366, ack 4917, win 501, options [nop,nop,TS val 2146316456 ecr 268257962], length 840
13:08:12.944400 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.45650: Flags [.], ack 1526, win 7, options [nop,nop,TS val 268257985 ecr 2146316456], length 0
13:08:12.944400 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.45650: Flags [.], ack 2366, win 7, options [nop,nop,TS val 268257985 ecr 2146316456], length 0
13:08:13.142956 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.45650: Flags [P.], seq 4917:5804, ack 2366, win 8, options [nop,nop,TS val 268258183 ecr 2146316456], length 887
13:08:13.147969 eno1  Out IP sesstim4.timone.univ-amu.fr.45650 > 172.65.32.248.https: Flags [P.], seq 2366:2728, ack 5804, win 501, options [nop,nop,TS val 2146316675 ecr 268258183], length 362
13:08:13.148003 eno1  Out IP sesstim4.timone.univ-amu.fr.45650 > 172.65.32.248.https: Flags [P.], seq 2728:3445, ack 5804, win 501, options [nop,nop,TS val 2146316675 ecr 268258183], length 717
13:08:13.164079 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.45650: Flags [.], ack 2728, win 7, options [nop,nop,TS val 268258205 ecr 2146316675], length 0
13:08:13.164079 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.45650: Flags [.], ack 3445, win 7, options [nop,nop,TS val 268258205 ecr 2146316675], length 0
13:08:13.320322 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.45650: Flags [P.], seq 5804:7061, ack 3445, win 8, options [nop,nop,TS val 268258361 ecr 2146316675], length 1257
13:08:13.364119 eno1  Out IP sesstim4.timone.univ-amu.fr.45650 > 172.65.32.248.https: Flags [.], ack 7061, win 501, options [nop,nop,TS val 2146316892 ecr 268258361], length 0
13:08:16.510836 eno1  Out IP sesstim4.timone.univ-amu.fr.45650 > 172.65.32.248.https: Flags [P.], seq 3445:3814, ack 7061, win 501, options [nop,nop,TS val 2146320038 ecr 268258361], length 369
13:08:16.510872 eno1  Out IP sesstim4.timone.univ-amu.fr.45650 > 172.65.32.248.https: Flags [P.], seq 3814:4544, ack 7061, win 501, options [nop,nop,TS val 2146320038 ecr 268258361], length 730
13:08:16.526481 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.45650: Flags [.], ack 3814, win 7, options [nop,nop,TS val 268261567 ecr 2146320038], length 0
13:08:16.526481 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.45650: Flags [.], ack 4544, win 7, options [nop,nop,TS val 268261567 ecr 2146320038], length 0
13:08:16.670248 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.45650: Flags [P.], seq 7061:7858, ack 4544, win 8, options [nop,nop,TS val 268261711 ecr 2146320038], length 797
13:08:16.670279 eno1  Out IP sesstim4.timone.univ-amu.fr.45650 > 172.65.32.248.https: Flags [.], ack 7858, win 501, options [nop,nop,TS val 2146320198 ecr 268261711], length 0
13:08:17.676978 eno1  Out IP sesstim4.timone.univ-amu.fr.45650 > 172.65.32.248.https: Flags [P.], seq 4544:4906, ack 7858, win 501, options [nop,nop,TS val 2146321204 ecr 268261711], length 362
13:08:17.677011 eno1  Out IP sesstim4.timone.univ-amu.fr.45650 > 172.65.32.248.https: Flags [P.], seq 4906:5623, ack 7858, win 501, options [nop,nop,TS val 2146321204 ecr 268261711], length 717
13:08:17.692612 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.45650: Flags [.], ack 4906, win 7, options [nop,nop,TS val 268262733 ecr 2146321204], length 0
13:08:17.692612 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.45650: Flags [.], ack 5623, win 7, options [nop,nop,TS val 268262733 ecr 2146321204], length 0
13:08:17.847957 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.45650: Flags [P.], seq 7858:9115, ack 5623, win 8, options [nop,nop,TS val 268262888 ecr 2146321204], length 1257
13:08:17.847986 eno1  Out IP sesstim4.timone.univ-amu.fr.45650 > 172.65.32.248.https: Flags [.], ack 9115, win 501, options [nop,nop,TS val 2146321375 ecr 268262888], length 0
13:08:20.856561 eno1  Out IP sesstim4.timone.univ-amu.fr.45650 > 172.65.32.248.https: Flags [P.], seq 5623:5985, ack 9115, win 501, options [nop,nop,TS val 2146324384 ecr 268262888], length 362
13:08:20.856595 eno1  Out IP sesstim4.timone.univ-amu.fr.45650 > 172.65.32.248.https: Flags [P.], seq 5985:6702, ack 9115, win 501, options [nop,nop,TS val 2146324384 ecr 268262888], length 717
13:08:20.872225 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.45650: Flags [.], ack 5985, win 7, options [nop,nop,TS val 268265913 ecr 2146324384], length 0
13:08:20.872225 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.45650: Flags [.], ack 6702, win 7, options [nop,nop,TS val 268265913 ecr 2146324384], length 0
13:08:21.015383 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.45650: Flags [P.], seq 9115:10650, ack 6702, win 8, options [nop,nop,TS val 268266056 ecr 2146324384], length 1535
13:08:21.015414 eno1  Out IP sesstim4.timone.univ-amu.fr.45650 > 172.65.32.248.https: Flags [.], ack 10650, win 494, options [nop,nop,TS val 2146324543 ecr 268266056], length 0
13:08:21.257514 eno1  Out IP sesstim4.timone.univ-amu.fr.45650 > 172.65.32.248.https: Flags [F.], seq 6702, ack 10650, win 501, options [nop,nop,TS val 2146324785 ecr 268266056], length 0
13:08:21.314078 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.45650: Flags [.], ack 6703, win 8, options [nop,nop,TS val 268266355 ecr 2146324785], length 0
13:08:21.412272 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.45650: Flags [F.], seq 10650, ack 6703, win 8, options [nop,nop,TS val 268266453 ecr 2146324785], length 0
13:08:21.412311 eno1  Out IP sesstim4.timone.univ-amu.fr.45650 > 172.65.32.248.https: Flags [.], ack 10651, win 501, options [nop,nop,TS val 2146324940 ecr 268266453], length 0

By the way, my network admin has checked the palo alto side and there too, he can't see any communication on port 80 (see attached file).

We need to separate your "is this normal" into two parts:

  • Communication with the ACME server by the client (in the begin):
    • yes, it's normal that only HTTPS on port 443 is used. The ACME protocol is HTTPS only as defined in RFC 8555
  • Lack of incoming http-01 challenge requests from the ACME validation server (more in the middle or at the end of the log):
    • no, this is NOT normal, but it IS expected when a firewall is blocking incoming requests. Which is exactly what we have diagnosed earlier, so that's confirmed by your tcpdump log.
1 Like

It is not expected that acme-v02.api.letsencrypt.org communicate via HTTP.
It is not expected for acme-v02.api.letsencrypt.org to try to validate your server [directly].
All inbound HTTP requests would come from other [validation] hosts.

3 Likes

Uch, I missed the last filter on the API URI.. :frowning:

That won't work indeed.

Here's the result of the unrestricted tcpdump on acme-v02.api.letsencrypt.org
From what I understand, the validation request is coming from a server other than acme-v02.api.letsencrypt.org

jcdufour@sesstim4:~$ sudo tcpdump -i any 'tcp port 80 or tcp port 443'
[sudo] Mot de passe de jcdufour :

tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
14:25:32.771907 eno1  Out IP sesstim4.timone.univ-amu.fr.57614 > 172.65.32.248.https: Flags [S], seq 3537008159, win 64240, options [mss 1460,sackOK,TS val 2150956299 ecr 0,nop,wscale 7], length 0
14:25:32.788075 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.57614: Flags [S.], seq 1396525453, ack 3537008160, win 65160, options [mss 1400,sackOK,TS val 3029996536 ecr 2150956299,nop,wscale 13], length 0
14:25:32.788128 eno1  Out IP sesstim4.timone.univ-amu.fr.57614 > 172.65.32.248.https: Flags [.], ack 1, win 502, options [nop,nop,TS val 2150956316 ecr 3029996536], length 0
14:25:32.881343 eno1  Out IP sesstim4.timone.univ-amu.fr.57614 > 172.65.32.248.https: Flags [P.], seq 1:518, ack 1, win 502, options [nop,nop,TS val 2150956409 ecr 3029996536], length 517
14:25:32.898070 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.57614: Flags [.], ack 518, win 7, options [nop,nop,TS val 3029996646 ecr 2150956409], length 0
14:25:33.118258 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.57614: Flags [P.], seq 1:1449, ack 518, win 8, options [nop,nop,TS val 3029996866 ecr 2150956409], length 1448
14:25:33.118258 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.57614: Flags [P.], seq 1449:3365, ack 518, win 8, options [nop,nop,TS val 3029996866 ecr 2150956409], length 1916
14:25:33.118297 eno1  Out IP sesstim4.timone.univ-amu.fr.57614 > 172.65.32.248.https: Flags [.], ack 1449, win 501, options [nop,nop,TS val 2150956646 ecr 3029996866], length 0
14:25:33.118314 eno1  Out IP sesstim4.timone.univ-amu.fr.57614 > 172.65.32.248.https: Flags [.], ack 3365, win 487, options [nop,nop,TS val 2150956646 ecr 3029996866], length 0
14:25:33.120931 eno1  Out IP sesstim4.timone.univ-amu.fr.57614 > 172.65.32.248.https: Flags [P.], seq 518:598, ack 3365, win 501, options [nop,nop,TS val 2150956648 ecr 3029996866], length 80
14:25:33.121276 eno1  Out IP sesstim4.timone.univ-amu.fr.57614 > 172.65.32.248.https: Flags [P.], seq 598:884, ack 3365, win 501, options [nop,nop,TS val 2150956649 ecr 3029996866], length 286
14:25:33.136852 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.57614: Flags [.], ack 598, win 7, options [nop,nop,TS val 3029996885 ecr 2150956648], length 0
14:25:33.137185 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.57614: Flags [.], ack 884, win 7, options [nop,nop,TS val 3029996885 ecr 2150956649], length 0
14:25:33.292677 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.57614: Flags [P.], seq 3365:3444, ack 884, win 8, options [nop,nop,TS val 3029997041 ecr 2150956649], length 79
14:25:33.292678 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.57614: Flags [P.], seq 3444:3523, ack 884, win 8, options [nop,nop,TS val 3029997041 ecr 2150956649], length 79
14:25:33.292812 eno1  Out IP sesstim4.timone.univ-amu.fr.57614 > 172.65.32.248.https: Flags [.], ack 3523, win 501, options [nop,nop,TS val 2150956820 ecr 3029997041], length 0
14:25:33.293173 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.57614: Flags [P.], seq 3523:4555, ack 884, win 8, options [nop,nop,TS val 3029997041 ecr 2150956649], length 1032
14:25:33.336201 eno1  Out IP sesstim4.timone.univ-amu.fr.57614 > 172.65.32.248.https: Flags [.], ack 4555, win 501, options [nop,nop,TS val 2150956864 ecr 3029997041], length 0
14:25:36.141713 eno1  Out IP sesstim4.timone.univ-amu.fr.57614 > 172.65.32.248.https: Flags [P.], seq 884:1176, ack 4555, win 501, options [nop,nop,TS val 2150959669 ecr 3029997041], length 292
14:25:36.157658 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.57614: Flags [.], ack 1176, win 7, options [nop,nop,TS val 3029999906 ecr 2150959669], length 0
14:25:36.314086 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.57614: Flags [P.], seq 4555:4917, ack 1176, win 8, options [nop,nop,TS val 3030000062 ecr 2150959669], length 362
14:25:36.314120 eno1  Out IP sesstim4.timone.univ-amu.fr.57614 > 172.65.32.248.https: Flags [.], ack 4917, win 501, options [nop,nop,TS val 2150959842 ecr 3030000062], length 0
14:25:36.320237 eno1  Out IP sesstim4.timone.univ-amu.fr.57614 > 172.65.32.248.https: Flags [P.], seq 1176:1526, ack 4917, win 501, options [nop,nop,TS val 2150959848 ecr 3030000062], length 350
14:25:36.320272 eno1  Out IP sesstim4.timone.univ-amu.fr.57614 > 172.65.32.248.https: Flags [P.], seq 1526:2366, ack 4917, win 501, options [nop,nop,TS val 2150959848 ecr 3030000062], length 840
14:25:36.336323 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.57614: Flags [.], ack 1526, win 7, options [nop,nop,TS val 3030000085 ecr 2150959848], length 0
14:25:36.336396 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.57614: Flags [.], ack 2366, win 7, options [nop,nop,TS val 3030000085 ecr 2150959848], length 0
14:25:36.526930 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.57614: Flags [P.], seq 4917:5804, ack 2366, win 8, options [nop,nop,TS val 3030000275 ecr 2150959848], length 887
14:25:36.531696 eno1  Out IP sesstim4.timone.univ-amu.fr.57614 > 172.65.32.248.https: Flags [P.], seq 2366:2728, ack 5804, win 501, options [nop,nop,TS val 2150960059 ecr 3030000275], length 362
14:25:36.531730 eno1  Out IP sesstim4.timone.univ-amu.fr.57614 > 172.65.32.248.https: Flags [P.], seq 2728:3445, ack 5804, win 501, options [nop,nop,TS val 2150960059 ecr 3030000275], length 717
14:25:36.547796 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.57614: Flags [.], ack 2728, win 7, options [nop,nop,TS val 3030000296 ecr 2150960059], length 0
14:25:36.547796 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.57614: Flags [.], ack 3445, win 7, options [nop,nop,TS val 3030000296 ecr 2150960059], length 0
14:25:36.708122 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.57614: Flags [P.], seq 5804:7061, ack 3445, win 8, options [nop,nop,TS val 3030000456 ecr 2150960059], length 1257
14:25:36.752117 eno1  Out IP sesstim4.timone.univ-amu.fr.57614 > 172.65.32.248.https: Flags [.], ack 7061, win 501, options [nop,nop,TS val 2150960280 ecr 3030000456], length 0
14:25:39.898812 eno1  Out IP sesstim4.timone.univ-amu.fr.57614 > 172.65.32.248.https: Flags [P.], seq 3445:3814, ack 7061, win 501, options [nop,nop,TS val 2150963426 ecr 3030000456], length 369
14:25:39.898851 eno1  Out IP sesstim4.timone.univ-amu.fr.57614 > 172.65.32.248.https: Flags [P.], seq 3814:4544, ack 7061, win 501, options [nop,nop,TS val 2150963426 ecr 3030000456], length 730
14:25:39.914533 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.57614: Flags [.], ack 3814, win 7, options [nop,nop,TS val 3030003663 ecr 2150963426], length 0
14:25:39.914533 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.57614: Flags [.], ack 4544, win 7, options [nop,nop,TS val 3030003663 ecr 2150963426], length 0
14:25:40.074115 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.57614: Flags [P.], seq 7061:7858, ack 4544, win 8, options [nop,nop,TS val 3030003822 ecr 2150963426], length 797
14:25:40.074150 eno1  Out IP sesstim4.timone.univ-amu.fr.57614 > 172.65.32.248.https: Flags [.], ack 7858, win 501, options [nop,nop,TS val 2150963602 ecr 3030003822], length 0
14:25:40.295550 eno1  In  IP ec2-35-90-107-179.us-west-2.compute.amazonaws.com.29810 > sesstim4.timone.univ-amu.fr.http: Flags [S], seq 3199237270, win 62727, options [mss 1460,sackOK,TS val 2042461935 ecr 0,nop,wscale 7], length 0
14:25:40.295600 eno1  Out IP sesstim4.timone.univ-amu.fr.http > ec2-35-90-107-179.us-west-2.compute.amazonaws.com.29810: Flags [S.], seq 2803143088, ack 3199237271, win 65160, options [mss 1460,sackOK,TS val 1349586993 ecr 2042461935,nop,wscale 7], length 0
14:25:40.474661 eno1  In  IP ec2-35-90-107-179.us-west-2.compute.amazonaws.com.29810 > sesstim4.timone.univ-amu.fr.http: Flags [.], ack 1, win 491, options [nop,nop,TS val 2042462115 ecr 1349586993], length 0
14:25:40.476075 eno1  In  IP ec2-35-90-107-179.us-west-2.compute.amazonaws.com.29810 > sesstim4.timone.univ-amu.fr.http: Flags [R.], seq 1, ack 1, win 491, length 0
14:25:41.080407 eno1  Out IP sesstim4.timone.univ-amu.fr.57614 > 172.65.32.248.https: Flags [P.], seq 4544:4906, ack 7858, win 501, options [nop,nop,TS val 2150964608 ecr 3030003822], length 362
14:25:41.080441 eno1  Out IP sesstim4.timone.univ-amu.fr.57614 > 172.65.32.248.https: Flags [P.], seq 4906:5623, ack 7858, win 501, options [nop,nop,TS val 2150964608 ecr 3030003822], length 717
14:25:41.096074 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.57614: Flags [.], ack 4906, win 7, options [nop,nop,TS val 3030004844 ecr 2150964608], length 0
14:25:41.096101 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.57614: Flags [.], ack 5623, win 7, options [nop,nop,TS val 3030004844 ecr 2150964608], length 0
14:25:41.254640 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.57614: Flags [P.], seq 7858:9115, ack 5623, win 8, options [nop,nop,TS val 3030005003 ecr 2150964608], length 1257
14:25:41.254674 eno1  Out IP sesstim4.timone.univ-amu.fr.57614 > 172.65.32.248.https: Flags [.], ack 9115, win 501, options [nop,nop,TS val 2150964782 ecr 3030005003], length 0
14:25:42.474314 eno1  In  IP ec2-3-142-220-129.us-east-2.compute.amazonaws.com.10398 > sesstim4.timone.univ-amu.fr.http: Flags [S], seq 3108226572, win 62727, options [mss 1460,sackOK,TS val 3878361049 ecr 0,nop,wscale 7], length 0
14:25:42.474353 eno1  Out IP sesstim4.timone.univ-amu.fr.http > ec2-3-142-220-129.us-east-2.compute.amazonaws.com.10398: Flags [S.], seq 596934787, ack 3108226573, win 65160, options [mss 1460,sackOK,TS val 463372117 ecr 3878361049,nop,wscale 7], length 0
14:25:42.579137 eno1  In  IP ec2-3-142-220-129.us-east-2.compute.amazonaws.com.10398 > sesstim4.timone.univ-amu.fr.http: Flags [.], ack 1, win 491, options [nop,nop,TS val 3878361154 ecr 463372117], length 0
14:25:42.580689 eno1  In  IP ec2-3-142-220-129.us-east-2.compute.amazonaws.com.10398 > sesstim4.timone.univ-amu.fr.http: Flags [R.], seq 1, ack 1, win 491, length 0
14:25:44.262752 eno1  Out IP sesstim4.timone.univ-amu.fr.57614 > 172.65.32.248.https: Flags [P.], seq 5623:5985, ack 9115, win 501, options [nop,nop,TS val 2150967790 ecr 3030005003], length 362
14:25:44.262785 eno1  Out IP sesstim4.timone.univ-amu.fr.57614 > 172.65.32.248.https: Flags [P.], seq 5985:6702, ack 9115, win 501, options [nop,nop,TS val 2150967790 ecr 3030005003], length 717
14:25:44.278440 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.57614: Flags [.], ack 5985, win 7, options [nop,nop,TS val 3030008027 ecr 2150967790], length 0
14:25:44.278441 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.57614: Flags [.], ack 6702, win 7, options [nop,nop,TS val 3030008027 ecr 2150967790], length 0
14:25:44.352879 eno1  In  IP outbound1d.letsencrypt.org.53412 > sesstim4.timone.univ-amu.fr.http: Flags [S], seq 1094448860, win 64240, options [mss 1436,sackOK,TS val 2852776843 ecr 0,nop,wscale 7], length 0
14:25:44.352918 eno1  Out IP sesstim4.timone.univ-amu.fr.http > outbound1d.letsencrypt.org.53412: Flags [S.], seq 2809253639, ack 1094448861, win 65160, options [mss 1460,sackOK,TS val 1349240639 ecr 2852776843,nop,wscale 7], length 0
14:25:44.439488 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.57614: Flags [P.], seq 9115:10372, ack 6702, win 8, options [nop,nop,TS val 3030008188 ecr 2150967790], length 1257
14:25:44.439521 eno1  Out IP sesstim4.timone.univ-amu.fr.57614 > 172.65.32.248.https: Flags [.], ack 10372, win 501, options [nop,nop,TS val 2150967967 ecr 3030008188], length 0
14:25:44.512013 eno1  In  IP outbound1d.letsencrypt.org.53412 > sesstim4.timone.univ-amu.fr.http: Flags [.], ack 1, win 502, options [nop,nop,TS val 2852777003 ecr 1349240639], length 0
14:25:44.513460 eno1  In  IP outbound1d.letsencrypt.org.53412 > sesstim4.timone.univ-amu.fr.http: Flags [R.], seq 1, ack 1, win 502, length 0
14:25:47.447606 eno1  Out IP sesstim4.timone.univ-amu.fr.57614 > 172.65.32.248.https: Flags [P.], seq 6702:7064, ack 10372, win 501, options [nop,nop,TS val 2150970975 ecr 3030008188], length 362
14:25:47.447640 eno1  Out IP sesstim4.timone.univ-amu.fr.57614 > 172.65.32.248.https: Flags [P.], seq 7064:7781, ack 10372, win 501, options [nop,nop,TS val 2150970975 ecr 3030008188], length 717
14:25:47.463284 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.57614: Flags [.], ack 7064, win 7, options [nop,nop,TS val 3030011211 ecr 2150970975], length 0
14:25:47.463284 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.57614: Flags [.], ack 7781, win 7, options [nop,nop,TS val 3030011211 ecr 2150970975], length 0
14:25:47.622563 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.57614: Flags [P.], seq 10372:11907, ack 7781, win 8, options [nop,nop,TS val 3030011371 ecr 2150970975], length 1535
14:25:47.622597 eno1  Out IP sesstim4.timone.univ-amu.fr.57614 > 172.65.32.248.https: Flags [.], ack 11907, win 494, options [nop,nop,TS val 2150971150 ecr 3030011371], length 0
14:25:47.864272 eno1  Out IP sesstim4.timone.univ-amu.fr.57614 > 172.65.32.248.https: Flags [F.], seq 7781, ack 11907, win 501, options [nop,nop,TS val 2150971392 ecr 3030011371], length 0
14:25:47.920454 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.57614: Flags [.], ack 7782, win 8, options [nop,nop,TS val 3030011669 ecr 2150971392], length 0
14:25:48.036142 eno1  In  IP 172.65.32.248.https > sesstim4.timone.univ-amu.fr.57614: Flags [F.], seq 11907, ack 7782, win 8, options [nop,nop,TS val 3030011784 ecr 2150971392], length 0
14:25:48.036177 eno1  Out IP sesstim4.timone.univ-amu.fr.57614 > 172.65.32.248.https: Flags [.], ack 11908, win 501, options [nop,nop,TS val 2150971564 ecr 3030011784], length 0
^C
73 packets captured
75 packets received by filter
0 packets dropped by kernel

As you can see, I'm not an expert.
Can you tell me if there's any trace of this validation attempt? And if so, what should I tell my network team so that they can confirm or not that the problem comes from the palo alto proxy?
Thank you for all your help.

Translated with DeepL

That traceroute capture seems accurate.
I see three unique sources making HTTP [validation] requests.

If that trace is between the Palo and the Internet, then you may also need to capture between the Palo and the web server [OR at the web server].

If that trace is after the Palo [at the web server], then the Palo let it pass and the problem is within the web server itself.

2 Likes

Okay, moving on. These traces are after Palo Alto collected on my server. So Palo Alto is letting the requests through. What advice can you give me to try and find the problem, which seems to be coming from my server?

Check verbose logging of your Apache webserver. It should log the incoming requests from ec2-35-90-107-179.us-west-2.compute.amazonaws.com/ec2-3-142-220-129.us-east-2.compute.amazonaws.com/outbound1d.letsencrypt.org (note that these hosts/IP addresses can differ).

If there are no incoming requests on port 80 in your Apache logs, it's probably a firewall on your server blocking requests. E.g. iptables.

I suspect something that can block specific user agents is involved.
Not sure if iptables can do that.

Have you anything enabled that can block specific user agents?

3 Likes