Hi everyone, I have a strange problem with a certificate, I used Let's Encrypt with certbot hundreds of times with no issues but in this case I'm really struggling to understand why it's not working.
I'm trying to generate a new certificate for a service which is behind a quite complex architecture with an old distribution (centos 6)
The site (http://www.site.tld) is hosted on Apache httpd and is behind two reverse proxy (an F5 frontend and an IBM WebSEAL) which are totally transparent to it.
BROWSER --> F5 --> WEBSEAL --> APACHE
The Apache webserver is running on an old CentOS 6 vm, so I can't use centbot with it, I tried to solve this installing certbot on another vm running CentOS 7 which is in the same local network with the Apache webserver.
I created a directory on the CentOS 7 server for the challenge files (/tmp/certbot), exported using NFS and mounted on the CentOS 6 server where Apache is running on a .well-known directory under the website DocumentRoot.
If I put a file (file.txt) on the nfs export directory I can perfectly browse it form web using url http://www.site.tld/.well-known/file.txt , no issues with file permissions or ownership.
I tried to run certbot on the CentOS 7 vm using this syntax
certbot certonly --dry-run --webroot -d www.site.tld -w /tmp/certbot
But I constantly have challenge errors, checking on the CentOS 6 Apache access logs I perfectly find requests made by the Let's Encrypt validation servers with http response 200, this is one example
18.104.22.168 - - [18/Mar/2021:22:28:40 +0100] "GET /.well-known/acme-challenge/N7qnZXBBeORhfd-ARKxH0V7Vi3W2BdBBwmkTK1fySLo HTTP/1.1" 200 87 "http://www.site.tld/.well-known/acme-challenge/N7qnZXBBeORhfd-ARKxH0V7Vi3W2BdBBwmkTK1fySLo" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
If I add --debug-challenges to certbot and check the nfs export I perfectly find the acme-challenge directory with the challenge file inside.
I don't find anything wrong from the webserver perspective on this setup, the only thing that makes me doubt is that the public ip of the site (www.site.tld) is different from the public ip used on the network gateway for the two servers, because the site ip is assigned to the F5 reverse proxy VIP and all the internal network is behind nat using another ip.
Do you think that this IP mismatch between the certbot request source (LAN gateway NAT ip) and the site public IP (DNS resolution is fine) can cause the challenge fail?
Honestly I don't know if the challenge process also check if the challenge request came from the same IP address of the site hostname resolved by DNS.
This is the error I got from certbot
2021-03-18 22:15:28,415:DEBUG:acme.client:Storing nonce: 0003FtN-XG2MemaBMSy_uS-W9dCt0TvK5z4LD_Wm6wUI_EQ 2021-03-18 22:15:28,415:WARNING:certbot._internal.auth_handler:Challenge failed for domain www.site.tld 2021-03-18 22:15:28,415:INFO:certbot._internal.auth_handler:http-01 challenge for www.site.tld 2021-03-18 22:15:28,416:DEBUG:certbot._internal.reporter:Reporting to user: The following errors were reported by the server: Domain: www.site.tld Type: unauthorized Detail: Invalid response from http://www.site.tld/.well-known/acme-challenge/0mpKRBDaCXgzYne94TmiNMBkZeBlrkqrHIB-PW52E48 [<SITE IP>]: "<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\" \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd\"><html x" To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address. 2021-03-18 22:15:28,416:DEBUG:certbot._internal.error_handler:Encountered exception: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/certbot/_internal/auth_handler.py", line 91, in handle_authorizations self._poll_authorizations(authzrs, max_retries, best_effort) File "/usr/lib/python2.7/site-packages/certbot/_internal/auth_handler.py", line 180, in _poll_authorizations raise errors.AuthorizationError('Some challenges have failed.') AuthorizationError: Some challenges have failed. 2021-03-18 22:15:28,416:DEBUG:certbot._internal.error_handler:Calling registered functions 2021-03-18 22:15:28,416:INFO:certbot._internal.auth_handler:Cleaning up challenges 2021-03-18 22:15:28,416:DEBUG:certbot._internal.plugins.webroot:Removing /tmp/certbot/.well-known/acme-challenge/0mpKRBDaCXgzYne94TmiNMBkZeBlrkqrHIB-PW52E48 2021-03-18 22:15:28,417:DEBUG:certbot._internal.plugins.webroot:All challenges cleaned up 2021-03-18 22:15:28,417:DEBUG:certbot._internal.log:Exiting abnormally: Traceback (most recent call last): File "/usr/bin/certbot", line 9, in <module> load_entry_point('certbot==1.11.0', 'console_scripts', 'certbot')() File "/usr/lib/python2.7/site-packages/certbot/main.py", line 15, in main return internal_main.main(cli_args) File "/usr/lib/python2.7/site-packages/certbot/_internal/main.py", line 1421, in main return config.func(config, plugins) File "/usr/lib/python2.7/site-packages/certbot/_internal/main.py", line 1294, in certonly lineage = _get_and_save_cert(le_client, config, domains, certname, lineage) File "/usr/lib/python2.7/site-packages/certbot/_internal/main.py", line 135, in _get_and_save_cert lineage = le_client.obtain_and_enroll_certificate(domains, certname) File "/usr/lib/python2.7/site-packages/certbot/_internal/client.py", line 441, in obtain_and_enroll_certificate cert, chain, key, _ = self.obtain_certificate(domains) File "/usr/lib/python2.7/site-packages/certbot/_internal/client.py", line 374, in obtain_certificate orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names) File "/usr/lib/python2.7/site-packages/certbot/_internal/client.py", line 421, in _get_order_and_authorizations authzr = self.auth_handler.handle_authorizations(orderr, best_effort) File "/usr/lib/python2.7/site-packages/certbot/_internal/auth_handler.py", line 91, in handle_authorizations self._poll_authorizations(authzrs, max_retries, best_effort) File "/usr/lib/python2.7/site-packages/certbot/_internal/auth_handler.py", line 180, in _poll_authorizations raise errors.AuthorizationError('Some challenges have failed.') AuthorizationError: Some challenges have failed. 2021-03-18 22:15:28,418:ERROR:certbot._internal.log:Some challenges have failed.