Hi everyone, I have a strange problem with a certificate, I used Let's Encrypt with certbot hundreds of times with no issues but in this case I'm really struggling to understand why it's not working.
I'm trying to generate a new certificate for a service which is behind a quite complex architecture with an old distribution (centos 6)
The site (http://www.site.tld) is hosted on Apache httpd and is behind two reverse proxy (an F5 frontend and an IBM WebSEAL) which are totally transparent to it.
BROWSER --> F5 --> WEBSEAL --> APACHE
The Apache webserver is running on an old CentOS 6 vm, so I can't use centbot with it, I tried to solve this installing certbot on another vm running CentOS 7 which is in the same local network with the Apache webserver.
I created a directory on the CentOS 7 server for the challenge files (/tmp/certbot), exported using NFS and mounted on the CentOS 6 server where Apache is running on a .well-known directory under the website DocumentRoot.
If I put a file (file.txt) on the nfs export directory I can perfectly browse it form web using url http://www.site.tld/.well-known/file.txt , no issues with file permissions or ownership.
I tried to run certbot on the CentOS 7 vm using this syntax
certbot certonly --dry-run --webroot -d www.site.tld -w /tmp/certbot
But I constantly have challenge errors, checking on the CentOS 6 Apache access logs I perfectly find requests made by the Let's Encrypt validation servers with http response 200, this is one example
34.209.232.166 - - [18/Mar/2021:22:28:40 +0100] "GET /.well-known/acme-challenge/N7qnZXBBeORhfd-ARKxH0V7Vi3W2BdBBwmkTK1fySLo HTTP/1.1" 200 87 "http://www.site.tld/.well-known/acme-challenge/N7qnZXBBeORhfd-ARKxH0V7Vi3W2BdBBwmkTK1fySLo" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
If I add --debug-challenges to certbot and check the nfs export I perfectly find the acme-challenge directory with the challenge file inside.
I don't find anything wrong from the webserver perspective on this setup, the only thing that makes me doubt is that the public ip of the site (www.site.tld) is different from the public ip used on the network gateway for the two servers, because the site ip is assigned to the F5 reverse proxy VIP and all the internal network is behind nat using another ip.
Do you think that this IP mismatch between the certbot request source (LAN gateway NAT ip) and the site public IP (DNS resolution is fine) can cause the challenge fail?
Honestly I don't know if the challenge process also check if the challenge request came from the same IP address of the site hostname resolved by DNS.
This is the error I got from certbot
2021-03-18 22:15:28,415:DEBUG:acme.client:Storing nonce: 0003FtN-XG2MemaBMSy_uS-W9dCt0TvK5z4LD_Wm6wUI_EQ
2021-03-18 22:15:28,415:WARNING:certbot._internal.auth_handler:Challenge failed for domain www.site.tld
2021-03-18 22:15:28,415:INFO:certbot._internal.auth_handler:http-01 challenge for www.site.tld
2021-03-18 22:15:28,416:DEBUG:certbot._internal.reporter:Reporting to user: The following errors were reported by the server:
Domain: www.site.tld
Type: unauthorized
Detail: Invalid response from http://www.site.tld/.well-known/acme-challenge/0mpKRBDaCXgzYne94TmiNMBkZeBlrkqrHIB-PW52E48 [<SITE IP>]: "<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\" \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd\"><html x"
To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address.
2021-03-18 22:15:28,416:DEBUG:certbot._internal.error_handler:Encountered exception:
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/certbot/_internal/auth_handler.py", line 91, in handle_authorizations
self._poll_authorizations(authzrs, max_retries, best_effort)
File "/usr/lib/python2.7/site-packages/certbot/_internal/auth_handler.py", line 180, in _poll_authorizations
raise errors.AuthorizationError('Some challenges have failed.')
AuthorizationError: Some challenges have failed.
2021-03-18 22:15:28,416:DEBUG:certbot._internal.error_handler:Calling registered functions
2021-03-18 22:15:28,416:INFO:certbot._internal.auth_handler:Cleaning up challenges
2021-03-18 22:15:28,416:DEBUG:certbot._internal.plugins.webroot:Removing /tmp/certbot/.well-known/acme-challenge/0mpKRBDaCXgzYne94TmiNMBkZeBlrkqrHIB-PW52E48
2021-03-18 22:15:28,417:DEBUG:certbot._internal.plugins.webroot:All challenges cleaned up
2021-03-18 22:15:28,417:DEBUG:certbot._internal.log:Exiting abnormally:
Traceback (most recent call last):
File "/usr/bin/certbot", line 9, in <module>
load_entry_point('certbot==1.11.0', 'console_scripts', 'certbot')()
File "/usr/lib/python2.7/site-packages/certbot/main.py", line 15, in main
return internal_main.main(cli_args)
File "/usr/lib/python2.7/site-packages/certbot/_internal/main.py", line 1421, in main
return config.func(config, plugins)
File "/usr/lib/python2.7/site-packages/certbot/_internal/main.py", line 1294, in certonly
lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
File "/usr/lib/python2.7/site-packages/certbot/_internal/main.py", line 135, in _get_and_save_cert
lineage = le_client.obtain_and_enroll_certificate(domains, certname)
File "/usr/lib/python2.7/site-packages/certbot/_internal/client.py", line 441, in obtain_and_enroll_certificate
cert, chain, key, _ = self.obtain_certificate(domains)
File "/usr/lib/python2.7/site-packages/certbot/_internal/client.py", line 374, in obtain_certificate
orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
File "/usr/lib/python2.7/site-packages/certbot/_internal/client.py", line 421, in _get_order_and_authorizations
authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
File "/usr/lib/python2.7/site-packages/certbot/_internal/auth_handler.py", line 91, in handle_authorizations
self._poll_authorizations(authzrs, max_retries, best_effort)
File "/usr/lib/python2.7/site-packages/certbot/_internal/auth_handler.py", line 180, in _poll_authorizations
raise errors.AuthorizationError('Some challenges have failed.')
AuthorizationError: Some challenges have failed.
2021-03-18 22:15:28,418:ERROR:certbot._internal.log:Some challenges have failed.