Challenge failed with complex setup

Hi everyone, I have a strange problem with a certificate, I used Let's Encrypt with certbot hundreds of times with no issues but in this case I'm really struggling to understand why it's not working.
I'm trying to generate a new certificate for a service which is behind a quite complex architecture with an old distribution (centos 6)

The site ( is hosted on Apache httpd and is behind two reverse proxy (an F5 frontend and an IBM WebSEAL) which are totally transparent to it.


The Apache webserver is running on an old CentOS 6 vm, so I can't use centbot with it, I tried to solve this installing certbot on another vm running CentOS 7 which is in the same local network with the Apache webserver.

I created a directory on the CentOS 7 server for the challenge files (/tmp/certbot), exported using NFS and mounted on the CentOS 6 server where Apache is running on a .well-known directory under the website DocumentRoot.

If I put a file (file.txt) on the nfs export directory I can perfectly browse it form web using url , no issues with file permissions or ownership.
I tried to run certbot on the CentOS 7 vm using this syntax

certbot certonly --dry-run --webroot -d -w /tmp/certbot

But I constantly have challenge errors, checking on the CentOS 6 Apache access logs I perfectly find requests made by the Let's Encrypt validation servers with http response 200, this is one example - - [18/Mar/2021:22:28:40 +0100] "GET /.well-known/acme-challenge/N7qnZXBBeORhfd-ARKxH0V7Vi3W2BdBBwmkTK1fySLo HTTP/1.1" 200 87 "" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +"

If I add --debug-challenges to certbot and check the nfs export I perfectly find the acme-challenge directory with the challenge file inside.

I don't find anything wrong from the webserver perspective on this setup, the only thing that makes me doubt is that the public ip of the site ( is different from the public ip used on the network gateway for the two servers, because the site ip is assigned to the F5 reverse proxy VIP and all the internal network is behind nat using another ip.

Do you think that this IP mismatch between the certbot request source (LAN gateway NAT ip) and the site public IP (DNS resolution is fine) can cause the challenge fail?
Honestly I don't know if the challenge process also check if the challenge request came from the same IP address of the site hostname resolved by DNS.

This is the error I got from certbot

2021-03-18 22:15:28,415:DEBUG:acme.client:Storing nonce: 0003FtN-XG2MemaBMSy_uS-W9dCt0TvK5z4LD_Wm6wUI_EQ
2021-03-18 22:15:28,415:WARNING:certbot._internal.auth_handler:Challenge failed for domain
2021-03-18 22:15:28,415:INFO:certbot._internal.auth_handler:http-01 challenge for
2021-03-18 22:15:28,416:DEBUG:certbot._internal.reporter:Reporting to user: The following errors were reported by the server:

Type:   unauthorized
Detail: Invalid response from [<SITE IP>]: "<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\" \"\"><html x"

To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address.
2021-03-18 22:15:28,416:DEBUG:certbot._internal.error_handler:Encountered exception:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/certbot/_internal/", line 91, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, best_effort)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/", line 180, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
AuthorizationError: Some challenges have failed.

2021-03-18 22:15:28,416:DEBUG:certbot._internal.error_handler:Calling registered functions
2021-03-18 22:15:28,416:INFO:certbot._internal.auth_handler:Cleaning up challenges
2021-03-18 22:15:28,416:DEBUG:certbot._internal.plugins.webroot:Removing /tmp/certbot/.well-known/acme-challenge/0mpKRBDaCXgzYne94TmiNMBkZeBlrkqrHIB-PW52E48
2021-03-18 22:15:28,417:DEBUG:certbot._internal.plugins.webroot:All challenges cleaned up
2021-03-18 22:15:28,417:DEBUG:certbot._internal.log:Exiting abnormally:
Traceback (most recent call last):
  File "/usr/bin/certbot", line 9, in <module>
    load_entry_point('certbot==1.11.0', 'console_scripts', 'certbot')()
  File "/usr/lib/python2.7/site-packages/certbot/", line 15, in main
    return internal_main.main(cli_args)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/", line 1421, in main
    return config.func(config, plugins)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/", line 1294, in certonly
    lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/", line 135, in _get_and_save_cert
    lineage = le_client.obtain_and_enroll_certificate(domains, certname)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/", line 441, in obtain_and_enroll_certificate
    cert, chain, key, _ = self.obtain_certificate(domains)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/", line 374, in obtain_certificate
    orderr = self._get_order_and_authorizations(, self.config.allow_subset_of_names)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/", line 421, in _get_order_and_authorizations
    authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/", line 91, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, best_effort)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/", line 180, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
AuthorizationError: Some challenges have failed.
2021-03-18 22:15:28,418:ERROR:certbot._internal.log:Some challenges have failed.
1 Like

Hi @bill.guarnere,

It would be more helpful if you could share your domain name, or else at least check it with some test tools like

(both of these made by community members here!)

Without your domain name, we can't check if you have a common problem like IPv6 and IPv4 DNS entries that don't point to the same device, or something.

From what you've described, I would suggest that the problem could be that one of the reverse proxy devices is blocking inbound connections from some IP addresses but not from others. Let's Encrypt has used multiperspective validation for a couple of years, so there will be multiple incoming connections from different source IP addresses (which are not published in advance and can change over time). These may be in different parts of the world, so if you have a blacklist on a particular device, it could prevent some of the connections while allowing others.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.