Certbot renewal fails for unknow reason

Help
1

Detail below:

certbot --renew is failing to renew, and this has worked in the past. We know we must be missing something, but can't see it.

My domain is:

cloud.waltoninstitute.ie

I ran this command:

certbot renew

We have been using a configuration file

Options used in the renewal process

[renewalparams]
account = XXXXXXXXXXXXXXXX
authenticator = standalone
server = https://acme-v02.api.letsencrypt.org/directory
http01_port = 8888

--

If we start up a mini webserver on port 8888 we can hit it from outside.

There is a Haproxy on the external IP, with tcptump we have verified that port 8888 is accessible from outside and can see the requests coming in as far as certbot by running tcpdump on the same host as certbot is running.

Here is the message from the logfile.
It produced this output:

Domain: YYYY.YYYY.YYYY
Type: connection
Detail: During secondary validation: XX.XX.XX.XX: Fetching http://YYYY.YYYY.YYYY/.well-known/acme-challenge/_VIb587g1Og3Ch9kgx0K_mevt62ys4wgZj1_RVcqIGc: Error getting validation data

To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address. Additionally, please check that your computer has a publicly routable IP address and that no firewalls are preventing the server from communicating with the client. If you're using the webroot plugin, you should also verify that you are serving files from the webroot path you provided.
2026-10-15 12:58:05,569:DEBUG:certbot.error_handler:Encountered exception:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/certbot/auth_handler.py", line 91, in handle_authorizations
self._poll_authorizations(authzrs, max_retries, best_effort)
File "/usr/lib/python3/dist-packages/certbot/auth_handler.py", line 180, in _poll_authorizations
raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.

2026-10-15 12:58:05,569:DEBUG:certbot.error_handler:Calling registered functions
2026-10-15 12:58:05,570:INFO:certbot.auth_handler:Cleaning up challenges
2026-10-15 12:58:05,570:DEBUG:certbot.plugins.standalone:Stopping server at :::8888...
2026-10-15 12:58:05,632:WARNING:certbot.renewal:Attempting to renew cert (YYYY.YYYY.YYYY) from /etc/letsencrypt/renewal/YYYY.YYYY.YYYY.ie.conf produced an unexpected error: Some challenges have failed.. Skipping.
2026-10-15 12:58:05,635:DEBUG:certbot.renewal:Traceback was:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/certbot/renewal.py", line 462, in handle_renewal_request
main.renew_cert(lineage_config, plugins, renewal_candidate)
File "/usr/lib/python3/dist-packages/certbot/main.py", line 1208, in renew_cert
renewed_lineage = _get_and_save_cert(le_client, config, lineage=lineage)
File "/usr/lib/python3/dist-packages/certbot/main.py", line 116, in _get_and_save_cert
renewal.renew_cert(config, domains, le_client, lineage)
File "/usr/lib/python3/dist-packages/certbot/renewal.py", line 320, in renew_cert
new_cert, new_chain, new_key, _ = le_client.obtain_certificate(domains, new_key)
File "/usr/lib/python3/dist-packages/certbot/client.py", line 348, in obtain_certificate
orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
File "/usr/lib/python3/dist-packages/certbot/client.py", line 396, in _get_order_and_authorizations
authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
File "/usr/lib/python3/dist-packages/certbot/auth_handler.py", line 91, in handle_authorizations
self._poll_authorizations(authzrs, max_retries, best_effort)
File "/usr/lib/python3/dist-packages/certbot/auth_handler.py", line 180, in _poll_authorizations
raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.

My web server is (include version):
apache2

The operating system my web server runs on is (include version):
ubuntu 20.04

My hosting provider, if applicable, is:
NA

I can login to a root shell on my machine (yes or no, or I don't know):
yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): NO

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

2

http-01 needs to use port 80, this might mean that your haproxy instance runs on port 80 and forwards to port 8888, however you should test your setup by connecting to the external IP address on port 80.

3

So the requests from the outside are on port 80, but if something is forwarding them to that host on port 8888 then sure that should work. How many incoming requests do you see? Since there should be at least 5.

You can run certbot with --debug-challenges to keep the standalone server running, and try from many external locations to confirm connectivity. A failure "During secondary validation" usually means that it's a problem with connectivity from some parts of the Internet, even if there is connectivity from the "primary" validation location.