Suddenly, "Some challenges have failed"

Help
1

My webserver is unable to renew or issue new certificates suddenly. I thought maybe it could be that the rate-limit exceeded, but after a week I'm still unable to issue new certificates.

My other webserver still works, but this is an different IP and hostname.

Is something broken or is it the rate-limit?

Type: unauthorized
Detail: During secondary validation: Invalid response
from
/.well-known/acme-challenge/7J6MqXefRH5RWy5c0ndGQ-AQcandz_jKXBPOWPLAQnw:
403

Letsencrypt log:

2022-10-06 11:55:09,003:DEBUG:certbot.plugins.webroot:All challenges cleaned up
2022-10-06 11:55:09,003:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
File "/bin/certbot", line 11, in
load_entry_point('certbot==0.40.0', 'console_scripts', 'certbot')()
File "/usr/lib/python3/dist-packages/certbot/main.py", line 1382, in main
return config.func(config, plugins)
File "/usr/lib/python3/dist-packages/certbot/main.py", line 1265, in certonly
lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
File "/usr/lib/python3/dist-packages/certbot/main.py", line 121, in _get_and_save_cert
lineage = le_client.obtain_and_enroll_certificate(domains, certname)
File "/usr/lib/python3/dist-packages/certbot/client.py", line 417, in obtain_and_enroll_certificate
cert, chain, key, _ = self.obtain_certificate(domains)
File "/usr/lib/python3/dist-packages/certbot/client.py", line 348, in obtain_certificate
orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
File "/usr/lib/python3/dist-packages/certbot/client.py", line 396, in _get_order_and_authorizations
authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
File "/usr/lib/python3/dist-packages/certbot/auth_handler.py", line 91, in handle_authorizations
self._poll_authorizations(authzrs, max_retries, best_effort)
File "/usr/lib/python3/dist-packages/certbot/auth_handler.py", line 180, in _poll_authorizations
raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.
2022-10-06 11:55:09,340:DEBUG:certbot.main:certbot version: 0.40.0
2022-10-06 11:55:09,341:DEBUG:certbot.main:Arguments: ['--domains', '']
2022-10-06 11:55:09,341:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2022-10-06 11:55:09,347:DEBUG:certbot.log:Root logging level set at 20
2022-10-06 11:55:09,347:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log

2

Not a rate limit. You would get a much different error message.

Without more info it is hard to give specific advice. But, there was a change recently that could affect older systems. Could this be it?

If not, please answer more of the questions on the form.

===========================

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:

I ran this command:

It produced this output:

My web server is (include version):

The operating system my web server runs on is (include version):

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don't know):

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

5 Likes
3
  • A 403 error means your webserver replied with a "access denied" error
  • "During secondary validation" means the primary validation succeeded, but one of the secondary validations did not

It's kinda weird for webservers to respond differently to different geographical locations.. Is there some kind of blocklist in play, refusing access from certain kinds of the world?

5 Likes
4

That was the first thing I thought with the 403 response error, but I didn't change things in my security features. What I did prior was installing PHP8 next to PHP7.4 and changed the CLI back to PHP7.4 after installing PHP8, but that shouldn't make a difference.

I did disabled fail2ban & mod_evasive on the webserver, but that didn't work either.
Also the Apache2 logs shows UserAgent 200 OK and there are no blockades in the logs.

5

There should be 4 log entries per attempt. 1 primary and 3 secondary. Of those 3 secondaries, 2 need to succeed and 1 may "vanish", but not return an invalid result.

5 Likes
6

Right now I'm testing with a new site, which I created 2 weeks ago (so the DNS is propagated).

I only received 1 response in my weblogs.
When I search in all the logs for the IP adres of LE, I only get the same result.

I have the requested information, but that information is a bit useless.

Apache2 Log:
:23.178.112.202 - - [07/Oct/2022:00:05:05 +0200] "GET /.well-known/acme-challenge/3QGKPqnsTWmYdhbzdjaAF0iQbW42if6cuQh9Cs3Lx1Y HTTP/1.1" 200 380 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

My domain is: cfws01.de-wolk.nl
It is a ISPConfig hosting server with multiple websites/domains and the first one in 10+ years that broke Let's Encrypt after about a year running.

I ran this command: certbot renew --dry-run
This I use for troubleshooting the renewal.

It produced this output: for all domains the same output error.
Domain: www.parkinsoncafewestbrabant.nl
Type: unauthorized
Detail: During secondary validation: 5.83.7.76: Invalid response
from
http://www.parkinsoncafewestbrabant.nl/.well-known/acme-challenge/0Z7KDB4E6u-w3rJ2GeGR3DbejQ70_cFRrNqtgJ3EuNI:
403

Domain: parkinsoncafewestbrabant.nl
Type: unauthorized
Detail: During secondary validation: 5.83.7.76: Invalid response
from
http://parkinsoncafewestbrabant.nl/.well-known/acme-challenge/LvGm1y1uveSvbanrbkim-GoClh5ME19HXYhuhJ1p4Ww:
403

To fix these errors, please make sure that your domain name was
entered correctly and the DNS A/AAAA record(s) for that domain
contain(s) the right IP address.

My web server is (include version): Apache/2.4.41

The operating system my web server runs on is (include version): Ubuntu Server 20.04 LTS x64 with kernel 5.4.0-126-generic.

My hosting provider, if applicable, is: I'm my own provider

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
ISPConfig & SSH Console

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 0.40.0

7

You have some kind of firewall responding with 403 Forbidden to many HTTP requests including the ACME Challenge requests. I get this same error just for your "home page" from my test server. I can readily reproduce this error with various test ACME challenge HTTP requests. The error page is:

<HTML><HEAD><TITLE>403 Forbidden</TITLE></HEAD>
<BODY><center>
<b>Threat Prevention</b></center>
<p>This site is blocked because it violates network policy.</p>
<p>Host: www.parkinsoncafewestbrabant.nl</p>
<p>URI: /.well-known/acme-challenge/7UNdM06ngS-FN9ZYI0RToecCy6Zuc0FNJWJtioy6eQw</p>
<p>Reason: Threat reputation No reputation</p>
<p>Please contact <a href='mailto:(redacted)'>Olaf Verheul</a></p>
</BODY></HTML>

EDIT:
There is not just one IP address for LE. There will be 4 different IP's for each HTTP challenge and even these may change from request to request.
Please see this FAQ topic and its linked page

5 Likes
8

This is correct but isn't the reason of the blockade.

You experienced the UTM which I temporary placed on a higher security level.

I changed the policy 2 days ago and the problem with LE has been going on for over 2 weeks. LE also ends up at the web server and your attempt got no further than the UTM and never reached the web server.

LE is even excluded from this security layer and is disabled with the tests.
Also, there are multiple webservers behind this UTM and all still working with LE (except the ISPConfig Hosting server).

All suspicious traffic will be blocked accept for IP addresses from the Netherlands and this layer will be placed on 'High Risks' only after I have fixed the issue.

9

There probably is something else blocking requests other than the UTM if that wasn't enabled previously.

It's fairly simple: you should see 4 requests per validation attempt. If there are not 3 to 4 requests in your log, then something is blocking those requests, especially if a 403 error is returned in the LE error message.

If it's not the UTM, it must be something else.

Alternatively you could look into the dns-01 challenge, but I highly urge to look into the http-01 challenge first: it should not be that a sysop does not know what's potentially blocking access to their webservers.

5 Likes
10

That's weird. all Linux servers behind this UTM use the same 7-layer security and this is the only server which has the problem recently.
I consulted ISPConfig support and it has since been solved by completely uninstalling and then reinstalling LE and reconfiguring the ISPConfig services using the update script.

Why 403 error message was displayed is unclear to me and it cannot be found in the weblogs.

If it's a security layer, you'd think it would still be broken after reinstalling.

Anyway I'm glad that the issue has ben resolved.

1 Like
11

How do you exclude LE from the UTM ? (by IP, user-agent, URL pattern, ...)

4 Likes
12

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.