Let me take a look at that log... 
1 Like
Here is the ciphers that we accept and what we deny.
ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
ssl-default-bind-options no-sslv3 no-tlsv10
tune.ssl.default-dh-param 2048
3 Likes
The fact that we could all get to your test file is a good thing. That usually means that things are happy.
You're using the webroot authenticator instead of the apache authenticator, which is fine (and sometimes preferred).
3 Likes
So looking at a dry-run it shows http here. Now the strange thing is I can go to the URL: http://acadia.k12.la.us/.well-known/acme-challenge/enB9Dfomeemc6JeRqcBw44xqb7TR6J-errXNj4Ta4iw
And it works while it is trying to do the dry-run
Domain: acadia.k12.la.us
Type: connection
Detail: Fetching
http://acadia.k12.la.us/.well-known/acme-challenge/enB9Dfomeemc6JeRqcBw44xqb7TR6J-errXNj4Ta4iw:
Timeout after connect (your server may be slow or overloaded)
3 Likes
Let's try something:
/usr/bin/certbot renew --cert-name=acadia.k12.la.us --dry-run --debug-challenges
This will cause certbot to pause once the challenge files have been created so that we can test against them ourselves. Let me know once it's running, whether the challenge files appear in /.well-known/acme-challenge/, and what their names are.
3 Likes
Running...
FrbtlZ8z82RWgEUntiENoeqeopKH5VSsRtojGawd21w
and
qfFOyOBviBahkqKrQmNrgL7-PHbLcWFPRrfzyj2tnh0
3 Likes
It did not pause and stay there let me try again.
3 Likes
bAHjNnp2AAAVIaP3IChkvv7G0y8FXjtRdMvd_lXG-Aw
tA2ZTsnVUDDZa2aIcIu0G3JySLNBVlG3fUS0fM_yBlk
3 Likes
https://acadia.k12.la.us/.well-known/acme-challenge/bAHjNnp2AAAVIaP3IChkvv7G0y8FXjtRdMvd_lXG-Aw
bAHjNnp2AAAVIaP3IChkvv7G0y8FXjtRdMvd_lXG-Aw.dxzKATtDb7LPjYpbLMOLltTCXYKQjPUnPrgnAvKGNbU
confirmed
https://acadia.k12.la.us/.well-known/acme-challenge/tA2ZTsnVUDDZa2aIcIu0G3JySLNBVlG3fUS0fM_yBlk
404
Did it unpause?
3 Likes
Okay so I copied the next set out of the folder and back in. Try these.
BmOxbbaZLxCE7dJPY50X6e_FYR86250y0Y9ApybCMEQ
IUm_7vJG2bYkGKfOiXOjC6SnRnVdHVFJ8NpsD-dIonI
That way then it can continue and they will remain now.
4 Likes
https://acadia.k12.la.us/.well-known/acme-challenge/BmOxbbaZLxCE7dJPY50X6e_FYR86250y0Y9ApybCMEQ
BmOxbbaZLxCE7dJPY50X6e_FYR86250y0Y9ApybCMEQ.dxzKATtDb7LPjYpbLMOLltTCXYKQjPUnPrgnAvKGNbU
confirmed
https://acadia.k12.la.us/.well-known/acme-challenge/IUm_7vJG2bYkGKfOiXOjC6SnRnVdHVFJ8NpsD-dIonI
IUm_7vJG2bYkGKfOiXOjC6SnRnVdHVFJ8NpsD-dIonI.dxzKATtDb7LPjYpbLMOLltTCXYKQjPUnPrgnAvKGNbU
confirmed
Do you use Fail2Ban or some other adaptive firewall scheme?
3 Likes
We have a regular firewall in place but nothing like fail2ban on this server.
I do fear something is going on though because I can see when we check for those files in the apache logs but I do not see letsencrypt checking for them. So it is like its not reaching the apache server. That is why I was hoping there was specific IP addresses. We do have a list of banned IP addresses on the firewall but I turned that rule completely off to insure that some how these IP addresses were not in that list. We mostly put denial of service attempt IP addresses in there. If someone comes in using /.well-known/acme-challenge/ then it auto reroutes to the proper apache server that serves up these requests. So in an effort to make sure nothing was getting passed by I checked the apache logs of the servers that would answer that request if it did not have /.well-known/acme-challenge in it. There are 0 of those that have been passed to the other servers.
3 Likes
Check the recent certbot logs/output if you would. I want to make sure you're not being rate-limited for failed authorizations. No point chasing our tail. What's the last error?
3 Likes
IMPORTANT NOTES:
-
The following errors were reported by the server:
Domain: acadia.k12.la.us
Type: connection
Detail: Fetching
http://acadia.k12.la.us/.well-known/acme-challenge/bAHjNnp2AAAVIaP3IChkvv7G0y8FXjtRdMvd_lXG-Aw:
Timeout after connect (your server may be slow or overloaded)
Not seeing anything like that. It would be really nice if they included the IP they were coming from in that response.
3 Likes
A reasonable suggestion to be sure.
Thank you for your wonderful cooperation with all this. I'm going to call for heavy reinforcements now. 
They're usually pretty expedient about responding. Give them some time though.
We could really use your input and guidance here.
- Manually confirmed http-01 challenge file creation and contents with 100% responsiveness over multiple acquisition attempts (i.e. load balancing not an issue)
- Attempts from Boulder do not appear in apache logs
- Testing against staging environment
- Fails due to timeout after connection
- Firewall suspected but no evidence seen
3 Likes
Thank you really. It is greatly appreciated. You have been awesome and have been extremely helpful.
3 Likes
You're quite welcome. I wish all of our visitors were as knowledgeable and responsive as you. 
3 Likes
This an error stems from the connection not responding to the validation attempt. On the Let's Encrypt side, the connection was successful and waited for a reply but never got one so it stops waiting and returns the error.
You looked at your Apache logs, but this is possibly a problem at your HA Proxy layer and if possible you should review the logs there. It sounds like your HA Proxy isn't transmitting the request to Apache so you don't see any attempts in the Apache logs.
I've sent a DM with the IP we attempted to validate for your domain that you can also review.
5 Likes
I would agree with you but we can test it every other way. If we test it with a web browser it works. If we test it on remote machines using curl it works. So I guess I am confused why it does not work with the letsencrypt servers.
4 Likes
I was able to access both challenge files myself using my Samsung phone and verify their contents. StealthMicro was also able to perform external validation. Any idea why the Let's Encrypt Boulder requests would be treated differently here?
Just a note: I did confirm that there's no IPv6 in play here.
2 Likes