Trouble renewing certificate - certbot 0.40.0

[StealthMicro]

Here is a run that shows what is happening. Below is the output from letsdebug and feel free to run your own as it is up full time.

/usr/bin/certbot renew --cert-name acadia.k12.la.us --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log

---

Processing /etc/letsencrypt/renewal/acadia.k12.la.us.conf

---

Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for acadia.k12.la.us
http-01 challenge for www.acadia.k12.la.us
Waiting for verification...
Challenge failed for domain acadia.k12.la.us
http-01 challenge for acadia.k12.la.us
Cleaning up challenges
Attempting to renew cert (acadia.k12.la.us) from /etc/letsencrypt/renewal/acadia.k12.la.us.conf produced an unexpected error: Some challenges have failed.. Skipping.
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/acadia.k12.la.us/fullchain.pem (failure)

---

** DRY RUN: simulating 'certbot renew' close to cert expiry
** (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/acadia.k12.la.us/fullchain.pem (failure)
** DRY RUN: simulating 'certbot renew' close to cert expiry
** (The test certificates above have not been saved.)

---

1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:

• The following errors were reported by the server:

  Domain: acadia.k12.la.us
  Type: connection
  Detail: Fetching
  http://acadia.k12.la.us/.well-known/acme-challenge/bSeDla6lzSTM56Nn9vSBwqOqTscthT0ytvzb4mnMvFY:
  Timeout after connect (your server may be slow or overloaded)

  To fix these errors, please make sure that your domain name was
  entered correctly and the DNS A/AAAA record(s) for that domain
  contain(s) the right IP address. Additionally, please check that
  your computer has a publicly routable IP address and that no
  firewalls are preventing the server from communicating with the
  client. If you're using the webroot plugin, you should also verify
  that you are serving files from the webroot path you provided.

![letsencrypt2|690x55]

If I run it while the process is happening I get the following returned in the web browser. I am only allowed to attach 1 media as I am a new user.

URL:
https://acadia.k12.la.us/.well-known/acme-challenge/Sxu_YMriaKuGjML2xV6pJ0iwPbjlpJAjWNoBGsNkUoA

Result in browser: Sxu_YMriaKuGjML2xV6pJ0iwPbjlpJAjWNoBGsNkUoA.dxzKATtDb7LPjYpbLMOLltTCXYKQjPUnPrgnAvKGNbU

[JuergenAuer]

Hi @StealthMicro

there are some curious things.

Why

@StealthMicro

Domain: acadia.k12.la.us
Type: connection
Detail: Fetching
http://acadia.k12.la.us/.well-known/acme-challenge/bSeDla6lzSTM56Nn9vSBwqOqTscthT0ytvzb4mnMvFY:
Timeout after connect (your server may be slow or overloaded)

is there a timeout? Looks like your server doesn't answer. Or is this a blocking firewall, a blocking .htaccess or a bot detection?

Checking your domain via https://check-your-website.server-daten.de/?q=acadia.k12.la.us - your name server has the same ip address. Normally, that shouldn't be a problem - but dns over tcp doesn't work. Isn't it configured or is this a blocking firewall?

Checking /.well-known/acme-challenge/random-filename, there is the expected http result 404 - Not Found.

But why shows letsdebug a http status 200? ( /.well-known/acme-challenge/letsdebug-test ). I don't think there is a file with that file name.

Your test url has the correct content - filename, dot, a hash of the account key.

PS: Ok, there - https://acadia.k12.la.us/.well-known/acme-challenge/letsdebug-test - is a file with that file name, so the http result 200 is expected.

[griffin]

I get redirected from http to https, so both work for me. The redirect is a 302 though instead of a 301 like it should be.

[griffin]

There's a load balancer in play...

load-balancer: ies-lb-v2-ssl

[rg305]

Ok it helps to RTFM (once in a while):

curl -L http://acadia.k12.la.us/.well-known/acme-challenge/letsdebug-test
letsdebug-test

[-L will follow redirection]

[griffin]

>>> http://acadia.k12.la.us/.well-known/acme-challenge/letsdebug-test

> --------------------------------------------
> 302 Found
> --------------------------------------------

Status: 302 Found
Code: 302
content-length: 0
Location: https://acadia.k12.la.us/.well-known/acme-challenge/letsdebug-test
cache-control: no-cache
connection: close

>>> https://acadia.k12.la.us/.well-known/acme-challenge/letsdebug-test

> --------------------------------------------
> 200 OK
> --------------------------------------------

Status: 200 OK
Code: 200
date: Wed, 25 Nov 2020 04:00:47 GMT
server: Apache
last-modified: Tue, 24 Nov 2020 20:49:40 GMT
etag: "f-5b4e0731b1f56"
accept-ranges: bytes
content-length: 15
load-balancer: ies-lb-v2-ssl
connection: close

[griffin]

[rg305]

Seeing this always makes my spidey senses tingle...

@griffin

server: Apache

@griffin

load-balancer: ies-lb-v2-ssl

But they seem to have got that working (as expected).

I think the main problem is that the HTTPS site is locked down tight (as it should be).
And may be blocking some of the challenge requests [that come from overseas].
The "fix" is not to allow them (in HTTPS); But to accept them in HTTP.
Don't redirect the challenge requests to HTTPS.

Here is a "sample" Apache code to get you started:
[which redirects only when the URL does not contain ".well-known"]

  <LocationMatch "^/(?!\.well-known)">
    #send all other requests to HTTPS
    RewriteEngine On
    RewriteCond %{HTTPS} !=on
    RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1
  </LocationMatch>

Any vhost using this code would need to specify the document root for when the URL does contain ".well-known".

Which has to jive with your:

@StealthMicro

Authenticator webroot

[StealthMicro]

@rg305

curl -L http://acadia.k12.la.us/.well-known/acme-challenge/letsdebug-test
letsdebug-test

I created a letsdebug-test file with letsdebug-test as the contents so I could see something other then a 404 file not found error.

[StealthMicro]

@griffin

There's a load balancer in play...

load-balancer: ies-lb-v2-ssl

Correct there is a haproxy load balancer in the middle.

[StealthMicro]

@rg305

But they seem to have got that working (as expected).

I think the main problem is that the HTTPS site is locked down tight (as it should be).
And may be blocking some of the challenge requests [that come from overseas].
The "fix" is not to allow them (in HTTPS); But to accept them in HTTP.
Don't redirect the challenge requests to HTTPS.

Here is a "sample" Apache code to get you started:
[which redirects only when the URL does not contain ".well-known"]

  <LocationMatch "^/(?!\.well-known)">
    #send all other requests to HTTPS
    RewriteEngine On
    RewriteCond %{HTTPS} !=on
    RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1
  </LocationMatch>

Any vhost using this code would need to specify the document root for when the URL does contain ".well-known".

Which has to jive with your:

@rg305

But they seem to have got that working (as expected).

I think the main problem is that the HTTPS site is locked down tight (as it should be).
And may be blocking some of the challenge requests [that come from overseas].
The "fix" is not to allow them (in HTTPS); But to accept them in HTTP.
Don't redirect the challenge requests to HTTPS.

Here is a "sample" Apache code to get you started:
[which redirects only when the URL does not contain ".well-known"]

  <LocationMatch "^/(?!\.well-known)">
    #send all other requests to HTTPS
    RewriteEngine On
    RewriteCond %{HTTPS} !=on
    RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1
  </LocationMatch>

Any vhost using this code would need to specify the document root for when the URL does contain ".well-known".

I set it to use http instead of https and still having the same exact issue. Feel free to test it that way as well. The interesting thing is it looks fine using a web browser and or curl but for some reason letsdebug fails (as well as certbot)

As an FYI we have been using this configuration for over a year and have over 500 SSL's with Lets Encrypt. This only started happening recently. It is possible that a update to one of the various services involved has caused this. haproxy or apache even maybe.

I did remove the letsdebug-test file so now it shows the proper 404 error. But certbot still fails.

[griffin]

I'm seeing that InternalProblem too with Let's Debug, yet I can visit the site just fine. Did you remove the test file? I don't see it anymore.

[StealthMicro]

Yes I did remove the test file so the normal 404 shows up now. certbot still fails. The strange thing is I do not see any attempts from letsencrypt in the apache log files using the certbot app.

[griffin]

Is your load balancer terminating SSL or are workers?

[StealthMicro]

Load balancer is. But the strange thing all of this worked fine until recently. It should also be coming over http unless the certbot or letsencrypt forces https on its own?

Any idea what IP addresses the certbot would have letsencrypt come back as? Is it the same as letsdebug? If not I could look for that or those IP addresses and try to track those down in the firewall and try to follow the packets.

[StealthMicro]

Here is a pastebin of the debug log. It is a lot of data so I am doing a pastebin so not to clutter up in here.

/usr/bin/certbot renew --cert-name=acadia.k12.la.usSaving debug log to /var/lo...

[griffin]

It looks like the site has a lot of "mixed content" (being served over htttp), which will cause the site to be shown as insecure (and no padlock to appear next to the address). There are also crawling errors.

Missing Padlock

[StealthMicro]

That normally would not happen. So what happens

@griffin

It looks like the site has a lot of "mixed content" (being served over htttp), which will cause the site to be shown as insecure (and no padlock to appear next to the address). There are also crawling errors.

So what happens on my side. Someone comes in under http and a file is checked to see if they have an SSL and if so it forces all traffic to https. This normally is the way it works but for testing here I simply removed that domain from the file so we could test letsencrypt to eliminate a https challenge issue. Once testing is done then I will simply add that domain back in and all those mixed content issues will be resolved.

[griffin]

@StealthMicro

Load balancer is. But the strange thing all of this worked fine until recently. It should also be coming over http unless the certbot or letsencrypt forces https on its own?

Depends upon your certbot renewal configuration. I haven't looked over the log yet. If you used an "installer" and specified to have certbot redirect to https then certbot maintains configuration changes to do that.

@StealthMicro

Any idea what IP addresses the certbot would have letsencrypt come back as? Is it the same as letsdebug? If not I could look for that or those IP addresses and try to track those down in the firewall and try to follow the packets.

I believe that Let's Encrypt uses AWS instances and that the IP addresses vary. There have been several (heated) topics about this practice about various points in this community. The general rule is to always accept requests over port 80 for /.well-known/ . It is fine to redirect to https and have the exception there. Let's Encrypt also ignores invalid certificates along the way.

[StealthMicro]

@griffin

Depends upon your certbot renewal configuration. I haven't looked over the log yet. If you used an "installer" and specified to have certbot redirect to https then certbot maintains configuration changes to do that.

I use certonly and do not have apache do the redirection (via the certbot "installer"

@griffin

I believe that Let's Encrypt uses AWS instances and that the IP addresses vary. There have been several (heated) topics about this practice about various points in this community. The general rule is to always accept requests over port 80 for /.well-known/ .

I actually redirect to the letsencrypt apache server via /.well-known/acme-challenge