Challenge times out on standalone mode reverse-proxied to different port

Help
#1

My domain is: darkvirtue.com

I ran this command:

certbot renew --renew-hook 'cat $RENEWED_LINEAGE/privkey.pem $RENEWED_LINEAGE/fullchain.pem > $RENEWED_LINEAGE/longpem.pem' --post-hook '/bin/systemctl reload haproxy dovecot postfix'

My /etc/letsencrypt/cli.ini is

email = admin@darkvirtue.com
authenticator = standalone
http-01-address = 127.0.0.1
http-01-port = 10443
rsa-key-size = 4096
non-interactive = True
text = True
keep-until-expiring = True
expand = True
agree-tos = True
max-log-backups = 10

It produced this output:

<snip>

Certbot failed to authenticate some domains (authenticator: standalone). The Certificate Authority reported these problems:
  Domain: books.darkvirtue.com
  Type:   connection
  Detail: 156.96.60.101: Fetching http://books.darkvirtue.com/.well-known/acme-challenge/UGafM1fxlZybxH99tLqepPTNOtZoYmdAWnrJL7cgG4M: Timeout during connect (likely firewall problem)

<snip>
Hint: The Certificate Authority failed to download the challenge files from the temporary standalone webserver started by Certbot on 127.0.0.1:10443. Ensure that the listed domains point to this machine and that it can accept inbound connections from the internet.

My reverse proxy is: HAProxy 2.2.9

Relevant parts of haproxy.cfg:

frontend http-virtual
        bind :::80
        bind *:80
        bind :::443 ssl crt /etc/haproxy/certs/ alpn h2,http/1.1
        bind *:443 ssl crt /etc/haproxy/certs/ alpn h2,http/1.1

        acl letsencrypt path_beg -i /.well-known/acme-challenge

        redirect scheme https code 301 if !{ ssl_fc }

        use_backend letsencrypt if letsencrypt

backend letsencrypt
        server certbot 127.0.0.1:10443

The operating system my web server runs on is (include version): Debian 11.5

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): 2.0.0

HAProxy appears to be correctly routing all request for /.well-known/acme-challenge to 127.0.0.1:10443. Likewise, Certbot is correctly listening on 127.0.0.1:10443. I verified by trying to request the HEAD while Certbot is running from an external server:

curl -I https://books.darkvirtue.com/.well-known/acme-challenge/

HTTP/2 501
date: Tue, 06 Dec 2022 18:42:47 GMT
content-type: text/html;charset=utf-8
content-length: 497
strict-transport-security: max-age=31536000; preload; includeSubDomains
x-frame-options: SAMEORIGIN
x-xss-protection: 1;mode=block
referrer-policy: no-referrer,same-origin,strict-origin,strict-origin-when-cross-origin
x-content-type-options: nosniff

While Certbot isn't running, I get the expected 503 error.

I've had this configuration in place for several years and it's been fine until today. Could anyone provide some insight as to why the Acme challenges are timing out? The firewall is clearly not the issue.

#2

That doesn't seem very positive.

2 Likes
#3

501 is the "Not implemented" error code. Maybe certbot's standalone webserver doesn't support HEAD requests? Or is something broken with my Certbot install?

I use pip3 install -U certbot to install and update Certbot. Python version is 3.9.2.

#4

Double-checking the docs, I've removed and reinstalled Certbot using in a venv. It appears I've hit the rate limit for failed validations, so I'll have to wait an hour to try again.

#5

Please use the staging environment for testing. Thank you.

3 Likes
#6

Thanks for the reminder about staging.

Using the fresh venv install of Certbot, I'm still getting the above connection timeout errors even though ports 80 and 443 are open and accessible, and traffic is being properly routed to certbot.

#7

image
image1203×536 17.4 KB

2 Likes
#8

Thanks for the link, Bruce5051, I'll add that to my bag of tricks.

Since the Let's Debug test also claims a timeout, now I wonder if that error is misreported. It should be a 503 error since the Let's Encrypt standalone server isn't currently running on the backend.

But I can't reproduce the issue outside of the Let's Encrypt context. http://books.darkvirtue.com properly redirects to https, and the https address works fine. I've had this reverse-proxy config in place for years and the last change was 6 months ago - I've successfully renewed other certificates on the server in that time. The SSL config was changed 6 months ago to update to the then-current Mozilla recommended settings.

I feel very confident this is not a firewall issue.

1 Like
#9

Here is what I presently see:

$ curl -I http://books.darkvirtue.com
HTTP/1.1 301 Moved Permanently
content-length: 0
location: https://books.darkvirtue.com/

$ curl -I http://books.darkvirtue.com/.well-known/acme-challenge/
HTTP/1.1 301 Moved Permanently
content-length: 0
location: https://books.darkvirtue.com/.well-known/acme-challenge/

$ curl -I http://books.darkvirtue.com/.well-known/acme-challenge/some_test_file
HTTP/1.1 301 Moved Permanently
content-length: 0
location: https://books.darkvirtue.com/.well-known/acme-challenge/some_test_file
2 Likes
#10

Here are a couple of odd things
https://www.ssllabs.com/ssltest/analyze.html?d=books.darkvirtue.com

image
image1680×976 37.7 KB

And

image
image1680×976 54.4 KB

1 Like
#11

Yes, that's correct. My HAProxy config has a rule for globally redirecting HTTP to HTTPS.

curl -I https://books.darkvirtue.com/.well-known/acme-challenge/some_test_file
HTTP/2 503
cache-control: no-cache
content-type: text/html
1 Like
#12

Yes, confirming that is what I see also:

$ curl -I https://books.darkvirtue.com/.well-known/acme-challenge/some_test_file
HTTP/2 503
cache-control: no-cache
content-type: text/html
2 Likes
#13

The redirect from HTTPS back to HTTP (the login page) is coming from Calibre-Web and not really relevant here since HAProxy is directing the traffic based on path.

The 503 is correct at the moment since the certbot standalone webserver is only running when it's trying to renew.

#14

However I also see this:

$ curl -I https://books.darkvirtue.com/
HTTP/2 302
content-type: text/html; charset=utf-8
content-length: 236
location: http://books.darkvirtue.com/login?next=%2F
content-security-policy: default-src 'self'  'unsafe-inline' 'unsafe-eval'; font-src 'self' data:; img-src 'self'  data:
strict-transport-security: max-age=31536000;
vary: Cookie
set-cookie: session=eyJfZmxhc2hlcyI6W3siIHQiOlsibWVzc2FnZSIsIlBsZWFzZSBsb2cgaW4gdG8gYWNjZXNzIHRoaXMgcGFnZS4iXX1dfQ.Y4-p3w.y9HEV8zn5ajFtA6Xm3M8QrdJSlk; HttpOnly; Path=/; SameSite=Lax; Secure
strict-transport-security: max-age=31536000; preload; includeSubDomains
x-frame-options: SAMEORIGIN
x-xss-protection: 1;mode=block
referrer-policy: no-referrer,same-origin,strict-origin,strict-origin-when-cross-origin
x-content-type-options: nosniff
2 Likes
#15

Yeah, that's fine. Calibre-Web is a little dumb about preserving the scheme. If you curl that http:// URL, you'll see it's then redirected to HTTPS (by HAProxy).

For visibility, these are the relevant pieces of my HAProxy configuration. I'll expand it to include the books backend:

frontend http-virtual
        bind :::80
        bind *:80
        bind :::443 ssl crt /etc/haproxy/certs/ alpn h2,http/1.1
        bind *:443 ssl crt /etc/haproxy/certs/ alpn h2,http/1.1

        acl letsencrypt path_beg -i /.well-known/acme-challenge

        redirect scheme https code 301 if !{ ssl_fc }

        use_backend letsencrypt if letsencrypt
        use_backend books if { hdr(host) -i books.darkvirtue.com }

backend letsencrypt
        server certbot 127.0.0.1:10443

backend books
        server calibre-web-books 127.0.0.1:8084

So if an URL path starts with /.well-known/acme-challenge, that request will be routed to Certbot. If Certbot isn't running, then it will result in a 503.

1 Like
#16

HTTP-01 challenge Challenge Types - Let's Encrypt

Let's Debug is failing because
/.well-known/acme-challenge/gbRpIdYpCAS05EHaJYXucYli7_rBNQ7c1MbEhaiQJ28
fails to retrieve the contents of that file, even with the redirect.

$ curl -I http://books.darkvirtue.com/.well-known/acme-challenge/gbRpIdYpCAS05EHaJYXucYli7_rBNQ7c1MbEhaiQJ28
HTTP/1.1 301 Moved Permanently
content-length: 0
location: https://books.darkvirtue.com/.well-known/acme-challenge/gbRpIdYpCAS05EHaJYXucYli7_rBNQ7c1MbEhaiQJ28

$ curl -I https://books.darkvirtue.com/.well-known/acme-challenge/gbRpIdYpCAS05EHaJYXucYli7_rBNQ7c1MbEhaiQJ28
HTTP/2 503
cache-control: no-cache
content-type: text/html
2 Likes
#17

Yes, the 503 is correct. I'm using the standalone plugin so the webserver backend that would answer requests for /.well-known/acme-challenge is only available while certbot is running - for example, only during renewal attempts.

The Let's Debug tool is misrepresenting the error. A 503 is not a timeout - it's an explicit message that the backend is not available. A 504 error indicates a timeout.

2 Likes
#18

@Vicerious,

Kindly wait for more knowledgeable Let's Encrypt community volunteers to assist.

2 Likes
#19

Will do, thanks @Bruce5051. I appreciate the help.

2 Likes
#20

Perhaps try:
server certbot 127.0.0.1:10443/.well-known/acme-challenge

[just taking a wild stab at this; I know nothing about HAProxy]

OR place a test text file at the expected challenge location at try:
http://127.0.0.1:10443/.well-known/acme-challenge/your.test.file

3 Likes