Certbot only works with `--debug-challenges`

Help
1

My domain is: fulaphex.duckdns.org

I ran this command: cerbot -d fulaphex.duckdns.org

It produced this output:

sudo ~/certbot certonly --standalone -d fulaphex.duckdns.org -m gabor94@gmail.com -v
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None
Requesting a certificate for fulaphex.duckdns.org
Performing the following challenges:                                                                                                                                                  http-01 challenge for fulaphex.duckdns.org                                                                                                                                            Waiting for verification...
Challenge failed for domain fulaphex.duckdns.org                                                                                                                                      http-01 challenge for fulaphex.duckdns.org
Certbot failed to authenticate some domains (authenticator: standalone). The Certificate Authority reported these problems:
Domain: fulaphex.duckdns.org
Type:   connection
Detail: 141.147.97.210: Fetching http://fulaphex.duckdns.org/.well-known/acme-challenge/gWLAJffTd_KTxpQ7h4IyXCMHhTzCUyiUbBGrb2JNi7E: Error getting validation data
Hint: The Certificate Authority failed to download the challenge files from the temporary standalone webserver started by Certbot on port 80. Ensure that the listed domains point to this machine and that it can accept inbound connections from the internet.

Cleaning up challenges                                                                                                                                                                Some challenges have failed.                                                                                                                                                          Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

The operating system my web server runs on is (include version): manjaro linux

I can login to a root shell on my machine (yes or no, or I don't know): yes

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 2.6.0

I have a setup with two machines:

  • A has a public IP and is running a proxy to
  • B that cannot be reached directly from the internet
    I'm running wireguard on A and connect from B through that, and A is running haproxy that proxies all the traffic on 80 and 443 to B. I can run certbot directly on A and it works okay, but I need to run it on B.
    When running certbot on B I get the error mentioned above. The only way that I got it to succeed was to run with --debug-challenges -v and waiting a couple seconds (I was testing that I can curl the challenge from a separate machine). By the time I copy and paste that url between terminal windows I can already curl the challenge, so the extra time there isn't more than a couple of seconds. Has somebody encountered such error before? Any ideas on what I could be doing wrong?

haproxy.conf from A

global
	log /dev/log	local0
	log /dev/log	local1 notice
	chroot /var/lib/haproxy
	stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
	stats timeout 30s
	user haproxy
	group haproxy
	daemon

	# Default SSL material locations
	ca-base /etc/ssl/certs
	crt-base /etc/ssl/private

	# See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
        ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
        ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
        ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
	log	global
	mode	http
	option	httplog
	option	dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
	errorfile 400 /etc/haproxy/errors/400.http
	errorfile 403 /etc/haproxy/errors/403.http
	errorfile 408 /etc/haproxy/errors/408.http
	errorfile 500 /etc/haproxy/errors/500.http
	errorfile 502 /etc/haproxy/errors/502.http
	errorfile 503 /etc/haproxy/errors/503.http
	errorfile 504 /etc/haproxy/errors/504.http

#---------------------------------------------------------------------
# Proxys to the webserver backend port 80
#---------------------------------------------------------------------
frontend http
	bind :80
	mode tcp
	option tcplog

	# Wait for a client hello for at most 5 seconds
	tcp-request inspect-delay 5s
	tcp-request content accept if { req_ssl_hello_type 1 }

	default_backend http

backend http
	mode tcp
	balance roundrobin
	server optiplex 10.23.0.2:80 check
#---------------------------------------------------------------------
# Proxys to the webserver backend port 443
#---------------------------------------------------------------------
frontend ssl
	bind :443
	mode tcp
	option tcplog

	# Wait for a client hello for at most 5 seconds
	tcp-request inspect-delay 5s
	tcp-request content accept if { req_ssl_hello_type 1 }

	default_backend ssl

backend ssl
	mode tcp
	balance roundrobin
	server optiplex 10.23.0.2:443 check
2

What were the problems that it reported? There is usually detailed error messages

never mind about that. I just realized the other text is far far right of the scroll window

2 Likes
3

I just realised, that when copy-pasting from terminal, I somehow copied all the terminal whitespace and that's why it was formatted in a wonky way. I tried to fix up the original post, should need better now

4

Can't you just copy the certs from A to B?

I have never heard that with --standalone and do not know how that could happen within Certbot. It sets up the challenge response internally so would be ready immediately before it requests the cert challenge from Let's Encrypt.

Could there be some timing issue caused by Wireguard or Haproxy such that they mishandle the incoming HTTP request(s)?

You should see multiple incoming HTTP requests from the Let's Encrypt servers. Do the Wireguard or HA logs show these arriving?

4 Likes
5

These two statements seem to contradict each other:

There must be more that has been left unsaid OR something has been overlooked...

Like where is :80 ?

2 Likes
6

The haproxy config file got messed up as well, I'll update the original post in a minute :frowning_face: sorry

EDIT:
configs are reformatted now

1 Like
7

technically, I could do that, but I need those certs on B, so I would still have to manually copy them over. I can as well try scripting the --debug-challenges to run on B, but ideally I'd like to figure out where things are getting lost. I'll check the haproxy logs and report back

1 Like
8

I thought Certbot ran fine on A. Did it also need --debug-challenges?

3 Likes
9

Sorry, certbot is working fine on A, but the certificates live on B, that's where I have the nginx reverse proxy which uses those certs.

I just realised that I made a typo, certbot runs fine on A but I want it running automatically and refreshing the certs on B

1 Like
10

OK. Understand. It is worth looking at your logs to see differences between running Certbot on A or B

Other than just copying the certs why do you need HTTPS between A and B? Can't you just terminate HTTPS in HAProxy and proxy back to B using HTTP? Isn't this a private local network? Or, would HAProxy allow using a self-signed cert on B for HTTPS?

3 Likes
11

I'm still confused, you continue to say:

And:

If haproxy sends ALL the traffic on 80 to B, then how can certbot on A work?

3 Likes
12

For testing certbot on A I stopped haproxy :slightly_smiling_face:

1 Like
14

So, do you know for sure if haproxy does send ALL traffic on 80 to B [when turned on]?
Are there any web server logs, on B, to show that?

2 Likes
15

Looking at haproxy logs shed some light on the situation. For anybody in the future doing something equally bizarre - the problem was with haproxy configuration.

backend http
	mode tcp
	balance roundrobin
	server optiplex 10.23.0.2:80 check

this backend has one server and it's running health checks on it. When the challenges from certbot are brought up in the standalone server it takes haproxy a couple of seconds to detect that server as healthy and start sending requests to it. if one removes the checks on the http traffic and changes it to

backend http
	mode tcp
	balance roundrobin
	server optiplex 10.23.0.2:80

then all the requests are sent immediately and certbot works flawlessly

2 Likes
16

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.