Haproxy giving 503 using Webroot method

Hi Everyone,

I’m currently attempting to setup Let’s Encrypt via HaProxy on a Centos 7 machine using the Webroot method. The issue I’m having is receiving 503 errors on the specific URL that’s generated:

(IP’s and URL edited for security)

certbot certonly --webroot -w /home/sites -d example.com -d www.example.com

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for example.com
http-01 challenge for www.example.com
Using the webroot path /var/www for all unmatched domains.
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. www.example.com (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.example.com/.well-known/acme-challenge/Mxh1WV4OE2grdgdXCVBIUe0P6EcJIEbI [111.222.333.444]: 503, 

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: www.domain.com
   Type:   unauthorized
   Detail: Invalid response from
   http://www.domain.com/.well-known/acme-challenge/Mxh1WV4OE2x5LpgdfgdfgXCVBIUe0P6EcJIEbI
   [111.222.333.444]: 503

Here is the config file inside of: /etc/haproxy/haproxy.cfg

frontend www-http
        bind 111.222.333.444:80
        reqadd X-Forwarded-Proto:\ http
        #default_backend www-backend
        acl letsencrypt-acl path_beg /.well-known/acme-challenge/
        use_backend letsencrypt-backend if letsencrypt-acl
        default_backend www-backend

frontend www-https
        bind 111.222.333.444:443 ssl crt /etc/haproxy/certs/example.com.pem
        reqadd X-Forwarded-Proto:\ https
        acl letsencrypt-acl path_beg /.well-known/acme-challenge/
        use_backend letsencrypt-backend if letsencrypt-acl
        default_backend www-backend

backend www-backend
        redirect scheme https if !{ ssl_fc }
        server www-1 111.222.333.777:80 check
        server www-2 111.222.333.888:80 check

backend letsencrypt-backend
        server letsencrypt 127.0.0.1:54321

At the moment, when I go to my specific URL in question, such as example.com/.well-known/acme-challenge/ then it gives me a 503 error on the browser. Therefore, when attempting to use web-root to generate my SSL, it’s failing with a 503 error.

I’ve compared my configs with around 20 other examples and they’re very similar in regards to the section for letsencrypt-acl. If I edit the URL inside the config file to something else, then the site loads fine on my browser, but the script still fails with a different error. So it’s deffo related to that specific section.

Doing an NMAP on my server, I can see that the port is closed, which I’ve attempted to open, although I doubt that it’s related. My assumption is that it’s attempting to load that URL over port 54321 which isn’t giving a response, so it’s generating a 503.

Is my theory correct and I need to look towards making that port accessible online, or do I need a different configuration to make it publicly accessible?

Thank you.

I use a very similar config - what you want is standalone and not webroot:

certbot certonly -a standalone --http-01-port 54321 -d example.org
1 Like

From the config you should understand that:
haproxy is bound to
111.222.333.444:80
111.222.333.444:443 (using ssl)
and when the path begins with
/.well-known/acme-challenge/
it will use the LE backend.
[Which is defined to be listening at 127.0.0.1:54321]

So…
There would have to be something listening at 127.0.0.1:54321 for this to work.
@_az says you can use certbot in standalone mode at that address:port.
Which makes a lot of sense and can secure and simplify things greatly.
Only certbot listens there and only does so when trying to renew a cert.
At all other times, there is no backend found and error 503 is returned.

I think you could, however, also provide any other web service at 127.0.0.1:54321 to get this to work.
So long as it used a document root that matches the --webroot you provided to certbot.
[–webroot -w /home/sites]

The certificate authentication results should be the same.
But the service would be enabled and listening at all times - all year long.
So, from a security perspective, I tend to favor the certbot standalone approach; Because it remains shutdown for 60 days and then only spins up when needed to validate a new cert (renewal) [which should be completed is under a minute - that’s less than 6 minutes per year]

1 Like

Hi,
Thanks for the suggestions.
I originally wanted to go with Webroot to prevent having to temporarily disable Haproxy to run the standalone version.

However, specifying the port for standalone seems to have resolved the issue. Configured a cron on the server to run every month to run certbot renewal and everything seems perfect.

Thanks a lot!

Although, I’m glad to hear that you got cert :slight_smile:
You should be made aware that Certbot will try to install an automatic renewal job.
And now you may have two jobs doing the same thing (at different intervals).

Please check the most common cron and systemd location settings with:
crontab -l
sudo crontab -l
systemctl list-timers | grep -i cert

and, in any case, please also show the command specified in your job,

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.