Executing a hook after obtaining challenges, before verifying them

I have two servers behind a load balancer, with >350 domains pointed to the load balancer’s IP, which are redirected 50/50 to one of two nodes. All management including certbot is to be carried out on node 01.

There is an NFS mount point for /var/www/vhosts and Unison synchronises specified files and directories including all Apache conf files, LE certs and LE http-01 challenge files.

The problem is that I need to run a command to force sync of the challenge files and reload apache on the 02 node between writing the challenge files and requesting authorization from the ACME server. Because if I don’t, only the 01 node has the challenge files and temporary conf edits, and when the auth request comes in via the load balancer, it is sometimes hitting the 02 node and getting a 404.

I haven’t found quite enough documentation on the --deploy-hook option, but I’ve tried pre and post hooks and they don’t work.

I appreciate a DNS challenge might be an alternative but the provider (Gandi) has an enforced minimum TTL and what appears to be slow propagation. I don’t believe I can achieve automation of this without other issues and would like certbot to handle Apache configuration for me, so the http-01 challenge makes sense. Any and all advice appreciated.

My domain is:
maple-motor-services.co.uk

I ran this command:
certbot --expand --reinstall --apache --redirect --non-interactive --preferred-challenges http --deploy-hook “/usr/bin/file_sync.sh && httpd -t && apachectl graceful && ssh -p2020 merr-web-02 apachectl graceful” -d adm02.clients.merrehill.co.uk,c.adm02.clients.merrehill.co.uk,www.adm02.clients.merrehill.co.uk,maple-motor-services.co.uk,c.maple-motor-services.co.uk,www.maple-motor-services.co.uk

It produced this output:
IMPORTANT NOTES:

  • The following errors were reported by the server:

Domain: c.adm02.clients.merrehill.co.uk

Type: unauthorized

Detail: Invalid response from

http://c.adm02.clients.merrehill.co.uk/.well-known/acme-challenge/foifZ0lk0QkjVcPCiixnUFcWX-34Pk3IREiTX5UV4Dc

[159.253.213.20]: "<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML

2.0//EN">\n<html><head>\n<title>404 Not

Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p"

Domain: www.maple-motor-services.co.uk

Type: unauthorized

Detail: Invalid response from

http://www.maple-motor-services.co.uk/.well-known/acme-challenge/Gn8sQkA6seLOVSwpfMmiAJxakUWgRVQOgKYjKeXqRGc

[159.253.213.20]: "<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML

2.0//EN">\n<html><head>\n<title>404 Not

Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p"

Domain: c.maple-motor-services.co.uk

Type: unauthorized

Detail: Invalid response from

http://c.maple-motor-services.co.uk/.well-known/acme-challenge/Xkp0lwDVIC3ROFemTUWOtZxUtxMKH8lT_N1ygkdZU28

[159.253.213.20]: "<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML

2.0//EN">\n<html><head>\n<title>404 Not

Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p"

To fix these errors, please make sure that your domain name was

entered correctly and the DNS A/AAAA record(s) for that domain

contain(s) the right IP address.

My web server is (include version):
Apache 2.4.6

The operating system my web server runs on is (include version):
CentOS 7.6.1810

My hosting provider, if applicable, is:
N/A

I can login to a root shell on my machine (yes or no, or I don’t know):
Yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):
Webmin 1.900/Virtualmin 6.06

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot):
certbot 0.30.2

Hi @Sam-Butler

these hooks don't help, it's not your situation.

Isn't it possible to create an exception in your load balancer, so that /.well-known goes to node 1?

Mhm ( https://check-your-website.server-daten.de/?q=maple-motor-services.co.uk ):

Domain: maple-motor-services.co.uk
Primary: a.dns.gandi.net
Mail: hostmaster.gandi.net
Serial: 1550942871
Refresh: 10800
Retry: 3600
Expire: 604800
TTL: 10800

10800 seconds = 180 minutes = 3 hours. Letsencrypt checks the authoritative nameservers directly, so I don't know if this is really a problem.

Thanks Jürgen, I appreciate your quick response. The LB is shared so I’ll need to approach the provider on the possibility of this configuration, but it sounds like it might solve a lot of issues.

Good point regarding the DNS challenge, I’ll have to look at how this could work with the Apache plugin, just in case.

Another thought I had is whether it might be possible to change the filesystem location of the http-01 challenge files - do you know?

Isn't it possible to create another server / domain that doesn't use the loadbalancer?

Then you can redirect all queries static (http status 301) to that server.

PS: My own service uses a direct code solution:

If it is http and starts with /.well-known/acme-challenge, then the query

domainname/.well-known/acme-challenge/filename

produces a check:

Is there a file in D:\sd-web\acme-challenge\domainname.filename.txt

If yes, the content of this file is sent.

My own client doesn't create files in webroot/.well-known/acme-challenge, instead all files are created in this special directory.

That's because those two hooks are not "integrated" to any of the authenticator plugins. They can be used with every method of authentication and are for "general" stuff to do, such as stopping a webserver if required.

I believe running the apache authenticator plugin with two servers is highly complex. I think you're better of choosing another authenticator plugin.

For example, the manual authenticator plugin has the --manual-auth-hook and --manual-cleanup-hook. Those two commands are integral to the manual plugin, which can do the http-01 challenge too.

Read more about the manual plugin here User Guide — Certbot 2.7.0.dev0 documentation and about the two hooks here: User Guide — Certbot 2.7.0.dev0 documentation

In the script used by the authentication hook, you'll need to "perform" the installation of the challenge file manually on both servers. So on the 01 webserver, running certbot, you'll need to script the installation of the challenge file in the webroot of the 01 webserver. And the script also needs to sync this challenge file to server 02.

Another method is redirecting all /.well-known/acme-challenge/ requests to a single "authenticator" webserver. For example, you might add the hostname acme-challenge.merrehil.co.uk to webserver 01 running certbot. And configure your load balancer that every request to acme-challenge.merrehil.co.uk shouldn't be balanced 50/50 between both servers, but only to server 01. Then, set up a redirect on server 02 for the /.well-known/acme-challenge/ directory to acme-challenge.merrehil.co.uk/.well-known/acme-challenge/. That way, if Let's Encrypt requests a challenge file, ends up on server 02, it gets redirected to the special acme-hostname, which ends up on server 01.

1 Like

As you've both mentioned performing challenges outside the LB I think that could be a solution, potentially a mix of the two is the following:

  1. Create a new vhost on a different (sub)domain (the "challenge host".)
  2. Use /var/lib/letsencrypt/http_challenges as the document root for this vhost.
  3. Point the challenge host's DNS directly to the 01 node, bypassing the load balancer.
  4. Redirect HTTP requests for the /.well-known/acme-challenge directory to the challenge host.

I would just need to solve (4) in Apache for hundreds of existing domains.

I will still approach the LB provider to see if they can add an exception as mentioned earlier.

The --manual-auth-hook mentioned by @Osiris is the one intended for use in your situation. It does mean that you have to do a little bit more work in the hook script than you would if you were using --apache. It is intended for use in a situation where your script itself performs the necessary steps to complete the challenge (given challenge details provided by Certbot in environment variables).

You can use -a manual -i apache --preferred-challenges http --manual-auth-hook /usr/bin/my_auth_hook.sh. Differently from what you're expecting with --apache, your hook will have to create the challenge files as well as synchronizing them, but Certbot will pass environment variables telling the script what it needs to do.

Unlike other uses of -a manual, this does work with certbot renew.

1 Like

It's a common mistake, but the TTL on the record has no bearing on how long it takes for Gandi's nameservers to propagate record changes you make. It's a value that instructs standard DNS clients to cache the query results for that length of time before re-requesting from the source nameserver. But as @JuergenAuer mentioned, the LE validation servers query the authoritative nameservers directly and don't cache results specifically for this reason. So as soon as the Gandi nameservers get the record, the validation should succeed.

In my highly un-scientific testing just now with Gandi, their nameservers respond almost instantly to changes. Like the time it took for me to click save in the browser GUI, switch to my terminal, and press up+enter to re-run my dig queries, the change I had made was live on all 3 nameservers that are authoritative for my domain.

2 Likes

This solved the problem. To redirect requests for the challenge directory, we used mod_rewrite, placing directives above the virtual host blocks and adding RewriteOptions InheritBefore to each. From Apache 2.4.8 you can simply use RewriteOptions InheritDownBefore at the parent level. Thanks to those who responded for the valuable help.

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.