Incomplete authorizations when requesting certificates for domains using CNAME record

Help
1

Hi,
First of all thanks for letsencrypt free services.
My scenario:- We’ve a domain app.mydomian.com hosted on two EC2 instances using aws load balancer.
We’ve hosting several other sites from app.mydomian.com (some subdomains, some custom domains e.g. www.abc.com). app.mydomian.com is using ssl/tls certificate from another CA. We want letsencrypt certificates for custom domains which have CNAME record as app.mydomian.com. Is it possible ?
I tried to get certificate for single custom domain & installing certbot on both using ansible ( by geerlingguy), but getting “Incomplete authorizations” error message. Full error note:
“To fix these errors, please make sure that your domain name was
entered correctly and the DNS A/AAAA record(s) for that domain
contain(s) the right IP address.”

Please guide me in right direction.

2

It should be possible to use CNAMEd records for LE certs.
CNAMEs merely delegate the IP resolution of FQDN1 to the CNAME (FQDN2).
The cert, site, and authentication, will still all reference FQDN1 (it will is just be served at FQDN2 system).

1 Like
3

@rg305 thanks for your prompt response. What am I missing here then ?

4

This may be problematic.
When the name resolves to multiple IPs, or is being load-balanced between multiple systems, then there is no guarantee which system will receive the authentication request.
You run it on server but server may get the request and can not full it, so the request is denied:

You could try it from both systems at the same time, if one is preferred in the load-balancer, then that one could get authenticated.
With the other... you could retry it a few times and it might (eventually) here the authentication request.
Or, if you could control the load-balancer for the duration of the request, you could insure which server gets the authentication requests.

1 Like
5

As I’m using ansible here to install certbot & issuing certificates for the domains, I think it tried to authorize at the same time.

6

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:

I ran this command:

It produced this output:

My web server is (include version):

The operating system my web server runs on is (include version):

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don’t know):

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):

1 Like
7

My domain is: www.matrixautosales.com(domain for which cert is requested) which CNAME to app.motorlot.com

I ran this command: letsencrypt certonly --standalone --noninteractive --agree-tos --email "myemail@gmail.com" -d www.matrixautosales.com" (This command was ran by ansible(provisioning tool) on both EC2 instances)

It produced this output: Incomplete authorizations

My web server is (include version): nginx/1.14.0

The operating system my web server runs on is (include version): ubuntu 16.04

My hosting provider, if applicable, is: Aws

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): No

8

Could you configure the web servers to proxy /.well-known/acme-challenge/* to the ansible server, run the letsencrypt command on the ansible server, and copy the resulting renewed fullchain.pem back to the web servers?

You can use the --reuse-key option if your version of certbot is recent enough (certbot is the new name for the letsencrypt program); otherwise it will generate a new private key with every renewal and you’ll have to copy that to the web servers too.

If your ansible server is not accessible from the web servers, you could instead use --manual with --manual-auth-hook and --manual-cleanup-hook scripts to copy the challenge files to the two web servers. Or maybe consider using getssl instead of certbot; it has a remote webroot feature that does the same thing a bit more conveniently.

1 Like
9

Thnaks for the help @jmorahan, but I think you got me wrong. ansible is only provisioning - install packages, install webserver, copy configuration files, tweak settings, create directories, etc.

Update:- Today tried using certbot-auto package(0.29.1) (again this was installed on both instances by ansible).
When I create certificate(Ansible runs command certbot-auto certonly --standalone --noninteractive --staging --agree-tos --email myemail@gmail.com -d www.matrixautosales.com one by one on both instances), it create one for one instance but gives 504 error on another instance.

Full error message on second instance: FailedChallenges: Failed authorization procedure. www.matrixautosales.com (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.matrixautosales.com/.well-known/acme-challenge/y1IsCWJPKEx0I-4SSUQBXGoi0xoCAMuyMj297S8NFJ0 : 504

Please point me in right direction. Thanks!

10

What else did it output?

Is http://www.matrixautosales.com/ using a load balancer?

Can you review the load balancer logs to see why it returned a 504 error?

Maybe a firewall or security group was blocking port 80?

1 Like
11

What else did it output?
Copy posting logs from letsencrypt.log file -

orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/client.py”, line 389, in _get_order_and_authorizations
authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/auth_handler.py”, line 82, in handle_authorizations
self._respond(aauthzrs, resp, best_effort)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/auth_handler.py”, line 161, in _respond
self._poll_challenges(aauthzrs, chall_update, best_effort)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/auth_handler.py”, line 232, in _poll_challenges
raise errors.FailedChallenges(all_failed_achalls)
FailedChallenges: Failed authorization procedure. www.matrixautosales.com (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.matrixautosales.com/.well-known/acme-challenge/y1IsCWJPKEx0I-4SSUQBXGoi0xoCAMuyMj297S8NFJ0

Is http://www.matrixautosales.com/ using a load balancer?

Yes, it Cname to app.motorlot.com which is served by load balancer & 2 EC2 instances.

Maybe a firewall or security group was blocking port 80?
No

12

Was there any more information in this line?

All we have to go on is that your website said 504.

Does it work if you try again?

What do Amazon's logs show?

1 Like
13

Thanks @mnordhoff for your help. When trying again today I didn’t get 504 error instead get different error message on one Ec2 instance app2 (same as yesterday), on other EC2 instance(app1) its just running fine. Also both having same configuration with no firewall.

I get this error message, copied exactly from letsencrypt.log

FailedChallenges: Failed authorization procedure. www.matrixautosales.com (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.matrixautosales.com/.well-known/acme-challenge/aRtIBlrftNM9eNxGiA8uyvE76VWKqpbYUfudiIwItMI: "\n\n<html lang=“en”>\n \n <meta charset=“utf-8” />\nMatrix Auto Sales, LLC\n<meta name=“descri”

14

How many EC2 instances are there serving http://www.matrixautosales.com/? Is it possible that the HTTP request is going to the wrong EC2 instance?

What do the logs say?

Edit:

Wait, you answered the first question already. 2 instances.

I think it’s likely that you’re running Certbot on one instance, and the ALB/ELB is sending the HTTP request from Let’s Encrypt to the other instance, which has no idea what’s going on and returns a 404 Not Found page.

The web server or ALB/ELB logs should be able to confirm that.

Do you want to issue 2 different certificates, 1 on each instance?

2 Likes
15

I think you are right, when certbot command execute on app2 server, it performs http-01 challenge on www.matrixautosales.com which maybe sending to app1 server. But performing same step on app1 is successful. But why it happens almost everytime ? I haven't enabled access logs for ELB.

Yes because it pretty handy to run same certbot command on both instances, otherwise have to copy paste files. Is this wrong way to do it ? If you know some better way with ELB please let me know.

16

In a vacuum, I’d suggest using DNS-01 validation, but you want to avoid that because it would inconvenience your users.

If you had more instances, I’d suggest having one obtain the certificates and then automatically copy them to the other instances – getting tons of duplicate certificates is inefficient and runs into the Let’s Encrypt rate limits. But with only two servers, it’s not that bad.

You can configure each of your instances to redirect or reverse proxy /.well-known/acme-challenge/ requests that don’t exist to the other instance. (But be sure to avoid creating a proxy/redirect loop.)

For example, you could have instance 1 do:

location /.well-known/acme-challenge/ {
    try_files $uri @redirect;
}
location @redirect {
    return http://instance-2.example.net$request_uri;
}

And then have instance 2 do the opposite.

That way it would serve the file if it existed, or send the client to the other server if it doesn’t.

2 Likes
17

Thanks @mnordhoff for your suggestions. I added proxy settings on both instances.

But now it is again failing with 504.

FailedChallenges: Failed authorization procedure. www.matrixautosales.com (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.matrixautosales.com/.well-known/acme-challenge/2lzKGldRVR2QmJdJg5M_Gi803R8WiXhHhJSPvl9m5VA : 504

I more thing I forget to specify earlier, every time this challenge fails to validate, that instance crashes!
I have to manually restart instance from aws console.

This time both app1 & app2 crashed after adding proxy settings on both servers. Sorry if it frustrating you now!

18

What’s the error message?

19

This one from letsencrypt logs

I tried to install two times with proxt configuration. First time error message was:
Failed authorization procedure. www.matrixautosales.com (http-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://app2.motorlot.com/.well-known/acme-challenge/PF_McQE0fM2_UR0Ye40TE0M-NPwvU3cLXGohUJsohxs: Timeout after connect (your server may be slow or overloaded)"

20

Do you have anything else to get it working @mnordhoff ?