Certbot - Hard Coded URL Rewrite Means HTTP-01 Challenge Fails on Apache

Server
1

Hi,

I have been using Let’s Encrypt for 3 months now on a CentOS 7 and Apache . It is time to renew my certs and I have not been able to do so. I spent the night reading other topics with similar issues but I found no solution. It is critical for me to renew and restore my website, it’s been down for 4 hours now. :((

What I have tried so far:

Running the command certbot renew generates the error:

  Cert is due for renewal, auto-renewing...
  /usr/lib64/python2.7/site-packages/cffi/model.py:526: UserWarning: 'point_conversion_form_t' has no values explicitly defined; next version will refuse to guess which integer type it is meant to be (unsigned/signed, int/long)
    % self._get_c_name())
  Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
  Renewing an existing certificate
  Performing the following challenges:
  tls-sni-01 challenge for mydomain.com.br
  tls-sni-01 challenge for www.mydomain.com.br
  Waiting for verification...
  Cleaning up challenges
  Error while running apachectl graceful.

  Job for httpd.service invalid.

  Attempting to renew cert from /etc/letsencrypt/renewal/mydomain.com.br.conf produced an unexpected error: Error while running apachectl graceful.

  Job for httpd.service invalid.
  . Skipping.

  All renewal attempts failed. The following certs could not be renewed:
    /etc/letsencrypt/live/mydomain.com.br/fullchain.pem (failure)
  1 renew failure(s), 0 parse failure(s)

  IMPORTANT NOTES:
   - The following errors were reported by the server:

     Domain: mydomain.com.br
     Type:   unauthorized
     Detail: Incorrect validation certificate for tls-sni-01 challenge.
     Requested
     883b5ce27275ef075b699399af96855c.75421e9dc4f12985d858d2afbd93fa37.acme.invalid
     from 123.12.12.123:443. Received 2 certificate(s), first
     certificate had names "mydomain.com.br, www.mydomain.com.br"

     Domain: www.mydomain.com.br
     Type:   unauthorized
     Detail: Incorrect validation certificate for tls-sni-01 challenge.
     Requested
     62709006899fc09be4fae4072edcd408.822483677484878c61c3ced918c4b756.acme.invalid
     from 123.12.12.123:443. Received 2 certificate(s), first
     certificate had names "mydomain.com.br, www.mydomain.com.br"

     To fix these errors, please make sure that your domain name was
     entered correctly and the DNS A record(s) for that domain
     contain(s) the right IP address.

I have also tried the following certbot certonly --expand --webroot --webroot-path /my_path/mydomain.com.br/public_html/ and ultimately got the following error:

     Failed authorization procedure. mydomain.com.br (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://mydomain.com.br/.well-known/acme-challenge/hqZLYG1Aops-04Jp-0nPCgo-_fqGCHcy9nnng4GJoLU: "<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
     <html xmlns="http:", www.mydomain.com.br (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.mydomain.com.br/.well-known/acme-challenge/pweTdryjacukKwLfei1PXdZZz4LQiE_oecQAJ_Bejq0: "<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
     <html xmlns="http:"

     IMPORTANT NOTES:
      - The following errors were reported by the server:

        Domain: mydomain.com.br
        Type:   unauthorized
        Detail: Invalid response from
        http://mydomain.com.br/.well-known/acme-challenge/hqZLYG1Aops-04Jp-0nPCgo-_fqGCHcy9nnng4GJoLU:
        "<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
        "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
        <html xmlns="http:"

        Domain: www.mydomain.com.br
        Type:   unauthorized
        Detail: Invalid response from
        http://www.mydomain.com.br/.well-known/acme-challenge/pweTdryjacukKwLfei1PXdZZz4LQiE_oecQAJ_Bejq0:
        "<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
        "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
        <html xmlns="http:"

        To fix these errors, please make sure that your domain name was
        entered correctly and the DNS A record(s) for that domain
        contain(s) the right IP address.

I checked my DNS Record A and it is pointing to my public IP. One thing that puzzled me however is, when I checked my snginx.conf file, the servcer is listening to my IP NAT inside, the private ip at the 443 port. I changed it to the same IP that the DNS Record A is pointing, but the nginx won’t restart returning an erro; so I am assuming it is suppose to point to my IP Nat (?). Maybe this has nothing to do with it.

Checking the httpd.conf and shttpd.conf files, they are pointing to my public IP and ports 8080 and 8443.

I think there might be a misconfiguration on these files, and then when certbot tries to renew the certificates, apache graceful fails because of this misconfiguration. It is getting to the point the more I look at it the more confused I get… I tried all sorts of things… please help!

2

Hi @umagrama,

Do you have multiple virtual hosts defined in a single Apache configuration file?

Can you normally run sudo apachectl graceful and sudo apachectl configtest without errors?

There are also other things we can look at for the TLS-SNI-01 and HTTP-01 challenge failures.

3

Thanks for responding schoen!

I figured out my misconfigurations of my apache server, for a reason that I am yet to learn, I have to use my IP NAT inside internally, which is referenced to from outside through my public IP. This issue is solved (one down! :))

Now I am back to trying to renew my certs. When I run certbot renew I get Error while running apachectl graceful. I also get the Incorrect validation certificate for tls-sni-01 challenge.

I ran apachectl graceful and apachectl configtest without a problem.

One thing that standouts to me is that in my snginx.conf file, I have this setup:

	ssl_certificate	 /etc/letsencrypt/live/mydomain.com/fullchain.pem ;
	ssl_certificate_key  /etc/letsencrypt/live/mydomain.com/privkey.pem ;

And in my shttpd.conf I have:

	SSLCertificateFile ..path/conf/web/ssl.mydomain.com.crt
	SSLCertificateKeyFile ..path/conf/web/ssl.mydomain.com.key
	SSLCertificateChainFile ..path/conf/web/ssl.mydomain.com.ca

Would you say this is correct?

What else can I check?

Good news is that the site back and running with SSL, but it will expire by the end of the day :X

4

I tried certbot certonly --webroot and pointed to my ..path/public_html, I receive the following error:

	Failed authorization procedure. mydomain.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: The key authorization file from the server did not match this challenge [wpbtQyjH3jjcJO33Z-LvfGm6oL2dcqrHy4aoyR0PHmM.HHFwSGhKzkhDd_OBfs4jkVrn3saF9Hp2kvysceRBrpg] != [wpbtQyjH3jjcJO33Z-LvfGm6oL2dcqrHy4aoyR0PHmM.yxgGDRnEH9R5fC2kXvBqXINg_USLfrOwMlTyy10HKwY], www.mydomain.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: The key authorization file from the server did not match this challenge [PBKCCcKT2ElvuLFOFDn6fwwNF4N7kyqJinf-zYdPEho.HHFwSGhKzkhDd_OBfs4jkVrn3saF9Hp2kvysceRBrpg] != [PBKCCcKT2ElvuLFOFDn6fwwNF4N7kyqJinf-zYdPEho.yxgGDRnEH9R5fC2kXvBqXINg_USLfrOwMlTyy10HKwY]

	IMPORTANT NOTES:
	 - The following errors were reported by the server:

	   Domain: mydomain.com
	   Type:   unauthorized
	   Detail: The key authorization file from the server did not match
	   this challenge
	   [wpbtQyjH3jjcJO33Z-LvfGm6oL2dcqrHy4aoyR0PHmM.HHFwSGhKzkhDd_OBfs4jkVrn3saF9Hp2kvysceRBrpg]
	   !=
	   [wpbtQyjH3jjcJO33Z-LvfGm6oL2dcqrHy4aoyR0PHmM.yxgGDRnEH9R5fC2kXvBqXINg_USLfrOwMlTyy10HKwY]

	   Domain: www.mydomain.com
	   Type:   unauthorized
	   Detail: The key authorization file from the server did not match
	   this challenge
	   [PBKCCcKT2ElvuLFOFDn6fwwNF4N7kyqJinf-zYdPEho.HHFwSGhKzkhDd_OBfs4jkVrn3saF9Hp2kvysceRBrpg]
	   !=
	   [PBKCCcKT2ElvuLFOFDn6fwwNF4N7kyqJinf-zYdPEho.yxgGDRnEH9R5fC2kXvBqXINg_USLfrOwMlTyy10HKwY]

	   To fix these errors, please make sure that your domain name was
	   entered correctly and the DNS A record(s) for that domain
	   contain(s) the right IP address.

I should also add the last error from /var/log/letsencrypt/letsencrypt.log:

	FailedChallenges: Failed authorization procedure. mydomain.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: The key authorization file from the server did not match this challenge [wpbtQyjH3jjcJO33Z-LvfGm6oL2dcqrHy4aoyR0PHmM.HHFwSGhKzkhDd_OBfs4jkVrn3saF9Hp2kvysceRBrpg] != [wpbtQyjH3jjcJO33Z-LvfGm6oL2dcqrHy4aoyR0PHmM.yxgGDRnEH9R5fC2kXvBqXINg_USLfrOwMlTyy10HKwY], www.mydomain.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: The key authorization file from the server did not match this challenge [PBKCCcKT2ElvuLFOFDn6fwwNF4N7kyqJinf-zYdPEho.HHFwSGhKzkhDd_OBfs4jkVrn3saF9Hp2kvysceRBrpg] != [PBKCCcKT2ElvuLFOFDn6fwwNF4N7kyqJinf-zYdPEho.yxgGDRnEH9R5fC2kXvBqXINg_USLfrOwMlTyy10HKwY]
5

@schoen, any other suggestion?

6

hi @umagrama

Yes. Get a working certificate any other way you can

I would suggest using the ZeroSSL client and manually configuring apache.

Certbot does a great job of automating installation and renewal but I think web admin should still be able to fix an apache config and obtain certificates manually

Alternatively you can look at clients such as acme.sh which are linux based and fairly easy to pick up

Andrei

7

For the webroot attempt, do you have multiple domains on the same certificate which might have different webroot paths?

8

Hi @ahaw021,

Thanks for the suggestions! I will look into it. After this issue and many hours of investigation, I am to understand it further.

@schoen,

No, I have only one domain and an alias with www on the certificate. What I find stange is when I ran certbot renew I get the apachectl error and an authorization failure. When I run with --webroot option, I only get the authorization error:

	   Domain: domain.com
	   Type:   unauthorized
	   Detail: The key authorization file from the server did not match
	   this challenge
	   [BPPsYv2YLCE44eOZwGre0PS8UIGkoF8ycxyYlzxm5Gg.HHFwSGhKzkhDd_OBfs4jkVrn3saF9Hp2kvysceRBrpg]
	   !=
	   [BPPsYv2YLCE44eOZwGre0PS8UIGkoF8ycxyYlzxm5Gg.yxgGDRnEH9R5fC2kXvBqXINg_USLfrOwMlTyy10HKwY]

I found a file in my …/conf folder called: nginx.conf_letsencrypt and in it we have:

	location ~ "^/\.well-known/acme-challenge/(.*)$" {
	    default_type text/plain;
	    return 200 "$1.yxgGDRnEH9R5fC2kXvBqXINg_USLfrOwMlTyy10HKwY";
	}

I tried changing the string returned from this function to the string above in the error but had no success.

On another thought, I also find stange that my nginx.conf file points to:

	ssl_certificate	 /etc/letsencrypt/live/domain.com/fullchain.pem ;
	ssl_certificate_key  /etc/letsencrypt/live/domain.com/privkey.pem ;

While my shttpd.conf points to:

	SSLCertificateFile ../conf/web/ssl.domain.com.crt
	SSLCertificateKeyFile ../conf/web/ssl.domain.com.key

I checked and they are not the same. Maybe one is server and the other is client? I don’t know. I’m still getting my head wrapped around how SSL works. Watching videos on youtube doesn’t answer all questions :stuck_out_tongue:

I use VestaCP and Webmin, if that makes any difference.

Anyway, I still haven’t given up. I know it’s something minor and when we find it I will be kicking my own head.

Thanks guys! Any other thoughts and ideas are welcome, I am running out of them.

9

That's a significant find. I would suggest removing this section entirely. This looks like it was inspired by

(and I wonder who or what put it there... this would be very important to figure out!). Unfortunately, if it was originally meant for another system or another account key, it will not work and it may also actively prevent Certbot from working with --webroot!

2 Likes
10

@Neilpang, do you know how your stateless recipe might have found its way into @umagrama’s Nginx configuration? A particular client or control panel integration?

11

@schoen, okay. I will remove it and try again.

Can you please remove my domain from my quote above? Forgot to escape it. Thanks.

12

@schoen, after removing the file entirely, I ran certbot certonly --expand --webroot --webroot-path ..my_path/example.com/public_html/ and got the same error again:

  Failed authorization procedure. www.domain.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: The key authorization file from the server did not match this challenge [PwYlJhNNjUywnfj8FL5iu3TKopXnbKCS6Y7KeWOXZVY.HHFwSGhKzkhDd_OBfs4jkVrn3saF9Hp2kvysceRBrpg] != [PwYlJhNNjUywnfj8FL5iu3TKopXnbKCS6Y7KeWOXZVY.yxgGDRnEH9R5fC2kXvBqXINg_USLfrOwMlTyy10HKwY], domain.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: The key authorization file from the server did not match this challenge [0_7_X0AJ9tvz173SqB09jaR7SZD61eL59hCJuIkFqCM.HHFwSGhKzkhDd_OBfs4jkVrn3saF9Hp2kvysceRBrpg] != [0_7_X0AJ9tvz173SqB09jaR7SZD61eL59hCJuIkFqCM.yxgGDRnEH9R5fC2kXvBqXINg_USLfrOwMlTyy10HKwY]

  IMPORTANT NOTES:
   - The following errors were reported by the server:

     Domain: www.domain.com
     Type:   unauthorized
     Detail: The key authorization file from the server did not match
     this challenge
     [PwYlJhNNjUywnfj8FL5iu3TKopXnbKCS6Y7KeWOXZVY.HHFwSGhKzkhDd_OBfs4jkVrn3saF9Hp2kvysceRBrpg]
     !=
     [PwYlJhNNjUywnfj8FL5iu3TKopXnbKCS6Y7KeWOXZVY.yxgGDRnEH9R5fC2kXvBqXINg_USLfrOwMlTyy10HKwY]

     Domain: domain.com
     Type:   unauthorized
     Detail: The key authorization file from the server did not match
     this challenge
     [0_7_X0AJ9tvz173SqB09jaR7SZD61eL59hCJuIkFqCM.HHFwSGhKzkhDd_OBfs4jkVrn3saF9Hp2kvysceRBrpg]
     !=
     [0_7_X0AJ9tvz173SqB09jaR7SZD61eL59hCJuIkFqCM.yxgGDRnEH9R5fC2kXvBqXINg_USLfrOwMlTyy10HKwY]

     To fix these errors, please make sure that your domain name was
     entered correctly and the DNS A record(s) for that domain
     contain(s) the right IP address.
13

It’s still apparently returning the hard-coded version. Did you restart nginx?

How about running

grep -r yxgGDRn /etc/nginx

to look for any other references to that hard-coded value in your configuration files?

14

Hi @schoen

The stateless mode needs such a conf, but such a conf doesn’t mean using acme.sh

In acme.sh stateless mode, we require the users to add such a conf by his hand, so if this is for acme.sh stateless mode, the user must remember what he had done before.

I doubt that there must be some other plugin in his web control panel, who added the conf.

@umagrama what control panel are you using ?

Personally, I would suggest your give a try on acme.sh, it has almost the same commandline options, but much more lightweighted. and it won’t dirty your system at all.

1 Like
15

Hi @Neilpang, I am using VestaCP. I am considering trying acme.sh… I am out of ideas on what to do and I can’t have my site down another day.

16

good, I also am using Vestacp. acme.sh is well tested on Vestacp.

You can install acme.sh in your server and issue cert by webroot mode, apache mode , or nginx mode.

BTW, why don’t you use the vestacp built-in letsencrypt support ?

17

I have a wrapper here, you may have a try:

1 Like
18

Yaaaaahoooooo! Yes! Yes! Yes! It’s working! I am doing the chicken dance!

So here is what happened… after @Neilpang’s suggestion, I gave acme.sh a shot; when I got to renewing the certificates I received a similar authorization error as previously. I was awestruck. Because the errors were exactly the same, I thought: .htaccess is f***ing with this… and it was! It allowed certbot to create .well-known folder, but .htaccess was limiting it to only that for security reasons. So allz I had to do was to disable .htaccess and renew my certificates. Duh.

Thanks @schoen and @Neilpang on identifying this problem and solving it! I really appreciate it.

Now I think I will create a CRON job and allow access to this particular folder… would that work for future renewals?

PS: @Neilpang, yes, I am using the built-in LetsEncrypt in VestaCP… it is really a great addition to VestaCP!

19

@Neilpang, I just looked at your wrapper. I am interested. What do I do if I already have my certificates issued? Just install vesta.acme.sh and it will take care of the rest?

20

Yes, please just read the readme on github.