Error 403 with HTTP-01 auth on Apache

Hello,

I'm having issues with renewal of my domain certificates. I use HTTP-01 because a TLS verification isn't possible for the Cloudfront domain.

It worked find up until a month or so ago. I don't recall us making any changes on the server that would prevent it working.

Any help would be greatly appreciated. I've searched every topic on the same error, but none of the solutions apply to me. I've checked that ipv6 is disabled.

Thanks in advance.

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: domain.com

I ran this command: sudo /opt/bitnami/letsencrypt/lego --http --email="email@gmail.com" --domains="domain1.com" --domains="www.domain1.com" --domains="d1ys848383dgb.cloudfront.net" --path="/opt/bitnami/letsencrypt" renew --days 90

It produced this output:

2021/06/16 16:01:46 error: one or more domains had a problem: [d1ysdj345333b.cloudfront.net] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: Invalid response from https://d1y343434w5dgb.cloudfront.net/.well-known/acme-challenge/ DNKd-E8ZsOIXqwC_q7TdctQQ_q2Y7HkfAy6C3qoY-TI [13.200.209.2]: "<!DOCTYPE HTML PUBLIC \"-//W3C //DTD HTML 4.01 Transitional//EN\" \"http://www.w3.org/TR/html4/loose.dtd\">\n<HTML><HEAD>< META HTTP-EQ" [domain1.com] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: Invalid resp onse from https://domain1.com/.well-known/acme-challenge/bWa5fAWAGnTXH3qJLMya7LERkn2xXafq JmqLVxvObUs [13.200.202.84]: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.01 Transitional//E N\" \"http://www.w3.org/TR/html4/loose.dtd\">\n<HTML><HEAD><META HTTP-EQ" [www.domain1.com] acme: error: 403 :: urn:ietf:params:acme:error:unauthorized :: Invalid response from https://www.domain1.com/.well-known/acme-challenge/pcSVWBhAsqhdvUrRV4YVQJX8 mSvSSaFI17bSx_ld7ak [13.32.191.122]: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.01 Transit ional//EN\" \"http://www.w3.org/TR/html4/loose.dtd\">\n<HTML><HEAD><META HTTP-EQ"

My web server is (include version): Apache 2.4.43 (Unix)

The operating system my web server runs on is (include version): Debian 10

My hosting provider, if applicable, is: Lightsail (AWS on Bitnami)

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): Using Lego

Is your setup Cloudfront ← Varnish ← Apache?

The lego command you've included will try to temporarily create a webserver on port 80 and respond to the challenge requests there.

If Apache or Varnish is already listening pn port 80, it seems like that should fail. However, since lego is not complaining about the address already being in use, does that mean you are stopping Varnish and/or Apache before you run lego? Or are your servers bound to other ports?

Some extra detail would be useful.

If Apache serves your site from a document root, you could consider running lego in webroot mode, i.e. adding:

--http.webroot /path/to/webroot
1 Like

Hey there,

Thanks for the reply.

Yep, that's the setup. Apache is on 81, Varnish on 80.

My cron job stops the httpd and varnish services before the script commences.

I've now tried manually specifying the webroot, but it doesn't make any difference.

Would it help at all if I posted the entire output of the command?

Cheers,

Well, that should work, as long as Cloudfront is talking to the origin over HTTP (port 80).

Do you by any chance have the "Origin Protocol Policy" in Cloudfront set to "HTTPS Only"?

With this strategy, you would keep Apache and Varnish running, and then run lego with --http and --http.webroot /path/to/webroot.

It should work, as long as your application is willing to serve files from its document root.

A quick way to test this is to drop a file in /path/to/webroot/.well-known/acme-challenge/test.txt and see whether you can download it from https://d1ysdj0uhw5dgb.cloudfront.net/.well-known/acme-challenge/test.txt.

1 Like

Hmm, well there you go.

I had Cloudfront set to 'HTTPS only' and switched it to HTTP, ran the script and it renewed successfully. Thank you so much for that.

Now my question is, is there any way for the script to stay operational with Cloudfront set to HTTPS only? It'll be an inconvenience if I need to disable it each time the script runs for renewal.

The most straightforward way is going to be to use the webroot method.

That way, you can keep "HTTPS only" enabled and you don't have to stop your webservers either.

1 Like

Cool - I believe I've hit a rate limit on certificate renewal.

What I'll do is retry the webroot method and remove the httpd and varnish stop commands from the cron job.

Thank you again for your help - it was such a simple solution!

I'm confused as to how you'd get a .cloudfront.net certificate through Let's Encrypt at all. Wouldn't AWS already be using their own certificate on that name? It looks like there's a CAA record for it too that wouldn't allow for Let's Encrypt certificates.

1 Like

Because I can't control the DNS record for the Cloudfront subdomain, I can't use the DNS challenge. The TLS-ALPN challenge also doesn't work with Cloudfront.

So instead the only way to issue a certificate for the Cloudfront subdomain, along with domain.com and www.domain.com is to use a HTTP challenge.

Please show a cert that contains both domains in the SAN.
Or any cert that ends with .cloudfront.net and was issued by LE.

All I see are:

This is to allow HTTPS Only in the Cloudfront origin protocol policy.

On the custom domains tab on Cloudfront I need to set domain.com and www.domain.com as custom domains.

Then on the server front, I need to issue a SAN certificate with a HTTP challenge for that origin protocol policy to work.

I'm not a security expert, so I have no idea what the technical reasons are behind it. But you can explore the troubleshooting further here: Lightsail CloudFront SSL certificate origin policy issues

Sure.

The CAA record is issuewild; only prohibitive for wildcards. I think AWS fully intended for customers to be able to issue certificates for their distributions. I helped someone some months ago set one up iirc.

2 Likes

Thanks; I was seeing this cert crt.sh | 4716776337 and was getting worried that something wasn't checking CAA right. I didn't realize that issuewild directives without any issue directives meant that anybody could issue non-wildcards, but it makes perfect sense in retrospect.

I still don't quite understand why you'd want the .cloudfront.net name on your cert rather than just using the one built into AWS Cloudfront, though, with a separate cert (without that name) to handle securing the traffic between Cloudfront and your back-end server.

1 Like

I think it might be something like:

  • Cloudfront->Origin is set to HTTPS-only
  • You want to be able to visit https://xxx.cloudfront.net directly
  • Cloudfront will use the outer request's Host as the inner request's SNI name when it connects to the Origin
  • If the Origin's certificate doesn't contain that xxx.cloudfront.net name, Cloudfront will fail the connection to the Origin.
3 Likes

Hmm. I'd assumed that it would use the hostname you used as setting up the origin on the inner request, not the outer one. I've only ever used Cloudfront with S3, though, and that with my own custom hostnames rather than the .cloudfront.net ones. Seems odd to me, but if that's how it works then I guess this approach does make some sense.

1 Like

There's a bit more detail here: Lightsail CloudFront SSL certificate origin policy issues

This is the troubleshooting process we went through when we first encountered the issue.

Lightsail doesn't offer any customisability with the Cloudfront distribution and the only way to offer a HTTPS origin policy is by specifying the Cloudfront domain in the SAN. Otherwise it just throws the error messages in the link above.

I think @_az helped solved that the first time around too.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.