Certbot failed to authenticate some domains, error renewing certificate

My domain is: dhis2poc.knowtechture.com

I ran this command: sudo certbot renew --dry-run

It produced this output:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/dhis2poc.knowtechture.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Simulating renewal of an existing certificate for dhis2poc.knowtechture.com and www.dhis2poc.knowtechture.com

Certbot failed to authenticate some domains (authenticator: nginx). The Certificate Authority reported these problems:
  Domain: dhis2poc.knowtechture.com
  Type:   unauthorized
  Detail: 108.157.150.129: Invalid response from https://dhis2poc.knowtechture.com/dhis-web-commons/security/login.action: "<!DOCTYPE HTML>\n<html class=\"loginPage\" dir=\"ltr\">\n<head>\n    <title>DHIS 2</title>\n    <meta name=\"description\" content=\"DHIS 2"

  Domain: www.dhis2poc.knowtechture.com
  Type:   unauthorized
  Detail: 108.157.150.29: Invalid response from https://www.dhis2poc.knowtechture.com/dhis-web-commons/security/login.action: "<!DOCTYPE HTML>\n<html class=\"loginPage\" dir=\"ltr\">\n<head>\n    <title>DHIS 2</title>\n    <meta name=\"description\" content=\"DHIS 2"

Hint: The Certificate Authority failed to verify the temporary nginx configuration changes made by Certbot. Ensure the listed domains point to this nginx server and that it is accessible from the internet.

Failed to renew certificate dhis2poc.knowtechture.com with error: Some challenges have failed.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
All simulated renewals failed. The following certificates could not be renewed:
  /etc/letsencrypt/live/dhis2poc.knowtechture.com/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

My web server is (include version): Nginx 1.18.0

The operating system my web server runs on is (include version): Ubuntu 22.04

My hosting provider, if applicable, is: It's an EC2 machine on AWS

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 2.11.0

Hi everyone, I'm currently trying to test the automatic renew of a certificate I generated for my site, but I'm having problems trying to understand what's wrong.
For more context, my site is behind a CDN (AWS CloudFront) but this was added after I generated the certificate using Certbot, now that I'm attempting to test the renewal it's giving me this error.

Any help or suggestions about where to start tackling this error will be greatly appreciated. Many thanks!

Nginx configuration:

user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
        worker_connections 768;
        # multi_accept on;
}

http {
  gzip on; # Enables compression, incl Web API content-types
  gzip_types
    "application/json;charset=utf-8" application/json
    "application/javascript;charset=utf-8" application/javascript text/javascript
    "application/xml;charset=utf-8" application/xml text/xml
    "text/css;charset=utf-8" text/css
    "text/plain;charset=utf-8" text/plain;

  # HTTP server - rewrite to force use of SSL

  server {
    listen     80;
    #server_name        dhis2poc.knowtechture.com;
    rewrite    ^ https://dhis2poc.knowtechture.com$request_uri? permanent;
  }

 # HTTPS server

  server {
    listen               443 ssl;
    #server_name          dhis2poc.knowtechture.com;
    client_max_body_size 10M;

    ssl                  on;
    ssl_certificate      /etc/letsencrypt/live/dhis2poc.knowtechture.com/fullchain.pem;
    ssl_certificate_key  /etc/letsencrypt/live/dhis2poc.knowtechture.com/privkey.pem;

    ssl_session_cache    shared:SSL:20m;
    ssl_session_timeout  10m;

    ssl_protocols              TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers                RC4:HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers  on;

    # Proxy pass to servlet container

    location / {
      proxy_pass                http://localhost:8080/;
      proxy_redirect            off;
      proxy_set_header          Host               $host;
      proxy_set_header          X-Real-IP          $remote_addr;
      proxy_set_header          X-Forwarded-For    $proxy_add_x_forwarded_for;
      proxy_set_header          X-Forwarded-Proto  https;
      proxy_buffer_size         128k;
      proxy_buffers             8 128k;
      proxy_busy_buffers_size   256k;
      proxy_cookie_path         ~*^/(.*) "/$1; SameSite=Lax";
    }

    location /glowroot {
      proxy_pass    http://localhost:4000/glowroot;
    }

  }
}

Hello @vmendoza, welcome to the Let's Encrypt community. :slightly_smiling_face:

I guess this is mostly just supplemental information.

This is what I see for the ACME HTTP-01 Challenge:

$ curl -Ii http://dhis2poc.knowtechture.com/.well-known/acme-challenge/sometestfile
HTTP/1.1 301 Moved Permanently
Server: CloudFront
Date: Tue, 02 Jul 2024 17:10:20 GMT
Content-Type: text/html
Content-Length: 167
Connection: keep-alive
Location: https://dhis2poc.knowtechture.com/.well-known/acme-challenge/sometestfile
X-Cache: Redirect from cloudfront
Via: 1.1 182d3a3dbb6658c964ee75cd45a42242.cloudfront.net (CloudFront)
X-Amz-Cf-Pop: HIO52-P2
X-Amz-Cf-Id: WgxQZ5vooeHaGN5VA3Q91f5PMY6SOPN6JgJDieou3PxCrly38-65oA==

And following the redirect

$ curl -Ii https://dhis2poc.knowtechture.com/.well-known/acme-challenge/sometestfile
HTTP/2 302
location: https://dhis2poc.knowtechture.com/dhis-web-commons/security/login.action
server: nginx/1.18.0 (Ubuntu)
date: Tue, 02 Jul 2024 17:10:51 GMT
set-cookie: JSESSIONID=6F2C65023B1F3ADC1F5848A63C817C45; Path=/; SameSite=Lax; HttpOnly
content-security-policy: frame-ancestors 'self';
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
x-cache: Miss from cloudfront
via: 1.1 4894bef31db1c311602a51393339af0a.cloudfront.net (CloudFront)
x-amz-cf-pop: HIO52-P2
x-amz-cf-id: dEslQwjnvkbcN250LXz3Q2guDBjaJ7aIi0oDldcEYg-etdBsnAMJEg==

The server changes from CloudFront to nginx/1.18.0 (Ubuntu)

Edit:

And now the significant piece of information

The final redirect goes to
https://dhis2poc.knowtechture.com/dhis-web-commons/security/login.action
which doesn't really seem what is wanted.

$ curl -Ii https://dhis2poc.knowtechture.com/dhis-web-commons/security/login.action
HTTP/2 200
content-type: text/html;charset=UTF-8
server: nginx/1.18.0 (Ubuntu)
date: Tue, 02 Jul 2024 17:14:00 GMT
set-cookie: JSESSIONID=F0624091A9C98EBA2DBAA8266582B193; Path=/; SameSite=Lax; HttpOnly
content-security-policy: script-src 'self' 'nonce-V3bIb2ob0sC8MS_6_FUBkL60JGETWT82';
content-security-policy: frame-ancestors 'none';
login-page: true
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
content-language: en
x-cache: Miss from cloudfront
via: 1.1 6a52d37737133b0b8a09947e5c586ec4.cloudfront.net (CloudFront)
x-amz-cf-pop: HIO52-P2
x-amz-cf-id: 4XOW5NIJcBHzKVLe2Z3PUrMgkVdjN9WBTxbK3oGYlCUxBfT_3dBEfg==

and can also be see here https://letsdebug.net/dhis2poc.knowtechture.com/2079360?debug=y

3 Likes

Hi @Bruce5051 Thank you for your help,

So if I understand correctly, the problem is that when the certification process is trying to access the ACME token, it is instead being redirected to the login page of my website? I should try to stop that redirect? Perhaps It's because of the rewrite parameter in the nginx configuration...

2 Likes

Correct.

Yes.

3 Likes

Hi again, I managed to stop the redirect adding the following to my Nginx config file

location /.well-known/acme-challenge/ {
      allow all;
}

However, now I'm facing a 404 error. Here's the certbot renew --dry-run output:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/dhis2poc.knowtechture.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Simulating renewal of an existing certificate for dhis2poc.knowtechture.com and www.dhis2poc.knowtechture.com

Certbot failed to authenticate some domains (authenticator: nginx). The Certificate Authority reported these problems:
  Domain: dhis2poc.knowtechture.com
  Type:   unauthorized
  Detail: 18.238.136.117: Invalid response from https://dhis2poc.knowtechture.com/.well-known/acme-challenge/-QWLYPVlayMxGgD3NdjMjN_mExI7hnZYFgXyXAfZsr8: 404

  Domain: www.dhis2poc.knowtechture.com
  Type:   unauthorized
  Detail: 18.238.136.120: Invalid response from https://www.dhis2poc.knowtechture.com/.well-known/acme-challenge/3fLSscqopmz0pmCWiDqxAcY5U4m1gjbS_SNSfsjq-84: 404

Hint: The Certificate Authority failed to verify the temporary nginx configuration changes made by Certbot. Ensure the listed domains point to this nginx server and that it is accessible from the internet.

Failed to renew certificate dhis2poc.knowtechture.com with error: Some challenges have failed.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
All simulated renewals failed. The following certificates could not be renewed:
  /etc/letsencrypt/live/dhis2poc.knowtechture.com/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)

Looking at the page Bruce shared (Let's Debug), I see now it's correctly redirecting to 'https://dhis2poc.knowtechture.com/.well-known/acme-challenge/letsdebug-test' but at the end I'm getting Server response: HTTP 404 Not Found

How is your Origin setup in CloudFront? Specifically, what is in Origin domain and what is/are your choice(s) of Protocol

3 Likes

Hi @MikeMcQ

In case image is unavailable my origin is the EC2 dns provided by Amazon and as protocol I'm using HTTPS only. Also I have a behavior for Viewer protocol policy to redirect HTTP to HTTPS.

Yeah, I don't think that will work well with Certbot and --nginx authenticator. You might have better luck with --webroot but below I suggest what may be easier and more versatile option.

The --nginx auth sets up for responses coming in for the domain in the cert. But, you are coming in as a different domain name now (the name in the Origin). So, your nginx processes that within its default server block. We can check but I'm guessing that isn't the one you already have. If it was it would have worked :slight_smile:

I think you can do clever things like setting a fixed HOST header in a Behavior for the ACME Challenge. But, I am not certain and the AWS docs are not great for that.

What I did is setup an EC2 server with its own DNS name and set CloudFront Origin to that name. That way you can connect directly to your EC2 using its own name or through a name CloudFront controls. Connecting with a CloudFront name gets its caching and other benefits.

You run Certbot on nginx like before just using this new name.

You can set the Protocol policy to HTTPS-only once you get that cert and test the HTTPS connect to that EC2 domain separately.

For example,

SETUP EC2
Setup an A record for your EC2 domain to point to its public IP
Let's call it backend.example.com (use your domain instead of example.com)
Change nginx server_name to this domain name for both port 80 and 443
Use Certbot to get cert for backend.example.com
This is just like what you did before but with a different name
It just avoids having to pass HTTP(S) requests through CloudFront
Test this from any browser using https://backend.example.com

SET CLOUDFRONT
Change Origin name to backend.example.com
Set Protocol to HTTPS-Only
Set other options as you prefer

Requests to CloudFront work the same as before. That is, requests to the DNS names that point to the CloudFront endpoint still connect to that first. CF makes a new connect, if/as needed, to your (newly named) backend server. Often, the names like example.com or www.example.com point to CloudFront.

Advanced tip: I also set a custom-header that gets passed to nginx from CloudFront. I can check for that in nginx and process those differently. Such as if I only want to process certain requests that come in directly to my backend server (bypassing CloudFront). Like requests I use specifically for admin purposes (like certs) or unusually long-running tasks that timeout if coming in through CloudFront

3 Likes

Thank you so much for the detailed response @MikeMcQ, before trying your approach I tried one more thing, and it actually worked!

Previously, I had configured my EC2 inbound rules to only listen to Cloudfront edge locations over port 443. But reading this documentation page (Challenge Types - Let's Encrypt) at one point it says:

The HTTP-01 challenge can only be done on port 80

So I tried opening port 80 and after that, the renewal succeeded! I don't know if I'm comfortable leaving port 80 open to everyone but seems like it's necessary for the automatic renewal to work.

Also, quoting from your previous response:

But, you are coming in as a different domain name now (the name in the Origin).

For Cloudfront I'm using an Alternate domain name which matches the domain I'm trying to generate the certificate for, so having the EC2 dns as origin should not be a problem, I think.
image

Thanks to everyone for the support.

1 Like

Some parts of that explanation don't quite follow. But, glad it's working.

An HTTP Challenge has the LE Server sending an HTTP request to the IP named in the DNS. In this case that DNS name points to CloudFront. And, you had HTTPS-Only set in your Protocol Policy. CF terminates the HTTP connection and forwards the request to your Origin as HTTPS. So, your EC2 instance never should see HTTP requests. I don't know why changing your Security Group rule should have mattered.

Often, CF would be set to redirect HTTP requests back to the user-agent as HTTPS. And, you have this setup. Which is another puzzle why opening port 80 for EC2 made any difference.

Note the LE support for an HTTP Challenge will follow such redirects (up to 10 deep).

As for the Alternate Domain Names, that is just how you tell CloudFront which names to manage its own cert for. If you look, you'll see the cert used by your two names are from Amazon itself. The Let's Encrypt cert is used for the connection between CF and your Origin

Your CloudFront edge redirecting HTTP requests to HTTPS back to the user-agent

curl -i http://www.dhis2poc.knowtechture.com/.well-known/acme-challenge/Test404
HTTP/1.1 301 Moved Permanently
Server: CloudFront
Location: https://www.dhis2poc.knowtechture.com/.well-known/acme-challenge/Test404
X-Cache: Redirect from cloudfront
3 Likes

Yes @MikeMcQ you are correct. I forgot to mention that I also had to change the CloudFront configuration to allow HTTP traffic and not redirect HTTP to HTTPS. Then I added the inbound rule on the EC2 to open port 80.

After I successfully renewed the certificate, I changed everything to how it was originally while I investigate the implications of these changes. I apologize for the confusion.

Thanks again for all the help @Bruce5051 @MikeMcQ I think I've learned a lot about how this process works. :slight_smile:

2 Likes

While you experiment you could use

sudo certbot renew --dry-run

for testing the auto-renewal

To test getting a cert with --nginx option you can use

sudo certbot certonly --nginx --dry-run -d example.com -d www.example.com 

The --nginx option does not support --dry-run on its own which is why you need to add certonly to test that. This tests only cert acquisition not cert installation.

I still think you'll be better off trying to follow that menu recipe I described earlier. But, the above two commands are helpful for testing any such Certbot setup

Keep in mind that CloudFront can have multiple Origin servers. Then, your Behaviors can send certain URI to specific ones. There is no reason for your Origin domain name to be related to the name(s) managed by CloudFront itself. One common example is an EC2 Origin for dynamic info and S3 for static.

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.