Certbot --nginx fails despite my domain hitting the nginx server

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: lucivia.net

I ran this command: sudo certbot --nginx

It produced this output:
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator nginx, Installer nginx
Please enter the domain name(s) you would like on your certificate (comma and/or
space separated) (Enter 'c' to cancel): lucivia.net
Requesting a certificate for lucivia.net
Performing the following challenges:
http-01 challenge for lucivia.net
Waiting for verification...
Challenge failed for domain lucivia.net
http-01 challenge for lucivia.net

Certbot failed to authenticate some domains (authenticator: nginx). The Certificate Authority reported these problems:
Domain: lucivia.net
Type: unauthorized
Detail: Invalid response from http://lucivia.net/.well-known/acme-challenge/XQqOVzsK4iAaSLmmJto_8XKvocsyyppQFCk9Qwnd1_A [143.244.210.50]: "\r\n404 Not Found\r\n\r\n

404 Not Found

\r\n
nginx\r\n"

Hint: The Certificate Authority failed to verify the temporary nginx configuration changes made by Certbot. Ensure the listed domains point to this nginx server and that it is accessible from the internet.

My web server is (include version): nginx v1.19.9

The operating system my web server runs on is (include version): ubuntu v20

My hosting provider, if applicable, is: DigitalOcean

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): 1.22.0

Background

I'm running Kube inside a DigitalOcean droplet (the cluster was instantiated using kind). I can hit http:lucivia.net with a browser to reach the nginx server. This is why I'm a confused about the error that is stating that certbot cannot reach the domain. Do I have to have https temporarily working?

Thank you in advance.

- E

3 Likes

Welcome to the Let's Encrypt Community, Edmund :slightly_smiling_face:

My first concern is about the multiple IP addresses (DNS A records) that produce different responses:

lucivia.net. 1464 IN A 159.203.147.43
lucivia.net. 1464 IN A 143.244.210.50
lucivia.net. 1464 IN A 159.203.99.5

My second concern is about the 503 failure:

http://lucivia.net/.well-known/acme-challenge/test
307 Temporary Redirect
https://lucivia.net/.well-known/acme-challenge/test
503 Service Unavailable

https://www.redirect-checker.org/index.php

5 Likes

Great observations. The .43 is pointing to a load-balancer that I just checked on. It's down. I also have a couple of ip addresses for the domain. I'll update the record and try again.

(Question to self: What is the purpose of DigitalOcean load-balancer outside of the droplet?)

3 Likes

Are you running certbot on the load-balancer or each worker? The latter will cause you issues.

Better yet, who is terminating SSL (e.g. serving the certificate and en/decrypting the traffic), the LB or the workers?

3 Likes

Good question. I understand that I need to set-up the certificate with the load-balancer. My current confusion is likely out-of-scope here. My understanding is that I need to point the dns record to the node that is running the ingress-nginx service. I'm in the process of figuring out what that ip is.

This does not explain the why I also have an external load-balancer. It may be "a side-effect" of exposing the service hosted in the kinD cluster..?

It should be the ingress-nginx instance. Yes?

3 Likes

That depends upon how you have your world configured. While it simplifies your configuration to have a frontend take care of the SSL termination, it doesn't scale well if you have many backend servers because the frontend would need to handle the en/decryption for all of them. On the other hand, having each backend server maintain its own certificate is wasteful and error-prone in the acquisition and renewal processes. Typically the best solution is to have the frontend manage the certificate and push the new certificate and private key to the backend servers so that they can handle SSL termination themselves. This also helps if the frontend and backend have unsecured network between them. Hopefully this makes sense.

3 Likes

This is helpful. I suspect what you are proposing is the best way to go. The DigitalOcean (DO) load-balancer is likely going to serve as the frontend referred to here. It routes requests to a droplet with the most available resource. Each droplet should be isolated from each other; so, each gets a private key to handle the termination. Does that logic seem right?

3 Likes

Each gets a copy of the certificate and its matching private key. Yep.

3 Likes

I cleaned-up the DNS records. The domain points to the load-balancer (lb). When I hit the http://lucivia.net endpoint I get the page I expect served-up by nginx (note: I have the host forward non-secure requests to a secure host). However, I get a similar error when I run sudo certbot --nginx as I did before.

As a starting point, I have the lb process the certificate (i.e., terminal, not a passthrough).

The relevant url: Lucivia - Etl for analytics

What page is the above requesting to validate the domain?

3 Likes

I may not understand your intention but your domain seems to respond fine.

I see requests to https://lucivia.net and the www domain return a current Let's Encrypt cert chain created today. This certificate covers both the apex and www names.

Presumably that is your LB responding as you have described your current setup. Seems fine. Your cert history also looks normal.

Are you trying to use https for your connection between your LB and your backend servers?

If so, based on the Digital Ocean docs you either just use http between them or a VPC. I don't have personal experience with Digital Ocean. Just using these docs:

5 Likes

Thank you @MikeMcQ -- Based on your inspection and my experience also, it seems to be working. I guess from a learning perspective, I'm not sure why the sudo certbot --nginx was not able to confirm/generate what I needed, or tell me that I'm otherwise "good to go". Do you know why?

In the end, it was DO that "automagically" set-up the certificate for me using certbot (I believe)... of course once I fixed the dns records given what @griffin observed earlier.

5 Likes

The challenge is sent to

http://lucivia.net/.well-known/acme-challenge/test

But the reply is trying to be sent by

https://lucivia.net/.well-known/acme-challenge/test

Certbot sends the challenge request to http, but the reply was coming from https, not from where Certbot is listening (http on port 80). They're speaking two different languages.

Glad to see you got your cert okay after fixing your dns records. :+1:

4 Likes

Ah. So if I turned off the forwarding http -> https to ensure users only use a secure connection, then the certbot would have likely worked. That makes sense. Thank you.

3 Likes

The Let's Encrypt validation server follows redirects, so that shouldn't matter to be honest...

3 Likes

Disabling redirections should not be necessary since Certbot 1.13:

  • The nginx authenticator now configures all matching HTTP and HTTPS vhosts for the HTTP-01 challenge. It is now compatible with external HTTPS redirection by a CDN or load balancer.
4 Likes

My bad. Completely forgot. :man_facepalming:

3 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.