Let's debug says everything is OK, But certbot keeps failing

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:
api.olympsis.com
I ran this command:
sudo certbot --nginx -d api.olympsis.com -v
It produced this output:
Saving debug log to /var/log/letsencrypt/letsencrypt.log

Plugins selected: Authenticator nginx, Installer nginx

Requesting a certificate for api.olympsis.com

Performing the following challenges:

http-01 challenge for api.olympsis.com

Waiting for verification...

Challenge failed for domain api.olympsis.com

http-01 challenge for api.olympsis.com

Certbot failed to authenticate some domains (authenticator: nginx). The Certificate Authority reported these problems:

Domain: api.olympsis.com

Type: unauthorized

Detail: 208.83.226.16: Invalid response from https://api.olympsis.com/.well-known/acme-challenge/nC2e6Vh06aQzDuMcx3Pj8LswbEVBi4uBXvCEYGTMf-c: 404

Hint: The Certificate Authority failed to verify the temporary nginx configuration changes made by Certbot. Ensure the listed domains point to this nginx server and that it is accessible from the internet.

Cleaning up challenges

Some challenges have failed.

Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
My web server is (include version):
nginx version: nginx/1.18.0 (Ubuntu)
The operating system my web server runs on is (include version):
ubuntu 22.04
My hosting provider, if applicable, is:
self hosting. using pfsense.
I can login to a root shell on my machine (yes or no, or I don't know):
yes
I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
no
The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
certbot 1.31.0

The server config:

server {
	listen 80;
	root /var/www/html;
	server_name api.olympsis.com;
        location ~ /.well-known/acme-challenge {
            allow all;
            root /var/www/html/;
        }
}

before I had a proxy pass to my api gateway but I changed it to this config to try to get an ssl cert. Been trying for 3 days T-T, I am pretty sure using pfsense is adding an extra layer of complexity to this problem. Before I changed the config I could access my api from that domain, and I can access the web server from this new config as well. I have no idea what could be going wrong, and why certbot is trying https even though my config only has http.

3

Welcome to the community @zoblod

Yes, that is odd the challenge was redirected to HTTPS. The --nginx plug-in makes temp changes to the nginx conf and it should not be redirected.

But, that server block you show is not the one currently in effect. Because when I try an http request I get redirected to https and there is nothing in your server block to do that.

Can you upload the log file? You might need to copy it to a .txt file first.

/var/log/letsencrypt/letsencrypt.log

Note: because of the --nginx plug-in, you don't need a location for the well-known path.

curl -i http://api.olympsis.com
HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Wed, 19 Oct 2022 16:45:04 GMT
Location: https://api.olympsis.com/
5 Likes
4

log.txt (600.1 KB)

5

Are you using any kind of URL Redirect service in your DNS provider? It looks like you are using GoDaddy and you need to set it so you use an A record for your IP

Or, is your pfSense redirecting the http requests and routing them to https?

The certbot updates look fine in the log but I don't think the http requests are making it to your nginx because of something in front of it (like above items)

5 Likes
6

Screenshot 2022-10-19 at 11.09.28 AM
Screenshot 2022-10-19 at 11.09.28 AM2242×1290 156 KB

I don't think I am using any redirect services. I can look into seeing if pfsense doing that. But I was using postman to access the domain/server before with just http. back then I had a reverse proxy to my api-gateway.

7

DNS forwarder is disabled, and resolver was enabled so I disabled that.

8

Yeah, that DNS screen looks fine. I am still getting redirected from http->https with a 301

Maybe try rebooting or killall nginx and start it fresh

What does this show

sudo systemctl status nginx
5 Likes
9

Screenshot 2022-10-19 at 11.18.47 AM
Screenshot 2022-10-19 at 11.18.47 AM2068×1424 401 KB

10

I'll try the kill all nginx. I've tried rebooting already.

11

You can copy/paste info from your ssh console to here. Then, add 3 backticks before and after to preserve format. Like
```
pasted info
```
Were there any messages at the bottom of that status display?

4 Likes
12

I did not see a "master" process in your nginx status. I'm not sure what that means other than that is not usual. My idle test server looks like this - note the master process and the ExecStart lines. This is an Ubuntu 22 with nginx 1.18. nginx was started with systemd

 sudo systemctl status nginx
● nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2022-10-19 17:20:06 UTC; 9min ago
       Docs: man:nginx(8)
    Process: 10837 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
    Process: 10838 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
   Main PID: 10839 (nginx)
      Tasks: 2 (limit: 1143)
     Memory: 9.2M
        CPU: 43ms
     CGroup: /system.slice/nginx.service
             ├─10839 "nginx: master process /usr/sbin/nginx -g daemon on; master_process on;"
             └─10840 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""

Oct 19 17:20:05 ip-172-31-57-61 systemd[1]: Starting A high performance web server and a reverse proxy server...
Oct 19 17:20:06 ip-172-31-57-61 systemd[1]: Started A high performance web server and a reverse proxy server.
3 Likes
13

Screenshot 2022-10-19 at 11.43.31 AM
nope

14

I don't know what else to say. My curl in post #3 still gets redirected. Yet, your nginx config has no such redirection. Either that is not the running nginx or something in front of it is interfering.

4 Likes
15

Thank you for your help thus far. I'll keep you posted.

2 Likes
16

Can we see?:
nginx -T

3 Likes
17

Screenshot 2022-10-19 at 2.22.45 PM
Screenshot 2022-10-19 at 2.22.45 PM1072×114 13.7 KB

18

That was a capital T. Maybe just upload resulting .txt file

sudo nginx -T >config.txt

The nginx conf is in the log you posted earlier but would be interesting to see if there is any diffs between this and the one certbot sees.

5 Likes
19

output.txt (7.4 KB)

I've been making some changes. I am now getting a 403 on the challenge endpoint looking into the permissions.

20

Hmm. There is still no redirect in your server config yet I get redirected http->https like I always have.

You should use --dry-run for your command until this gets sorted. It uses the Let's Encrypt test system and not affected (much) by failed rate limits

sudo certbot --nginx -d api.olympsis.com -v --dry-run

Can you show this

sudo ss -pant | grep 80

and the top part of a sudo systemctl status nginx again (thru the line for CPU:)

Edit: copy/paste with the 3 backticks like earlier

5 Likes
21 
LISTEN    0      4096                127.0.0.1:9099                 0.0.0.0:*     users:(("calico-node",pid=1858074,fd=9))                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
LISTEN    0      511                   0.0.0.0:80                   0.0.0.0:*     users:(("nginx",pid=1919537,fd=6),("nginx",pid=1919536,fd=6),("nginx",pid=1919535,fd=6),("nginx",pid=1919534,fd=6),("nginx",pid=1919533,fd=6),("nginx",pid=1919532,fd=6),("nginx",pid=1919531,fd=6),("nginx",pid=1919530,fd=6),("nginx",pid=1919529,fd=6),("nginx",pid=1919528,fd=6),("nginx",pid=1919527,fd=6),("nginx",pid=1919526,fd=6),("nginx",pid=1919525,fd=6),("nginx",pid=1919524,fd=6),("nginx",pid=1919523,fd=6),("nginx",pid=1919522,fd=6),("nginx",pid=1919521,fd=6),("nginx",pid=1919520,fd=6),("nginx",pid=1919519,fd=6),("nginx",pid=1919518,fd=6),("nginx",pid=1919517,fd=6),("nginx",pid=1919516,fd=6),("nginx",pid=1919515,fd=6),("nginx",pid=1919514,fd=6),("nginx",pid=1919513,fd=6),("nginx",pid=1919512,fd=6),("nginx",pid=1919511,fd=6),("nginx",pid=1919510,fd=6),("nginx",pid=1919509,fd=6),("nginx",pid=1919508,fd=6),("nginx",pid=1919507,fd=6),("nginx",pid=1919506,fd=6),("nginx",pid=1919505,fd=6),("nginx",pid=1919504,fd=6),("nginx",pid=1919503,fd=6),("nginx",pid=1919502,fd=6),("nginx",pid=1919501,fd=6),("nginx",pid=1919500,fd=6),("nginx",pid=1919499,fd=6),("nginx",pid=1919498,fd=6),("nginx",pid=1914380,fd=6))
ESTAB     0      0               192.168.1.124:54846           10.152.183.1:443   users:(("calico-node",pid=1858074,fd=10))                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
TIME-WAIT 0      0               192.168.1.124:48078          192.168.1.124:7472                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
ESTAB     0      0               192.168.1.124:22               192.168.2.2:44634 users:(("sshd",pid=1804380,fd=4),("sshd",pid=1804123,fd=4))                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
TIME-WAIT 0      0               192.168.1.123:53946            10.1.219.80:7472                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
TIME-WAIT 0      0               192.168.1.123:58674           10.1.219.103:8084                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
TIME-WAIT 0      0               192.168.1.123:48858           10.1.219.111:8082                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
TIME-WAIT 0      0               192.168.1.123:34242            10.1.219.80:7472                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
ESTAB     0      0               192.168.1.124:54844           10.152.183.1:443   users:(("calico-node",pid=1858075,fd=9))                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
TIME-WAIT 0      0               192.168.1.123:49592           10.1.219.111:8082                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
TIME-WAIT 0      0               192.168.1.123:34238            10.1.219.80:7472                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
TIME-WAIT 0      0               192.168.1.123:50178            10.1.219.80:7472                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
TIME-WAIT 0      0               192.168.1.123:42978           10.1.219.103:8084                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
TIME-WAIT 0      0               192.168.1.124:7472           192.168.1.124:48078                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
ESTAB     0      0               192.168.1.124:22               192.168.2.2:44636 users:(("sshd",pid=1804614,fd=4),("sshd",pid=1804418,fd=4))                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
ESTAB     0      0               192.168.1.124:54852           10.152.183.1:443   users:(("calico-node",pid=1858072,fd=10))                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
ESTAB     0      0               192.168.1.124:54804           10.152.183.1:443   users:(("speaker",pid=1857748,fd=10))                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
TIME-WAIT 0      0               192.168.1.123:34906           10.1.219.103:8084                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
TIME-WAIT 0      0               192.168.1.123:40440           10.1.219.103:8084                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
TIME-WAIT 0      0               192.168.1.123:38474           10.1.219.111:8082                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
TIME-WAIT 0      0               192.168.1.124:7472           192.168.1.124:48080                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
TIME-WAIT 0      0               192.168.1.123:57226            10.1.219.77:8080                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
LISTEN    0      511                      [::]:80                      [::]:*     users:(("nginx",pid=1919537,fd=7),("nginx",pid=1919536,fd=7),("nginx",pid=1919535,fd=7),("nginx",pid=1919534,fd=7),("nginx",pid=1919533,fd=7),("nginx",pid=1919532,fd=7),("nginx",pid=1919531,fd=7),("nginx",pid=1919530,fd=7),("nginx",pid=1919529,fd=7),("nginx",pid=1919528,fd=7),("nginx",pid=1919527,fd=7),("nginx",pid=1919526,fd=7),("nginx",pid=1919525,fd=7),("nginx",pid=1919524,fd=7),("nginx",pid=1919523,fd=7),("nginx",pid=1919522,fd=7),("nginx",pid=1919521,fd=7),("nginx",pid=1919520,fd=7),("nginx",pid=1919519,fd=7),("nginx",pid=1919518,fd=7),("nginx",pid=1919517,fd=7),("nginx",pid=1919516,fd=7),("nginx",pid=1919515,fd=7),("nginx",pid=1919514,fd=7),("nginx",pid=1919513,fd=7),("nginx",pid=1919512,fd=7),("nginx",pid=1919511,fd=7),("nginx",pid=1919510,fd=7),("nginx",pid=1919509,fd=7),("nginx",pid=1919508,fd=7),("nginx",pid=1919507,fd=7),("nginx",pid=1919506,fd=7),("nginx",pid=1919505,fd=7),("nginx",pid=1919504,fd=7),("nginx",pid=1919503,fd=7),("nginx",pid=1919502,fd=7),("nginx",pid=1919501,fd=7),("nginx",pid=1919500,fd=7),("nginx",pid=1919499,fd=7),("nginx",pid=1919498,fd=7),("nginx",pid=1914380,fd=7))
ESTAB     0      0      [::ffff:192.168.1.124]:16443   [::ffff:10.1.219.80]:39300 users:(("kubelite",pid=1852229,fd=103))

changed my server block to this, its serving the html file at least:

server {
	listen 80 default_server;
	listen [::]:80 default_server;
	index index.html;
	root /var/www/api.omlypsis.com/html;
	server_name api.olympsis.com;
}