Let's debug says everything is OK, But certbot keeps failing

MikeMcQ · October 19, 2022, 9:32pm

Pls show sudo systemctl status nginx too (just top part thru CPU:)

And, also show this. Maybe it's the simplest explanation of all

curl -4 http://ifconfig.co

zoblod · October 19, 2022, 9:33pm

● nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2022-10-19 15:25:41 MDT; 7min ago
       Docs: man:nginx(8)
    Process: 1914378 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
    Process: 1914379 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
   Main PID: 1914380 (nginx)
      Tasks: 41 (limit: 77090)
     Memory: 31.6M
        CPU: 652ms
     CGroup: /system.slice/nginx.service
             ├─1914380 "nginx: master process /usr/sbin/nginx -g daemon on; master_process on;"
             ├─1919498 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             ├─1919499 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             ├─1919500 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             ├─1919501 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             ├─1919502 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             ├─1919503 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             ├─1919504 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             ├─1919505 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             ├─1919506 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             ├─1919507 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             ├─1919508 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             ├─1919509 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             ├─1919510 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             ├─1919511 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             ├─1919512 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             ├─1919513 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             ├─1919514 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             ├─1919515 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             ├─1919516 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             ├─1919517 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             ├─1919518 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             ├─1919519 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             ├─1919520 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             ├─1919521 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             ├─1919522 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             ├─1919523 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             ├─1919524 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             ├─1919525 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             ├─1919526 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             ├─1919527 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             ├─1919528 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             ├─1919529 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             ├─1919530 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             ├─1919531 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             ├─1919532 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             ├─1919533 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             ├─1919534 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             ├─1919535 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             ├─1919536 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""
             └─1919537 "nginx: worker process" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" "" ""

Oct 19 15:25:41 master systemd[1]: Starting A high performance web server and a reverse proxy server...
Oct 19 15:25:41 master systemd[1]: Started A high performance web server and a reverse proxy server.

the second command returns my public ip:

208.83.226.16

_az · October 19, 2022, 9:43pm

If you run this on the server itself, what's the output?

curl -I --resolve api.olympsis.com:80:127.0.0.1 http://api.olympsis.com/

To me it seems like you've got the port forwarded to the wrong server.

MikeMcQ · October 19, 2022, 9:49pm

I remain somewhat baffled. The main pid for nginx shows as listening to port 80 (that's good).
The public IP from ifconfig.co matches your DNS A record

The cert I see on your https port is a pfSense self-signed cert.

I can maybe explain the 404. The HTTP challenge from the Let's Encrypt server is being redirected to https (by pfSense). The HTTPS request is then rejected by pfSense with a 404. I am not certain it is pfSense but fairly sure. An https request for your home page gets a pfSense rejected due to "Potential DNS Rebind attack" (http status 200 though for home page)

I still think pfSense is doing the redirect. It could also be a port forwarding issue like _az suggests but I think pfSense is more likely

curl -iL api.olympsis.com
HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Wed, 19 Oct 2022 21:33:28 GMT
Location: https://api.olympsis.com/

openssl s_client -connect api.olympsis.com:443
---
Certificate chain
 0 s:O = pfSense webConfigurator Self-Signed Certificate, CN = pfSense-6328ea2d885f4
   i:O = pfSense webConfigurator Self-Signed Certificate, CN = pfSense-6328ea2d885f4
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Sep 19 22:16:13 2022 GMT; NotAfter: Oct 22 22:16:13 2023 GMT

_az · October 19, 2022, 9:50pm

Sometimes with port forwarding, you have to relocate your modem/router's web interface off port 80/443 to other ports, otherwise it causes your port forwarding to be ineffective.

zoblod · October 19, 2022, 9:50pm

HTTP/1.1 200 OK
Server: nginx/1.18.0 (Ubuntu)
Date: Wed, 19 Oct 2022 21:50:34 GMT
Content-Type: text/html
Content-Length: 177
Last-Modified: Wed, 19 Oct 2022 19:29:34 GMT
Connection: keep-alive
ETag: "6350501e-b1"
Accept-Ranges: bytes

zoblod · October 19, 2022, 9:52pm

oh! I am trying to understand why pfsense would deny it even after doing a split dns override.

MikeMcQ · October 19, 2022, 9:53pm

Ding ding ding. Note the "Server" header for my curl redirect is just "nginx" not with the version number.

Again points to pfSense (I think it's based on nginx)

zoblod · October 19, 2022, 9:54pm

ohhh ill try moving the web interface for pfsense off 80/443 like @_az suggests.

zoblod · October 19, 2022, 10:36pm

I guess I'll go to the pfsense forum now lol. thanks for the help. for some reason its redirecting it to my gui.

Plugins selected: Authenticator nginx, Installer nginx
Requesting a certificate for api.olympsis.com
Performing the following challenges:
http-01 challenge for api.olympsis.com
Waiting for verification...
Challenge failed for domain api.olympsis.com
http-01 challenge for api.olympsis.com

Certbot failed to authenticate some domains (authenticator: nginx). The Certificate Authority reported these problems:
  Domain: api.olympsis.com
  Type:   connection
  Detail: 208.83.226.16: Fetching https://api.olympsis.com:6001/.well-known/acme-challenge/tojcxC_Cvz8AU3U1I_mVe_j2jeMRdusrAbG72NZ2XrU: Invalid port in redirect target. Only ports 80 and 443 are supported, not 6001

Hint: The Certificate Authority failed to verify the temporary nginx configuration changes made by Certbot. Ensure the listed domains point to this nginx server and that it is accessible from the internet.

Cleaning up challenges
Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

gui is now in port 6001

_az · October 19, 2022, 10:55pm

Have you tried disabling the WebGUI redirect?

zoblod · October 20, 2022, 2:17am

that fixed the redirecting to https issue.

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator nginx, Installer nginx
Requesting a certificate for api.olympsis.com
Performing the following challenges:
http-01 challenge for api.olympsis.com
Waiting for verification...
Challenge failed for domain api.olympsis.com
http-01 challenge for api.olympsis.com

Certbot failed to authenticate some domains (authenticator: nginx). The Certificate Authority reported these problems:
  Domain: api.olympsis.com
  Type:   connection
  Detail: 208.83.226.16: Fetching http://api.olympsis.com/.well-known/acme-challenge/37HmfcSd9kkb-m1ZUz6MeIa83ZpPAEZF5nXN32gmrw0: Timeout during connect (likely firewall problem)

Hint: The Certificate Authority failed to verify the temporary nginx configuration changes made by Certbot. Ensure the listed domains point to this nginx server and that it is accessible from the internet.

Cleaning up challenges
Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

thanks guys.

zoblod · October 20, 2022, 2:38am

Plugins selected: Authenticator nginx, Installer nginx
Requesting a certificate for api.olympsis.com
Performing the following challenges:
http-01 challenge for api.olympsis.com
Waiting for verification...
Cleaning up challenges

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/api.olympsis.com/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/api.olympsis.com/privkey.pem
This certificate expires on 2023-01-18.
These files will be updated when the certificate renews.
Certbot has set up a scheduled task to automatically renew this certificate in the background.

Deploying certificate
Deploying Certificate to VirtualHost /etc/nginx/sites-enabled/api.olympsis.com
Successfully deployed certificate for api.olympsis.com to /etc/nginx/sites-enabled/api.olympsis.com
Redirecting all traffic on port 80 to ssl in /etc/nginx/sites-enabled/api.olympsis.com
Congratulations! You have successfully enabled HTTPS on https://api.olympsis.com
Subscribe to the EFF mailing list (email: jjoseph@coronislabs.com).

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
If you like Certbot, please consider supporting our work by:
 * Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
 * Donating to EFF:                    https://eff.org/donate-le
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

I fixed it!!!!
Let's recap. The issue was narrowed down to pfsense forcing all traffic going to my ip to the web GUI. After moving the web GUI to another port and checking to disable web GUI redirect, I did a split dns override entry for my domain name on the pfsense DNS resolver which pointed to my server. I did a port forward pointing my server as well. After I did those things I was able to make a get request to my website using postman, after that I ran certbot again and it worked.

Again thank you guys so much for the help.

system · November 19, 2022, 2:39am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Certbot SSL Certification Failing Help	6	85	December 23, 2024
Challenge failed Help	19	621	July 28, 2023
Certbot failed to authenticate some domains ; It works for most of my domains but few of them not working Help	5	717	July 17, 2022
Certbot failed to authenticate some domains Help	2	323	July 22, 2023
Certbot failed to authenticate some domains Help	4	484	May 5, 2022

Let's debug says everything is OK, But certbot keeps failing

Related topics