Certbot keeps trying to get Certificate using https

My domain is: proxim.khas.dev

I ran this command:

certbot --nginx -d proxim.khas.dev

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Requesting a certificate for proxim.khas.dev

Certbot failed to authenticate some domains (authenticator: nginx). The Certificate Authority reported these problems:
  Domain: proxim.khas.dev
  Type:   unauthorized
  Detail: 95.217.226.251: Invalid response from https://proxim.khas.dev/?url=/.well-known/acme-challenge/ofFBbu-c-GqCUNKGuiMNCn-hamnx4TiikQN4B9piH3M: "<!doctype html>\n<!--[if IE 8 ]><html lang=\"en\" class=\"ie ie8 lte9 lte8 no-js\"><![endif]-->\n<!--[if IE 9 ]><html lang=\"en\" class="

Hint: The Certificate Authority failed to verify the temporary nginx configuration changes made by Certbot. Ensure the listed domains point to this nginx server and that it is accessible from the internet.

Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

My web server is (include version):

nginx version: nginx/1.18.0

The operating system my web server runs on is (include version):

Debian GNU/Linux 11 (bullseye)

I can login to a root shell on my machine (yes or no, or I don't know): yes

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

certbot 1.28.0

My Nginx configuration:
Default

# cat /etc/nginx/sites-available/default
server {
        listen 80 default_server;
        listen [::]:80 default_server;

        root /var/www/html;

        index index.html index.htm index.nginx-debian.html;

        server_name _;

        location / {
                # First attempt to serve request as file, then
                # as directory, then fall back to displaying a 404.
                try_files $uri $uri/ =404;
        }
}

Proxim

cat /etc/nginx/sites-available/proxim.khas.dev
server {
    listen 80;
    listen [::]:80;

    root /var/www/proxim/html;
    index index.html index.htm index.nginx-debian.html;

    server_name proxim.khas.dev;

    location ~ /.well-known {
      root "/var/www/html";
      allow all;
    }

    location / {
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_pass https://localhost:8006;
        proxy_buffering off;
        client_max_body_size 0;
        proxy_connect_timeout  3600s;
        proxy_read_timeout  3600s;
        proxy_send_timeout  3600s;
        send_timeout  3600s;
    }
}

Sites Enabled directory:

# ls -al /etc/nginx/sites-enabled/
proxim.khas.dev -> /etc/nginx/sites-available/proxim.khas.dev

Nginx Status

# systemctl status nginx.service
● nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2022-06-28 23:51:56 CEST; 12min ago
       Docs: man:nginx(8)
    Process: 1560165 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
    Process: 1560166 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
   Main PID: 1560167 (nginx)
      Tasks: 13 (limit: 77040)
     Memory: 11.2M
        CPU: 82ms
     CGroup: /system.slice/nginx.service
             ├─1560167 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
             ├─1560781 nginx: worker process
             ├─1560782 nginx: worker process
             ├─1560783 nginx: worker process
             ├─1560784 nginx: worker process
             ├─1560785 nginx: worker process
             ├─1560786 nginx: worker process
             ├─1560787 nginx: worker process
             ├─1560788 nginx: worker process
             ├─1560789 nginx: worker process
             ├─1560790 nginx: worker process
             ├─1560791 nginx: worker process
             └─1560792 nginx: worker process

Jun 28 23:51:56 proxim systemd[1]: Starting A high performance web server and a reverse proxy server...
Jun 28 23:51:56 proxim systemd[1]: Started A high performance web server and a reverse proxy server.

The question is why Certbot is trying to get the certificate over https? If https is not currently available? And unfortunately I can't figure out how it gets to https. Nowhere in my configurations can I find a redirect or anything else.

2 Likes

Welcome to the community @khashashin

Very nice report - thanks. Yes, the Let's Encrypt server always starts with an HTTP request. It will follow redirects and that's why you see HTTPS in the error message. It looks like your OPNsense system is doing the redirect.

curl -I proxim.khas.dev/.well-known/acme-challenge/ForumTest

HTTP/1.1 301 Moved Permanently
Location: https://proxim.khas.dev/.well-known/acme-challenge/ForumTest
Date: Tue, 28 Jun 2022 22:35:24 GMT
Server: OPNsense

EDIT: @khashashin Oops. I re-pasted the curl results and dropped the most important line the second time! See the Server header.

7 Likes

Thanks Mike, it's been a while since I installed OPNsense and I need to investigate further in that direction. Thanks for pointing that out.

3 Likes

I thought it is desirable to have HTTP/1.1 301 Moved Permanently for http to get to https?
Or am I miss understanding the desired goal?

5 Likes

It is once HTTPS is configured and working :slight_smile: (there were no nginx port 443 servers shown among other items I noticed). What also wasn't shown was OPNsense was ultimately responding with a failure page (my bad, was in a hurry).

Still, there is no reason to redirect the http challenge request from the Let's Encrypt server - ever. It is quicker and less error prone to handle the initial http request.

Looking at this now @khashashin, OPNsense is still interfering. A test http challenge is redirected (in a different way) and then results in an HTTP 200 OK. I should see a 404 Not Found because the test page is not there. The results (condensed below) say something about OPNsense rejecting a possible DNS Rebind attack.

Let's Encrypt will try to follow redirects so that is not wrong by itself. But, it must reach the nginx server to resolve correctly.

curl -iL proxim.khas.dev/.well-known/acme-challenge/ForumTest642
HTTP/1.1 302 Found
Location: /?url=/.well-known/acme-challenge/ForumTest642
Server: OPNsense

HTTP/1.1 200 OK
Content-type: text/html; charset=UTF-8
Server: OPNsense

<!doctype html>
<!--[if IE 8 ]><html lang="en" class="ie ie8 lte9 lte8 no-js"><![endif]-->
<!--[if IE 9 ]><html lang="en" class="ie ie9 lte9 no-js"><![endif]-->
<!--[if (gt IE 9)|!(IE)]><!--><html lang="en" class="no-js"><!--<![endif]-->
  <head>
...
<a href="/index.php?logout">A potential 
<a href="http://en.wikipedia.org/wiki/DNS_rebinding">
DNS Rebind attack</a> has been detected.<br />
Try to access the router by IP address instead of by hostname. 
You can disable this check if needed under System: Settings: Administration.</a>
6 Likes

Ah, I believe I was taking the subject to literal and seeing :eyes: that I should have read between the lines a little. Thanks @MikeMcQ :+1:

6 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.