Certbot standalone failed, even though no other server is running

I have a simple nginx setup that was working well for dev.camsync.io. Now I'm trying to add a few variants of that domain name, and I'm running into issues. At first I added camsync.io to my same nginx config file, and ran certbot. It prompted me to set up both those domains, then prompted me to (E)xpand the existing cert which covered dev.. I canceled out of that, and made more changes to my nginx config, wanting to separate *dev.camsync.io and *camsync.io configurations.

I had two server blocks in my nginx config; the HTTP/port 80 stuff to serve up .well-known and redirect everything else to HTTPS, and the HTTPS block for dev.camsync.io:

server {

        listen [::]:443 ssl;
        listen 443 ssl;

        server_name dev.camsync.io;
        
        ssl_certificate /etc/letsencrypt/live/dev.camsync.io/fullchain.pem; # managed by Certbot
        ssl_certificate_key /etc/letsencrypt/live/dev.camsync.io/privkey.pem; # managed by Certbot
        include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
        ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
        
        # Serve up any static assets that exist here…
        
        root /var/www/dev.camsync.io/;
        
        location / {
                try_files $uri $uri/ $uri/index.html @proxy;
        }
        
        # Everything else goes to the API webapp running in a Docker container…
        
        location @proxy {
                proxy_pass http://127.0.0.1:8888;
                proxy_pass_header Server;
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_connect_timeout 3s;
                proxy_read_timeout 10s;
        }
}

I duplicated that block and removed all the dev. bits in it (paths, everything) hoping certbot would see it and create the appropriate files on disk under /etc/letsencrypt/live. It didn't like that, so I commented that block out, and decided to kill the server and just run certbot standalone to create files for two of the domains.

My domain is: camsync.io, api.camsync.io

I ran this command:

certbot certonly --nginx -d camsync.io -d api.camsync.io

It produced this output:

# sudo certbot certonly --standalone -d camsync.io -d api.camsync.io
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Requesting a certificate for camsync.io and api.camsync.io

Certbot failed to authenticate some domains (authenticator: standalone). The Certificate Authority reported these problems:
  Domain: api.camsync.io
  Type:   unauthorized
  Detail: 2606:50c0:8000::153: Invalid response from http://api.camsync.io/.well-known/acme-challenge/I26fIA6BL6zn7Gl5kYciPgh0Mwp99WY2dqZbKtVN9tQ: 404

  Domain: camsync.io
  Type:   unauthorized
  Detail: 2606:50c0:8002::153: Invalid response from http://camsync.io/.well-known/acme-challenge/ub8e5tPwmiqKeBvc4zzjXJ3fdq3OnO0OXr1grdSG4DE: 404

Hint: The Certificate Authority failed to download the challenge files from the temporary standalone webserver started by Certbot on port 80. Ensure that the listed domains point to this machine and that it can accept inbound connections from the internet.

Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

My web server is (include version): standalone

The operating system my web server runs on is (include version): Ubuntu 22.04

My hosting provider, if applicable, is: DigitalOcean

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

# certbot --version
certbot 2.6.0

Certbot log here.

I went back to trying to run it via Nginx, but alas I've tried 5 times and letsencrypt.org is throttling me (frustrating), figuring I'd try webroot instead.

I’ve checked to make sure nothing is running on port 80 using curl.

UPDATE: Trying with webroot and nginx running, it created a cert for api.dev.camsync.io. I'm still throttled for the other domains. I had to be explicit about the root directory, which didn't exist and I had to create it. What does certbot default to when invoked with --nginx?

Based on your error message and me checking just now, both your domains point to pages.github.com. Am I missing something? How does your nginx server relate to Github Pages?

3 Likes

Ugh, yes, thank you. I forgot I had set that up before for something else, and I was blindly copying IP addresses in my DNS config, not realizing it was GH.

Thanks again!

1 Like

Actually, I take it back. I'm looking at my DNS config, and none of it is pointing to GH any more. Can you tell me what you get for dig a camsync.io and what the TTL is?

My Nginx config starts with:

server {
        listen 80;
        listen [::]:80;

        server_name camsync.io api.camsync.io dev.camsync.io api.dev.camsync.io;

        location /.well-known/acme-challenge/ {
                root /var/www/certbot;
        }

        location / {
                return 301 https://$host$request_uri;
        }
}

I put a foo.html file in /var/www/certbot, but when I try to fetch it:

$ curl -v http://camsync.io/.well-known/acme-challenge/foo.html
*   Trying 206.189.208.247:80...
* Connected to camsync.io (206.189.208.247) port 80 (#0)
> GET /.well-known/acme-challenge/foo.html HTTP/1.1
> Host: camsync.io
> User-Agent: curl/7.88.1
> Accept: */*
> 
< HTTP/1.1 404 Not Found
< Server: nginx/1.18.0 (Ubuntu)
< Date: Mon, 29 May 2023 05:17:42 GMT
< Content-Type: text/html
< Content-Length: 162
< Connection: keep-alive

That's definitely the right server, and I can see it in my server logs.

Curiously, I'm also seeing a constant stream of

35.233.151.123 - - [29/May/2023:05:13:27 +0000] "GET /.well-known/acme-challenge/pYufSq0C2_6H9KmdqASotClGP8J2EZEUyagTb3CaMmY HTTP/1.1" 404 134 "-" "cert-manager-challenges/v1.8.0 (linux/amd64) cert-manager/e466a521bc5455def8c224599c6edcd37e86410c"

every few seconds.

Anyway, I'm at a loss. Maybe my nginx config doesn't do what I think it does: serving up anything under /.well-known/acme-challenge/ from /var/www/certbot.

(I also found an interesting bug in Safari. After I unpublished my GH pages, I reloaded the page in Safari, and it said it couldn't connect, as expected. Then I reloaded again, and it rendered the page (a very nearly empty HTML file). I reloaded again, and it can't connect. Reload again, basic page. Back and for like this ad infinitum).

Ah, a bit more clue. I had IPV6 addresses still being served up for GH. I've taken those out. Maybe in a bit that will fix things.

Well, I was finally able to get it to work with --standalone, but not with nginx running (using --nginx).

Your nginx config looks clean so --nginx should work.

You could also try --webroot which uses the running nginx system (avoids downtime compared to standalone)

You could test it with

sudo certbot certonly --dry-run --webroot -w /var/www/certbot -d camsync.io -d api.camsync.io  --deploy-hook 'nginx reload command`

I chose these two domain names as those are the names in your active cert for camsync.io right now

Replace nginx reload command with your actual command to reload nginx (restart not needed)

Once the --dry-run succeeds remove --dry-run to get a production cert

PS: Oh, the --nginx authenticator does not place a file in /.well-known/acme-challenge. Instead it makes a temp change to your nginx config using a rewrite and return to provide the correct challenge response. The changes are shown in the letsencrypt.log file

4 Likes

Oh interesting. I had guessed it looked at the config to see the acme-challenge was going to /var/www/certbot and wrote files there.

Does --webroot rewrite the config?

1 Like

No. --webroot is only an authenticator. It just places a file in the folder specified with --webroot-path (or just -w) on the certbot command line and removes the file after. It has no need to inspect your nginx config.

It is not an "installer" like the --nginx plug-in can be. It is also possible to use the --nginx plug-in as just the authenticator by using certbot certonly --nginx. But, if it wasn't working before it would require more debug to find out why. And, note that even if used with certonly that the --nginx plug-in must reload your nginx after the temp changes and reload it again after to unwind the change (and which picks up the new cert). This may not be optimal with very large nginx configs.

4 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.