Seeking Clarity and Consistency on Configuring HTTP-01 challenge for multiple domains

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:
schemefusion.com
I ran this command:

export DOMAINS="-d schemefusion.com -d www.schemefusion.com -d share.schemefusion.com -d architecture.schemefusion.com -d submit.schemefusion.com -d juo.schemefusion.com -d reunion.schemefusion.com"
sudo certbot certonly\
    --standalone\
    --http-01-port 8000\
    --deploy-hook 'systemctl reload nginx'\
    --cert-name schemefusion.com\
    --dry-run\
    $DOMAINS

It produced this output:

Simulating renewal of an existing certificate for schemefusion.com and 6 more domains

Certbot failed to authenticate some domains (authenticator: standalone). The Certificate Authority reported these problems:
  Domain: architecture.schemefusion.com
  Type:   unauthorized
  Detail: xxxx:xxxx::xxxx:xxxx:xxxx:xxxx: Invalid response from http://architecture.schemefusion.com/.well-known/acme-challenge/XYtsQe-5CFIgAmL332wLmkaN2Jrl2cy2Q5A5jbT5tg4: 404

  Domain: juo.schemefusion.com
  Type:   unauthorized
  Detail: xxxx:xxxx::xxxx:xxxx:xxxx:xxxx: Invalid response from http://juo.schemefusion.com/.well-known/acme-challenge/UHxiDdDYA-3lfh0af1Tuj_N_Ed7bT7tWomZbKlMZNc8: 404

  Domain: reunion.schemefusion.com
  Type:   unauthorized
  Detail: xxxx:xxxx::xxxx:xxxx:xxxx:xxxx: Invalid response from http://reunion.schemefusion.com/.well-known/acme-challenge/RhF85dUFRZ3vkr9qDsdd2ZBUgUc9RdRBNjhpXiaqWQ0: 404

  Domain: schemefusion.com
  Type:   unauthorized
  Detail: xxxx:xxxx::xxxx:xxxx:xxxx:xxxx: Invalid response from http://schemefusion.com/.well-known/acme-challenge/cWiFpm5bH6f0016yW8vSVzdzKQEpL_J_vUSbbvEOzMc: 404

  Domain: share.schemefusion.com
  Type:   unauthorized
  Detail: xxxx:xxxx::xxxx:xxxx:xxxx:xxxx: Invalid response from http://share.schemefusion.com/.well-known/acme-challenge/fh9IicsO3c8NbmiNLuCZjVAwGRVxTxJvydLAgW3b67o: 404

  Domain: submit.schemefusion.com
  Type:   unauthorized
  Detail: xxxx:xxxx::xxxx:xxxx:xxxx:xxxx: Invalid response from http://submit.schemefusion.com/.well-known/acme-challenge/a8f61jV1jngVtLWZdSKcenmi8xXCBO1lRYDb9Kja3pw: 404

  Domain: www.schemefusion.com
  Type:   unauthorized
  Detail: xxxx:xxxx::xxxx:xxxx:xxxx:xxxx: Invalid response from http://www.schemefusion.com/.well-known/acme-challenge/5lMFIkaSb8HGbn1U2ryAd6CrJn4gN2Baqk-tqQI-pq4: 404

Hint: The Certificate Authority failed to download the challenge files from the temporary standalone webserver started by Certbot on port 8000. Ensure that the listed domains point to this machine and that it can accept inbound connections from the internet.

My web server is (include version):
nginx/1.30.1

The operating system my web server runs on is (include version):
Fedora Server 43 (6.19.7-200.fc43.x86_64)

My hosting provider, if applicable, is:
Linode (Akamai)

I can login to a root shell on my machine (yes or no, or I don't know):
yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
certbot 4.1.1

2

To clarify further, most subdomains are serviced by proxies (gunicorn) via nginx. Therefore, I am wondering how to establish a .well-known/acme-challenge/ folder for such scenarios. Perhaps I am missing a configuration that allows certbot to issue certificates for multiple gunicorn sites.

Context:

3

Why are you using these options?

5 
--standalone

At some point in the past, I needed this and added it to the script. See also Certbot renew --dry-run fails with 502 - #12 by rg305

--http-01-port 8000\

Because of the following server in nginx (for the challenge)

upstream certbot {
    server 127.0.0.1:8000;
}
6

Please provide all the relevant configuration in one go instead of making us ask for every specific snippet. It's tedious, takes a long time and doesn't help you getting a quick solution either.

And no, I'm not going to sift through all kinds of previous threads where I might find an answer, sorry.

7

Your prior thread had you using a single default server block for HTTP requests for all domain names.

If the HTTP request was an HTTP Challenge you sent it to that "upstream" and redirected all other requests to HTTPS. Perhaps a bit more complicated than needed but that should be fine.

The problem is you no longer have a default server block that does that. I can tell because a test HTTP Challenge should fail with a proxy timeout yet it returns a 404. And, other HTTP requests should redirect to HTTPS but they do not. They get a default Fedora page instead. Something is very different about your nginx config than it used to be.

I would need to see your entire nginx config to sort out what has gone wrong.

Please post the entire contents of this

sudo nginx -T

It will be very long. An upper case T is essential

Ideally, you would run this then upload the config.txt file to this thread but either way is fine

sudo nginx -T >~/config.txt

That puts the config.txt in your home directory. You could put it somewhere else just don't put in within the /etc/nginx as nginx might process it as new config.

The nginx server replying to HTTP requests says it is 1.28.3

curl -i http://schemefusion.com
HTTP/1.1 200 OK
Server: nginx/1.28.3