Is port 80 forwarded to that nginx server? Or your Icecast server?
Because a 502 is impossible for this kind of request if that server block was active
curl -i http://stream.no-soap.net/.well-known/acme-challenge/Test404
HTTP/1.1 502 Bad Gateway
Server: nginx/1.22.1
I think so - just ran sudo netstat -tulpn | grep LISTEN and got the following:
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 729/sshd: /usr/sbin
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 1118/nginx: master
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 1819/cupsd
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 1974/icecast2
tcp 0 0 0.0.0.0:8000 0.0.0.0:* LISTEN 1974/icecast2
tcp6 0 0 :::22 :::* LISTEN 729/sshd: /usr/sbin
tcp6 0 0 :::80 :::* LISTEN 1118/nginx: master
tcp6 0 0 ::1:631 :::* LISTEN 1819/cupsd
so if i'm reading this correctly, port 80 is pointed at the nginx server, and 443 and 8000 are both pointed to the icecast server, yeah?
Well, no, that just means nginx is listening on port 80. And, icecast on 8000 (and 443)
But, if you have port forwarding active you could have sent port 80 to port 8000, for example. And, I wasn't even sure your nginx and icecast were the same machine but I guess that proves at least that much.
Did you restart nginx after making those changes? Because really there shouldn't be a 502 Gateway problem for a URL that is not handled by a proxy
Morning, @MikeMcQ !
Okay, understood. Went ahead and restarted both the server, and when that didn't work, rebooted the whole machine. Still getting the same error seen above.
Double checked my port forwarding and as far as I can tell, 80 is passing through. I'm wondering if I'm getting bit by the LetsEncrypt geoblocking bug from earlier this year? I haven't tried renewing since this update.
No, that is a completely different error and set of symptoms.
You could check the two nginx logs. Can you rerun the renew --dry-run and then show any recent entries in those logs around the time you retried this.
Ideally you would add those two lines to the port 80 server block BUT use different file names so we can be sure it is getting to the right place. Maybe like below. Be sure to restart nginx after making changes
access_log /var/log/nginx/access80.log;
error_log /var/log/nginx/error80.log;
here's the log from access.log:
[01/Sep/2024:09:36:26 -0400] "GET /.well-known/acme-challenge/ZdI_tgjtluoht6HxKtQcwDJHtYo2ZRZafrmM0WQlu1I HTTP/1.1" 404 125 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)
and error.log:
2024/09/01 09:36:26 [error] 1117#1117: *21 open() "/usr/share/nginx/(/usr/share/icecast2/web)/.well-known/acme-challenge/ZdI_tgjtluoht6HxKtQcwDJHtYo2ZRZafrmM0WQlu1I" failed (2:
No such file or directory), client: 66.133.109.36, server: stream.no-soap.net, request: "GET /.well-known/acme-challenge/ZdI_tgjtluoht6HxKtQcwDJHtYo2ZRZafrmM0WQlu1I HTTP/1.1",
host: "stream.no-soap.net"
Okay, well progress. There is now a 404 rather than a 502 Gateway failure.
The 404 means the webroot path used for the request (renew or fresh request) did not match the root folder value in your nginx config.
Would you show contents of this? (you can redact the account number if you wish)
/etc/letsencrypt/renewal/stream.no-soap.net.conf
okay, great! chipping away!
here's the renewal file:
# renew_before_expiry = 30 days
version = 2.1.0
archive_dir = /etc/letsencrypt/archive/stream.no-soap.net
cert = /etc/letsencrypt/live/stream.no-soap.net/cert.pem
privkey = /etc/letsencrypt/live/stream.no-soap.net/privkey.pem
chain = /etc/letsencrypt/live/stream.no-soap.net/chain.pem
fullchain = /etc/letsencrypt/live/stream.no-soap.net/fullchain.pem
# Options used in the renewal process
[renewalparams]
account = (redacted! lol)
authenticator = webroot
webroot_path = /usr/share/icecast2/web,
server = https://acme-v02.api.letsencrypt.org/directory
key_type = ecdsa
[[webroot_map]]
stream.no-soap.net = /usr/share/icecast2/web
.......is it as simple as that extra comma at the end of webroot_path.......?
No. They always have that. Used as delimiter in case of multi-path. Might even fail if you remove it - not sure.
Just to make sure we are talking about same failure would you show result of this
sudo certbot renew --dry-run --cert-name stream.no-soap.net
Yep, here's the dry run:
Saving debug log to /var/log/letsencrypt/letsencrypt.log
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/stream.no-soap.net.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Simulating renewal of an existing certificate for stream.no-soap.net
Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
Domain: stream.no-soap.net
Type: unauthorized
Detail: 174.161.46.112: Invalid response from http://stream.no-soap.net/.well-known/acme-challenge/Cssc0vDQdSniqdXg4BBcYqvBle727_4v8naRyTVGYjo: 404
Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.
Failed to renew certificate stream.no-soap.net with error: Some challenges have failed.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
All simulated renewals failed. The following certificates could not be renewed:
/etc/letsencrypt/live/stream.no-soap.net/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
That folder looks odd in your error log. Do you have any kind of mapping of one folder to another? I'm not sure why nginx shows the path that way. Maybe it always does I will have to check but need to step away for a bit.
Not intentionally! /usr/share/icecast2/web should be my webroot path - is there a way to update that path to remove the nginx pathing?
also, meta question - is there a reason why installation of the original certificate went smoothly, but now the 404 error has popped up on renewal?
I don't have many good guesses on why that path shows like that in the error log. A root folder is prefixed to the URI and this becomes the path to the file. You can do other clever things like alias but I did not see one in your config. The only other ideas involve odd drive setup like using symlinks or virtual mapped drives or such. Sometimes permission problems can create 404 but you are running as root.
Here is what a not-found looks like usually:
2024/09/01 18:29:56 [error] 1537#1537: *1 open() "/var/www/html/Test404" failed (2: No such file or directory),
In the above case the root was /var/www/html and just Test404 as the URI
Something to try is this: What does this show?
sudo certbot certonly --dry-run --nginx -d stream.no-soap.net
Well, yes, clearly there is some reason! 
We just do not know what it is yet.
Could also show this
sudo ls -l /usr/share/icecast2
Actually, we should look at the Certbot log too. Maybe further details about it writing the challenge token file will give a clue.
Look in /var/log/letsencrypt and post the log file for the failed run above. Not the log for the new command using --nginx option
Okay! Weird result! running sudo certbot certonly --dry-run --nginx -d stream.no-soap.net seemed to work?
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Simulating renewal of an existing certificate for stream.no-soap.net
The dry run was successful.
here's sudo ls -l /usr/share/icecast2
total 8
drwxr-xr-x 2 root root 4096 Feb 24 2024 admin
drwxr-xr-x 2 root root 4096 Sep 1 14:22 web
Yes, good. Trying to resolve the --webroot path would be a lot of trial and error. And, like I said, I don't have many good ideas (or tons of spare time).
The --nginx method inserts temp code into the port 80 server block to satisfy the challenge. it does it in a way that bypasses the file system. It's actually a nice method.
To switch to that just re-run that same command without --dry-run. You will get a new production cert and your /etc/letsencrypt/renewal config profile will get updated to use this method. You can then test this new renew profile with:
sudo certbot renew --dry-run
The --nginx plugin is common option and should work well going forward. If you want to post that Certbot log file I described to see if any clues why --webroot failed we could. I have worked on a very large number of problems and have never seen this kind of failure. I am a little curious but the juice may not be worth the squeeze - as the saying goes.
Mike! That worked! Per usual, you're the best. Cert renewed, and the dry run worked as well.
I do notice that I still have a warning on Letsdebug: Let's Debug wondering if I need to reboot the machine to set the changes?
I'll also go ahead and dig up that certbot log file as a reference so folks who might have a similar issue can see how we (you) solved it!
and of course, as soon as I post the link, the debug test passes! Let's Debug
Good you are on a successful path for certs again. Looks like you still have to get your icecast server to recognize the new cert though. I see it using a very old cert yet in reply to HTTPS requests (on port 443).
Note the Let's Debug says "OK" but in your case that is a false result. When Let's Debug runs a test it expects to see "404" errors. It cannot create files on your server so of course your server says "not found" when asked for them. Let's Debug is checking mostly for comms connections and other obvious misconfigurations.
It is a very useful tool and always a good place to start. But, in unusual cases like yours you have to understand all the twists and turns to know what conclusions can be drawn from its results.
In your case, the "404" is caused by some oddity on your system we never explained. And, as long as Let's Debug reports 404 rather than 502 or Timeout or something then at least that's helpful to know.
As far as that goes it was a temp problem using IPv6 to connect from the Let's Debug server to the Let's Encrypt API server. Nothing to do with your system at all Just one of those quirks that happens sometimes.