Renewing failing on synology with Reverse proxies


#1

I have a sinology where I created the current certificates right before the shutdown of the deprecated method back in Feb. When I got those certs I had several web-proxies that passed the certificate setup, but now that the certs need to be renewed, they are failing.

Renewal for the certs fails with

# /usr/syno/sbin/syno-letsencrypt renew -c <ID1>
{"error":200,"file":"client.cpp","msg":"new_authz: unexpect httpcode."}

Main server conf

server {
		server_name www.kreme.com;
			listen 80;
			listen 443;
			     location /.well-known {
					proxy_pass https://wwww.kreme.com/.well-known;
					proxy_set_header Host $host;
					proxy_set_header X-Real-IP $remote_addr;
					proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
					} 
}

One of the names in the second cert

server {
	server_name sonarr.kreme.com;
	listen 80;
	     location /.well-known {
             proxy_pass https://wwww.kreme.com/.well-known;
             proxy_set_header Host $host;
             proxy_set_header X-Real-IP $remote_addr;
             proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
     }     
	return 301 https://sonarr.kreme.com
		;
}

(I actualy two different Ids to renew since there are too many for Synologies UI to accept, but they fail identically). I suspect there is something wrong in the main server conf file? I tried removing the listen 443; line, but no difference. I also tried simply moving the conf file for the main domain aside, but again, no difference.


#2

Hi @LwsBtlr

I don’t understand your setup. Listen 80 and a proxy_pass to https - does that really work?

But checking your domain names there are things that can’t work.

Your sonarr ( https://check-your-website.server-daten.de/?q=sonarr.kreme.com ):

Domainname Http-Status redirect Sec. G
http://sonarr.kreme.com/
65.121.55.46 302 http://sonarr.kreme.com:5000 0.300 D
http://sonarr.kreme.com:5000 200 0.520 H
https://sonarr.kreme.com/
65.121.55.46 -2 1.433 V
ConnectFailure - Unable to connect to the remote server No connection could be made because the target machine actively refused it 65.121.55.46:443
http://sonarr.kreme.com/.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de
65.121.55.46 302 http://sonarr.kreme.com:5000 0.304 D
Visible Content: 302 Found nginx

/.well-known/acme-challenge is redirected to port 5000, Letsencrypt doesn’t follow such a redirect -> validation fails.

And your main domain ( https://check-your-website.server-daten.de/?q=kreme.com ):

Domainname Http-Status redirect Sec. G
http://kreme.com/
65.121.55.45 302 https://www.covisp.net/ 0.293 E
http://www.kreme.com/
65.121.55.46 302 http://www.kreme.com:5000 0.300 D
http://www.kreme.com:5000 200 0.520 H
https://kreme.com/
65.121.55.45 200 2.610 N
Certificate error: RemoteCertificateNameMismatch
https://www.kreme.com/
65.121.55.46 -2 1.443 V
ConnectFailure - Unable to connect to the remote server No connection could be made because the target machine actively refused it 65.121.55.46:443
https://www.covisp.net/ 200 2.520 A
http://kreme.com/.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de
65.121.55.45 302 https://www.covisp.net/.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de 0.304 E
Visible Content: Found The document has moved here .
http://www.kreme.com/.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de
65.121.55.46 302 http://www.kreme.com:5000 0.300 D
Visible Content: 302 Found nginx
https://www.covisp.net/.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de 404 1.257 A
Not Found
Visible Content: Not Found The requested URL /.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de was not found on this server.

The non-www /.well-known/acme-challenge is redirected to https + www + covisp.net - does your Letsencrypt client really saves there the validation file?

The www version /.well-known/ is redirected to the root and port 5000 -> that doesn’t work.

Looks like there are too much different things. The port 5000 isn’t visible in your definition. Same with the other domain.


#3

You can’t listen to HTTP and HTTPS in the same block - choose one.

There are four W’s in that name… is that intentional?


#4

Yeh, I have no idea what is going on here. There is not mention of covisp.net anywhere in the entire nginx configuration, nor of port 5000 in any of the reverse proxy settings or the sites-enabled files, thought it is the default port the synology users for its DSM web face, nor any idea why 65.121.55.45 is showing up for you anywhere as that is an entirely different IP (one of the name servers for my domain).

I copied the location block from another post here (minus the wwww tyop). About Letsencrypt behind a reverse proxy

Fixing the wwww issue results in a different error.

{"error":102,"file":"client.cpp","msg":"Invalid response from https://sab.kreme.com/login/ [65.121.55.46]: \"<html lang=\\\"en\\\">\\n<head>\\n    <title>SABnzbd - Log in</title>\\n\\n    <meta http-equiv=\\\"Content-Type\\\" content=\\\"text/html; charset=utf\""}

So the location block isn’t doing what I expected it to do, as that is the login page for sabnzbd (its sites-enabled file is essentially identical to the sonarr one, other than the server name).

The sub domains now load properly, so all of that configuration failure seems to have been caused entirely by the typo in the proxy line. One other thing that is not obvious here is that the main domain is setup for HSTS, so it is https: only.

changing the proxy to http:// and disabling HSTS has not made a difference.


#5

Then your nameserver settings are wrong.

So every check (my tool or Letsencrypt) checks the wrong ip address.


#6

All those errors in your check were caused by the wwww typo.


#7

Your last check ( https://check-your-website.server-daten.de/?q=sonarr.kreme.com ) shows a new error:

http + /.well-known/acme-challenge/unknown-file is redirected to https (this is ok) and the “/”, that’s bad. Letsencrypt can’t find the validation file in your root directory.

A redirect shouldn’t remove the REQUEST_URI (folder and file name):

RewriteEngine on
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,QSA,R=permanent]

(~~ same with nginx).


#8

yes, the 301 is intentional. The purpose of the reverse proxy is to have these services on my synology on a https connection.

So proxy_pass http://www.kreme.com/.well-known; should be proxy_pass http://www.kreme.com/.well-known$request_uri;?

I’ll give that a shot.

Wel, that gets back to the {"error":200,"file":"client.cpp","msg":"new_authz: unexpect httpcode."} error. :smiley:


#9

The main question: Does your Letsencrypt client knows that configuration?

What client do you use? Which authentication?

If the Letsencrypt client saves the validation file in another directory, that can’t work.


#10

Synology has its own (somewhat opaque) letsencrypt setup. I’m getting 502 errors or 301 error, depending on the configuration, when trying to look at .well-known, it seems like the location /.well-known block is ignored if the return 301 is there.


#11

If you use the Synology integrated solution:

Perhaps you should redirect http + /.well-known to your Synology http + /.well-known, not to https.

If Synology uses own location definitions (http + /.well-known), that can’t work if you create a https redirect.


#12

I have “solved” the immediate issue by moving aside all the site-enable files, renewing the certs, and putting them back. Not ideal, but it’s either that or leave the subdomains open to http requests.


#13

yes, I tried that.

I will continue to try various things now that I have another 90 days.


#14

It’s possible, though usually sites should have separate server blocks with different settings. (E.g. a redirect to HTTPS in the HTTP block, and an HSTS header in the HTTPS block.)

listen 80;
listen [::]:80;
listen 443 http2 ssl;
listen [::]:443 http2 ssl;

You can do something like:

location / {
    return http://example.com$request_uri;
}
location /.well-known/acme-challenge/ {
    other;
    stuff;
}

#15

Yeah, I did try that.

server {
	server_name sonarr.kreme.com ;
	listen 80 ;
	location /.well-known {
        proxy_pass http://www.kreme.com$request_uri;
		proxy_set_header Host $host;
		proxy_set_header X-Real-IP $remote_addr;
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
	}
	location / {
		    return 301 https://sonarr.kreme.com ;
	}
}

Trying to access sonarr.kreme.com/.well-known with curl returns a 502 error. Trying to access it in a web-browser redirects to https://sonarr.kreme.com/ ¯_(ツ)_/¯


#16

Yep, that’s your last check-result.

But that’s the problem - your Synology may not understand that.


#17

man, I thought I had this figured out. The Synology-created nginx.conf has the following:

OK, This DID work. I forgot to check specifically for the acme-challenge subdirectory.

location ^~ /.well-known/acme-challenge {
  root /var/lib/letsencrypt;
  default_type text/plain;
}

{EDIT} Certs renewed with this configuration