Renewing failing on synology with Reverse proxies

LwsBtlr · March 30, 2019, 10:13pm

I have a sinology where I created the current certificates right before the shutdown of the deprecated method back in Feb. When I got those certs I had several web-proxies that passed the certificate setup, but now that the certs need to be renewed, they are failing.

Renewal for the certs fails with

# /usr/syno/sbin/syno-letsencrypt renew -c <ID1>
{"error":200,"file":"client.cpp","msg":"new_authz: unexpect httpcode."}

Main server conf

server {
		server_name www.kreme.com;
			listen 80;
			listen 443;
			     location /.well-known {
					proxy_pass https://wwww.kreme.com/.well-known;
					proxy_set_header Host $host;
					proxy_set_header X-Real-IP $remote_addr;
					proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
					} 
}

One of the names in the second cert

server {
	server_name sonarr.kreme.com;
	listen 80;
	     location /.well-known {
             proxy_pass https://wwww.kreme.com/.well-known;
             proxy_set_header Host $host;
             proxy_set_header X-Real-IP $remote_addr;
             proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
     }     
	return 301 https://sonarr.kreme.com
		;
}

(I actualy two different Ids to renew since there are too many for Synologies UI to accept, but they fail identically). I suspect there is something wrong in the main server conf file? I tried removing the listen 443; line, but no difference. I also tried simply moving the conf file for the main domain aside, but again, no difference.

JuergenAuer · March 30, 2019, 10:40pm

Hi @LwsBtlr

I don’t understand your setup. Listen 80 and a proxy_pass to https - does that really work?

But checking your domain names there are things that can’t work.

Your sonarr ( https://check-your-website.server-daten.de/?q=sonarr.kreme.com ):

Domainname	Http-Status	redirect	Sec.	G
• http://sonarr.kreme.com/
65.121.55.46	302	http://sonarr.kreme.com:5000	0.300	D

• http://sonarr.kreme.com:5000	200		0.520	H

• https://sonarr.kreme.com/
65.121.55.46	-2		1.433	V
ConnectFailure - Unable to connect to the remote server No connection could be made because the target machine actively refused it 65.121.55.46:443

• http://sonarr.kreme.com/.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de
65.121.55.46	302	http://sonarr.kreme.com:5000	0.304	D
Visible Content: 302 Found nginx

/.well-known/acme-challenge is redirected to port 5000, Letsencrypt doesn’t follow such a redirect -> validation fails.

And your main domain ( https://check-your-website.server-daten.de/?q=kreme.com ):

Domainname	Http-Status	redirect	Sec.	G
• http://kreme.com/
65.121.55.45	302	https://www.covisp.net/	0.293	E

• http://www.kreme.com/
65.121.55.46	302	http://www.kreme.com:5000	0.300	D

• http://www.kreme.com:5000	200		0.520	H

• https://kreme.com/
65.121.55.45	200		2.610	N
Certificate error: RemoteCertificateNameMismatch

• https://www.kreme.com/
65.121.55.46	-2		1.443	V
ConnectFailure - Unable to connect to the remote server No connection could be made because the target machine actively refused it 65.121.55.46:443

• https://www.covisp.net/	200		2.520	A

• http://kreme.com/.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de
65.121.55.45	302	https://www.covisp.net/.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de	0.304	E
Visible Content: Found The document has moved here .

• http://www.kreme.com/.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de
65.121.55.46	302	http://www.kreme.com:5000	0.300	D
Visible Content: 302 Found nginx

• https://www.covisp.net/.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de	404		1.257	A
Not Found
Visible Content: Not Found The requested URL /.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de was not found on this server.

The non-www /.well-known/acme-challenge is redirected to https + www + covisp.net - does your Letsencrypt client really saves there the validation file?

The www version /.well-known/ is redirected to the root and port 5000 -> that doesn’t work.

Looks like there are too much different things. The port 5000 isn’t visible in your definition. Same with the other domain.

rg305 · March 31, 2019, 1:29am

You can’t listen to HTTP and HTTPS in the same block - choose one.

There are four W’s in that name… is that intentional?

LwsBtlr · March 31, 2019, 9:28am

Yeh, I have no idea what is going on here. There is not mention of covisp.net anywhere in the entire nginx configuration, nor of port 5000 in any of the reverse proxy settings or the sites-enabled files, thought it is the default port the synology users for its DSM web face, nor any idea why 65.121.55.45 is showing up for you anywhere as that is an entirely different IP (one of the name servers for my domain).

I copied the location block from another post here (minus the wwww tyop). About Letsencrypt behind a reverse proxy

Fixing the wwww issue results in a different error.

{"error":102,"file":"client.cpp","msg":"Invalid response from https://sab.kreme.com/login/ [65.121.55.46]: \"<html lang=\\\"en\\\">\\n<head>\\n    <title>SABnzbd - Log in</title>\\n\\n    <meta http-equiv=\\\"Content-Type\\\" content=\\\"text/html; charset=utf\""}

So the location block isn’t doing what I expected it to do, as that is the login page for sabnzbd (its sites-enabled file is essentially identical to the sonarr one, other than the server name).

The sub domains now load properly, so all of that configuration failure seems to have been caused entirely by the typo in the proxy line. One other thing that is not obvious here is that the main domain is setup for HSTS, so it is https: only.

changing the proxy to http:// and disabling HSTS has not made a difference.

JuergenAuer · March 31, 2019, 10:00am

Then your nameserver settings are wrong.

So every check (my tool or Letsencrypt) checks the wrong ip address.

LwsBtlr · March 31, 2019, 10:09am

All those errors in your check were caused by the wwww typo.

JuergenAuer · March 31, 2019, 10:15am

Your last check ( https://check-your-website.server-daten.de/?q=sonarr.kreme.com ) shows a new error:

Domainname	Http-Status	redirect	Sec.	G
• http://sonarr.kreme.com/
65.121.55.46	301	https://sonarr.kreme.com	0.304	A

• https://sonarr.kreme.com	303	https://sonarr.kreme.com/login?returnUrl=/	1.260	A

• https://sonarr.kreme.com/
65.121.55.46	303	https://sonarr.kreme.com/login?returnUrl=/	1.377	A

• https://sonarr.kreme.com/login?returnUrl=/	200		1.247	A

• http://sonarr.kreme.com/.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de
65.121.55.46	301	https://sonarr.kreme.com	0.303	A
Visible Content: 301 Moved Permanently nginx

http + /.well-known/acme-challenge/unknown-file is redirected to https (this is ok) and the “/”, that’s bad. Letsencrypt can’t find the validation file in your root directory.

A redirect shouldn’t remove the REQUEST_URI (folder and file name):

RewriteEngine on
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,QSA,R=permanent]

(~~ same with nginx).

LwsBtlr · March 31, 2019, 10:21am

yes, the 301 is intentional. The purpose of the reverse proxy is to have these services on my synology on a https connection.

So proxy_pass http://www.kreme.com/.well-known; should be proxy_pass http://www.kreme.com/.well-known$request_uri;?

I’ll give that a shot.

Wel, that gets back to the {"error":200,"file":"client.cpp","msg":"new_authz: unexpect httpcode."} error.

JuergenAuer · March 31, 2019, 10:26am

The main question: Does your Letsencrypt client knows that configuration?

What client do you use? Which authentication?

If the Letsencrypt client saves the validation file in another directory, that can’t work.

LwsBtlr · March 31, 2019, 10:49am

Synology has its own (somewhat opaque) letsencrypt setup. I’m getting 502 errors or 301 error, depending on the configuration, when trying to look at .well-known, it seems like the location /.well-known block is ignored if the return 301 is there.

JuergenAuer · March 31, 2019, 11:08am

If you use the Synology integrated solution:

Perhaps you should redirect http + /.well-known to your Synology http + /.well-known, not to https.

If Synology uses own location definitions (http + /.well-known), that can’t work if you create a https redirect.

LwsBtlr · March 31, 2019, 11:08am

I have “solved” the immediate issue by moving aside all the site-enable files, renewing the certs, and putting them back. Not ideal, but it’s either that or leave the subdomains open to http requests.

LwsBtlr · March 31, 2019, 11:10am

yes, I tried that.

I will continue to try various things now that I have another 90 days.

mnordhoff · March 31, 2019, 12:45pm

It’s possible, though usually sites should have separate server blocks with different settings. (E.g. a redirect to HTTPS in the HTTP block, and an HSTS header in the HTTPS block.)

listen 80;
listen [::]:80;
listen 443 http2 ssl;
listen [::]:443 http2 ssl;

You can do something like:

location / {
    return http://example.com$request_uri;
}
location /.well-known/acme-challenge/ {
    other;
    stuff;
}

LwsBtlr · March 31, 2019, 1:32pm

Yeah, I did try that.

server {
	server_name sonarr.kreme.com ;
	listen 80 ;
	location /.well-known {
        proxy_pass http://www.kreme.com$request_uri;
		proxy_set_header Host $host;
		proxy_set_header X-Real-IP $remote_addr;
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
	}
	location / {
		    return 301 https://sonarr.kreme.com ;
	}
}

Trying to access sonarr.kreme.com/.well-known with curl returns a 502 error. Trying to access it in a web-browser redirects to https://sonarr.kreme.com/ ¯_(ツ)_/¯

JuergenAuer · March 31, 2019, 1:39pm

Yep, that’s your last check-result.

But that’s the problem - your Synology may not understand that.

LwsBtlr · March 31, 2019, 2:45pm

man, I thought I had this figured out. The Synology-created nginx.conf has the following:

OK, This DID work. I forgot to check specifically for the acme-challenge subdirectory.

location ^~ /.well-known/acme-challenge {
  root /var/lib/letsencrypt;
  default_type text/plain;
}

{EDIT} Certs renewed with this configuration

system · April 30, 2019, 2:45pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.