Nginx Manager Certs

bignellrp · February 3, 2021, 9:45am

Hi,

I have attempted to move to CloudFlare for my dns provider and use Nginx Proxy Manager to point at my ISPConfig3 VM but also have the option using the proxy manager to point sub domains to other internal hosts.

The problem im having is with the certs. CloudFlare gives all the domains a free ssl cert anyway but has the option for full end to end encryption. This seems to want SSL between the proxy and the host and fails if i disable SSL.

The problem comes when i try and register a new cert for a domain that used to live on ISPConfig. So the cert still exists and hasnt yet expired.

When you request a new cert on Ngnix Proxy Manager you just get an Internal Failure message. So i looked in the docker logs and tried to run the command manually and i get this:

root@Tower:~# docker exec -it NginxProxyManager /bin/bash

bash-5.0# certbot certonly --non-interactive --config "/etc/letsencrypt.ini" --cert-name "npm-15" --agree-tos --email "bignellrp@gmail.com" --preferred-challenges "dns,http" --domains "www.northamptonmorrismen.co.uk"
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for www.northamptonmorrismen.co.uk
Using the webroot path /config/letsencrypt-acme-challenge for all unmatched domains.
Waiting for verification...
Challenge failed for domain www.northamptonmorrismen.co.uk
http-01 challenge for www.northamptonmorrismen.co.uk
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:

The following errors were reported by the server:

Domain: www.northamptonmorrismen.co.uk
Type: unauthorized
Detail: Invalid response from
https://www.northamptonmorrismen.co.uk/.well-known/acme-challenge/AlslvQOqBlSQusvOE_U4lM_G5SMdsDcu2n-GVRHC5Jo
[2606:4700:3037::6815:3be5]: "\n<ht"

To fix these errors, please make sure that your domain name was
entered correctly and the DNS A/AAAA record(s) for that domain
contain(s) the right IP address.

rg305 · February 3, 2021, 9:55am

It sounds like you've made multiple changes and now are unsure which one is breaking the certificate renewals...
[which have worked well for quite some time: crt.sh | www.northamptonmorrismen.co.uk]

Let's start with a basic design/flow.
Please detail as best you can how the Internet reaches your sites in a straight line path.
Then also show the renewal config file the domain www.northamptonmorrismen.co.uk

bignellrp · February 3, 2021, 10:03am

Yes that is a good summary. I have made a few changes at once and the renewal via ISPConfig has been working for a long time with no issues.

My path from the internet to this domain is as follows:

Client > DNS (cloudflare) > My public IP > my firewall > Nginx Proxy Manager on port 4443 > ISPConfig on port 443.

bignellrp · February 3, 2021, 4:54pm

Can i add in case it helps that i have manually copied the current certs that are still valid until March over to Nginx Proxy Manager and they work fine. Is it because im trying to renew on the proxy when they are not due for renewal yet? Surely the error would suggest that right?

rg305 · February 3, 2021, 5:34pm

I'm not 100% certain (yet) and you would have to show the exact error message to make sense of it.

I need some more information:
Q. When the firewall hears HTTP (port 80) requests, to which system does it send them?
Q. When the firewall hears HTTPS (port 443) requests, to which system does it send them?

bignellrp · February 3, 2021, 6:19pm

The firewall forwards port 80 to port 8080 on the Nginx proxy. The firewall also sends 443 to 4443 on the proxy. The proxy is the server doing the new cert request. The proxy then sends the traffic to either the ISPconfig server or individual docker hosts based on the domain forward rules.

I could run the certbot with -vvv but it looks like it has lots of secure token and IP information.

Here is a snip from vvv. Is it saying that 80 is failing? The proxy is set to forward to 443 so im not sure 80 will work as it should be forcing 80 to 443 anyway.

  "status": "invalid",
  "error": {
    "type": "urn:ietf:params:acme:error:unauthorized",
    "detail": "Invalid response from https://www.northamptonmorrismen.co.uk/.well-known/acme-challenge/FHN6L0-_sdGVb9PylwN0_zh_hlPxFEQEN8PZb_w_rXE [2606:4700:3037::6815:3be5]: \"\u003c!DOCTYPE html PUBLIC \\\"-//W3C//DTD XHTML 1.0 Transitional//EN\\\"\\n   \\\"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd\\\"\u003e\\n\u003cht\"",
    "status": 403
  },
  "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/10581253361/T2PkXg",
  "token": "FHN6L0-_sdGVb9PylwN0_zh_hlPxFEQEN8PZb_w_rXE",
  "validationRecord": [
    {
      "url": "http://www.northamptonmorrismen.co.uk/.well-known/acme-challenge/FHN6L0-_sdGVb9PylwN0_zh_hlPxFEQEN8PZb_w_rXE",
      "hostname": "www.northamptonmorrismen.co.uk",
      "port": "80",
      "addressesResolved": [
        "104.21.59.229",
        "172.67.184.212",
        "2606:4700:3037::6815:3be5",
        "2606:4700:3036::ac43:b8d4"
      ],
      "addressUsed": "2606:4700:3037::6815:3be5"
    },

rg305 · February 3, 2021, 7:59pm

OK that clears things up, thanks.

You are using the webroot authenticator - this is good and likely the best way to traverse any CDN.

Let's review and compare the specified webroot path defined in the renewal config file and the document root defined in the HTTPS server block for this domain.
[please share those file contents]

bignellrp · February 3, 2021, 10:27pm

The webroot path mentioned seems to be empty....

Using the webroot path /config/letsencrypt-acme-challenge for all unmatched domains.
Creating root challenges validation dir at /config/letsencrypt-acme-challenge/.well-known/acme-challenge
Attempting to save validation to /config/letsencrypt-acme-challenge/.well-known/acme-challenge/FHN6L0-_sdGVb9PylwN0_zh_hlPxFEQEN8PZb_w_rXE
Waiting for verification...

bash-5.0# ls -la /config/letsencrypt-acme-challenge/
total 0
drwxr-xr-x 1 app users 0 Feb 3 19:33 .
drwxrwxrwx 1 app users 252 Feb 3 21:37 ..

"{
"identifier": {
"type": "dns",
"value": "www.northamptonmorrismen.co.uk"
},
"status": "invalid",
"expires": "2021-02-10T18:16:10Z",
"challenges": [
{
"type": "http-01",
"status": "invalid",
"error": {
"type": "urn:ietf:params:acme:error:unauthorized",
"detail": "Invalid response from https://www.northamptonmorrismen.co.uk/.well-known/acme-challenge/FHN6L0-_sdGVb9PylwN0_zh_hlPxFEQEN8PZb_w_rXE [2606:4700:3037::6815:3be5]: "\u003c!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\"\n \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd\"\u003e\n\u003cht"",
"status": 403
},
"url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/10581253361/T2PkXg",
"token": "FHN6L0-_sdGVb9PylwN0_zh_hlPxFEQEN8PZb_w_rXE","

Im not sure what you mean by document root when its a proxy as im new to using nginx in this way but here is the config file that nginx manager has generated. It currently has a copy of the original cert i uploaded. This is what im trying to update to use LetsEncrypt here instead so it renews automatically on the proxy and not on the ispconfig server):

server {
set $forward_scheme https;
set $server "192.168.0.223";
set $port 443;

listen 8080;
listen [::]:8080;

listen 4443 ssl http2;
listen [::]:4443;

server_name www.northamptonmorrismen.co.uk;

Custom SSL

ssl_certificate /data/custom_ssl/npm-16/fullchain.pem;
ssl_certificate_key /data/custom_ssl/npm-16/privkey.pem;

access_log /config/log/proxy_host-5.log proxy;
location / {

# Proxy!
include conf.d/include/proxy.conf;

}

Custom

include /data/nginx/custom/server_proxy[.]conf;
}

Unless you mean the documentRoot from the ispconfig server the proxy is pointing at?

            DocumentRoot /var/www/clients/client3/web5/web
            ServerName northamptonmorrismen.co.uk
            ServerAlias www.northamptonmorrismen.co.uk
            ServerAdmin webmaster@northamptonmorrismen.co.uk
            ErrorLog /var/log/ispconfig/httpd/northamptonmorrismen.co.uk/error.log

rg305 · February 3, 2021, 10:30pm

I meant for you to show the contents of the LE renewal config file usually located in:
/etc/letsencrypt/renewal/{your.domain}.conf

rg305 · February 3, 2021, 10:35pm

Sorry... more confusion
Q. Which system will be running certbot?
Q. Which system will be terminating the inbound TLS connections?
[I think I know but want NOT to assume anything]

bignellrp · February 3, 2021, 10:36pm

There doesnt seem to be one for the domain im trying to add. Only one in there for a domain that was successful:

bash-5.0# cat /etc/letsencrypt/renewal/npm-5.conf

renew_before_expiry = 30 days

version = 1.4.0
archive_dir = /etc/letsencrypt/archive/npm-5
cert = /etc/letsencrypt/live/npm-5/cert.pem
privkey = /etc/letsencrypt/live/npm-5/privkey.pem
chain = /etc/letsencrypt/live/npm-5/chain.pem
fullchain = /etc/letsencrypt/live/npm-5/fullchain.pem

Options used in the renewal process

[renewalparams]
account = a1f1016c820eac1d050ee73bc541f9fb
pref_challs = dns-01, http-01
authenticator = webroot
webroot_path = /config/letsencrypt-acme-challenge,
server = https://acme-v02.api.letsencrypt.org/directory
[[webroot_map]]
www.richardbignell.co.uk = /config/letsencrypt-acme-challenge

rg305 · February 3, 2021, 10:37pm

Can we compare the web config of the one that worked with the one that doesn't within the proxy?

And also show:
certbot certificates

bignellrp · February 3, 2021, 10:38pm

Previously the ispconfig server ran certbot. But now the nginx proxy sits in the middle. This is where im now running it.

The nginx proxy will terminate the inbound TLS then forward onto the ispconfig server

bignellrp · February 3, 2021, 10:39pm

bash-5.0# certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log

Found the following certs:
Certificate Name: npm-5
Serial Number: 4f3e8c3d68344909705038da8e43dc12034
Domains: www.richardbignell.co.uk
Expiry Date: 2021-04-30 18:35:03+00:00 (VALID: 85 days)
Certificate Path: /etc/letsencrypt/live/npm-5/fullchain.pem
Private Key Path: /etc/letsencrypt/live/npm-5/privkey.pem

When you say webconfig?

rg305 · February 3, 2021, 10:40pm

The nginx proxy configuration

Q. When did you switch that working cert domain ("www.richardbignell.co.uk") to CloudFlare CDN?

Pending...:

bignellrp · February 3, 2021, 10:45pm

            #APACHE
			
			DocumentRoot /var/www/clients/client2/web2/web
            ServerName richardbignell.co.uk
            ServerAlias www.richardbignell.co.uk
            ServerAdmin webmaster@richardbignell.co.uk
            ErrorLog /var/log/ispconfig/httpd/richardbignell.co.uk/error.log
			
			# PROXY
			server {
			  set $forward_scheme https;
			  set $server         "192.168.0.223";
			  set $port           443;

			  listen 8080;
			listen [::]:8080;

			listen 4443 ssl http2;
			listen [::]:4443;
			  server_name www.richardbignell.co.uk;


			  Let's Encrypt SSL
			  include conf.d/include/letsencrypt-acme-challenge.conf;
			  include conf.d/include/ssl-ciphers.conf;
			  ssl_certificate /etc/letsencrypt/live/npm-5/fullchain.pem;
			  ssl_certificate_key /etc/letsencrypt/live/npm-5/privkey.pem;

			  access_log /config/log/proxy_host-2.log proxy;
			  
			  location /ispconfig {
			    proxy_set_header Host $host;
			    proxy_set_header X-Forwarded-Scheme $scheme;
			    proxy_set_header X-Forwarded-Proto  $scheme;
			    proxy_set_header X-Forwarded-For    $remote_addr;
			    proxy_pass       https://scudvm.richardbignell.co.uk:8080;

			  }
			  location / {
			    Force SSL
			    include conf.d/include/force-ssl.conf;
			    Proxy!
			    include conf.d/include/proxy.conf;
			  }

			  Custom
			  include /data/nginx/custom/server_proxy[.]conf;
			}

bignellrp · February 3, 2021, 10:46pm

they all switched at the same time to CDN

bignellrp · February 3, 2021, 10:51pm

One thing to note is that on the cloudflare dns northamptonmorrismen cnames to richardbignell.co.uk

Then richardbignell.co.uk cnames to my ddns name

rg305 · February 3, 2021, 10:52pm

That is not a specific date, nor does it satisfy my question.
Try this one instead:
Q. Did you switch before or after that cert renewed?
It renewed about 5 days ago:

I need an exact date to compare that with.

Some observations:

The cert doesn't cover all the names being served:

The webroot and DocumentRoot don't seem to match; as the webroot is not specified in the command nor within an existing renewal config file, it uses the default:

But the web server block shows:

You are listening to HTTP and HTTPS within the same server block - that is difficult to do correctly (even for pros):

bignellrp:

listen 8080;
listen [::]:8080;
listen 4443 ssl http2;
listen [::]:4443;
More than likely it is doing HTTPS on both ports.
But as you are using CloudFlare it may become an non-issue; as no one can actually reach your sites via HTTPS [CloudFlare will redirect all requests to HTTPS and only connect to your site via HTTPS - if configured securely]
So fixing it won't fix anything - but I would still fix it OCD rulez me!

There is no special handling of the ACME challenge requests seen.
Please show the https server block for the correctly renewed cert.

bignellrp · February 3, 2021, 11:04pm

This is all handled by Nginx Proxy Manager. Its a gui tool which is why the config looks to be written by a PRO. It also means it should be very difficult to get wrong unless there is a bug.

You seem to be debugging the domain that is working fine rather than the cert that isnt.

The exact date this moved to cloudflare was 28th Jan. None of the certs were due for renewal, i am just moving them to the proxy instead of having them on the server so i can use the proxy to direct to different internal web servers.

Maybe i should just try again when then are due for renewal.

Topic		Replies	Views
Unable to generate a SSL Certificate - Nginx proxy manager docker Help	11	5885	September 17, 2023
Internal Error when creating SSL Certs/NGINX Help	7	548	December 18, 2022
Error creating cert with nginx proxy manager Help	2	803	August 14, 2022
Ssl internal error in nginx proxy manager Help	3	6004	October 19, 2022
Internal Error: Trouble acquiring SSL Certs with Nginx Proxy Manager Help	3	111	December 14, 2024

Nginx Manager Certs

Custom SSL

Custom

renew_before_expiry = 30 days

Options used in the renewal process

Related topics