Letsencrypt is not working

I am sorry for the somewhat vague topic name, but I am not sure how to put it differently.

I used the jwilder docker and it has worked flawlessly for many months. And from one day to the next it stopped working, and I cannot get to my site anymore.

After that it was a very long and annoying road to a solution, but all ended up in the same result: it does not work anymore.

From the logs it looks fine when everything is starting up:
WARNING: /etc/nginx/dhparam/dhparam.pem was not found. A pre-generated dhparam.pem will be used for now while a new one

is being generated in the background. Once the new dhparam.pem is in place, nginx will be reloaded.

forego | starting dockergen.1 on port 5000

forego | starting nginx.1 on port 5100

dockergen.1 | 2020/09/21 16:08:42 Generated '/etc/nginx/conf.d/default.conf' from 5 containers

dockergen.1 | 2020/09/21 16:08:42 Running 'nginx -s reload'

dockergen.1 | 2020/09/21 16:08:42 Watching docker events

dockergen.1 | 2020/09/21 16:08:42 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification 'nginx -s reload'

2020/09/21 16:08:47 [notice] 46#46: signal process started

Generating DH parameters, 2048 bit long safe prime, generator 2

This is going to take a long time

dhparam generation complete, reloading nginx

dockergen.1 | 2020/09/21 16:09:35 Received event start for container a7a9713af01b

dockergen.1 | 2020/09/21 16:09:35 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification 'nginx -s reload'

dockergen.1 | 2020/09/21 16:11:12 Received event start for container b54a169f2459

dockergen.1 | 2020/09/21 16:11:12 Generated '/etc/nginx/conf.d/default.conf' from 7 containers

nginx.1 | bee-network.nl 52.28.236.88 - - [21/Sep/2020:16:11:32 +0000] "GET /.well-known/acme-challenge/GaEf9a-be8pCPwjIu1R8j-JqIf7DUeJ2DsdUd6lPQlY HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

nginx.1 | bee-network.nl 3.128.26.105 - - [21/Sep/2020:16:11:32 +0000] "GET /.well-known/acme-challenge/GaEf9a-be8pCPwjIu1R8j-JqIf7DUeJ2DsdUd6lPQlY HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

nginx.1 | bee-network.nl 64.78.149.164 - - [21/Sep/2020:16:11:33 +0000] "GET /.well-known/acme-challenge/GaEf9a-be8pCPwjIu1R8j-JqIf7DUeJ2DsdUd6lPQlY HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

nginx.1 | bee-network.nl 34.209.232.166 - - [21/Sep/2020:16:11:33 +0000] "GET /.well-known/acme-challenge/GaEf9a-be8pCPwjIu1R8j-JqIf7DUeJ2DsdUd6lPQlY HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

nginx.1 | bee-network.nl 192.168.178.1 - - [21/Sep/2020:16:12:50 +0000] "GET / HTTP/1.1" 301 169 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_0) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15"

But then every time I try to reach my website I get this:

nginx.1 | _ 192.168.178.1 - - [21/Sep/2020:16:12:50 +0000] "\x16\x03\x01\x02\x00\x01\x00\x01\xFC\x03\x03\xF3~L;\x92Y\xF8\xF1\xEB\xA2;\x99j\xE6\xAEV\x16)\xA4\x99\xB1E\xFC\xFB[*h\xEB\xBE\xB5\xC4\xE5 \xE0\xC4\xB6X\xC0\x00\x101e\x00\xB7!\x7F\x1D\x91\x1E\x07\x07Re" 400 157 "-" "-"

nginx.1 | _ 192.168.178.1 - - [21/Sep/2020:16:12:50 +0000] "\x16\x03\x01\x02\x00\x01\x00\x01\xFC\x03\x03*\x8B\xC2\xFE\xDD\x8A#\xC5\x84\xB5\xE3\xE6\xC5\x22\x94\xC6\x8B\xE9\x86\x10\xD0\x17\x99X{\xBD\xE6\xFFjEt\xB7 \x1A\x894\xC5w\xEA\xBCa\xB3\xFD\x98\xAF\xC5Q\xCB\x86\xDE\xAB\x0F\x9B\xFB\xA2~\xDC\x12\xEC?p\x97\x87)" 400 157 "-" "-"

nginx.1 | _ 192.168.178.1 - - [21/Sep/2020:16:12:50 +0000] "\x16\x03\x01\x00\x94\x01\x00\x00\x90\x03\x01\xCAO1\xAF*\xE9\x898\x1E7\xB7^\x86\xD9\xFC\xEB\xD2\x93\x8D-\xAC\x1E\x80>\xF0\x11\xC2\x0F\x16\xAB\x8FA\x00\x00\x14\xC0" 400 157 "-" "-"

And it must be something I am missing, because this is the result with every way of installing the dockers I have tried sofar.

So I am hoping this points to something silly that has changed, because I am getting fed up with all the people that report that its working fine for them :frowning:

2 Likes

I think so too, but I'm not sure you have provided enough information to solve the problem.
Working with what is known so far... "bee-network.nl"
I can see that HTTP forwards to HTTPS [this is good]
But HTTPS fails with:

curl -Iki https://bee-network.nl/
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

and this seems to indicate that you are missing "ssl engine on" or some equivalent:

curl -Iki http://bee-network.nl:443/
HTTP/1.1 301 Moved Permanently
Server: nginx/1.17.6
Date: Mon, 21 Sep 2020 16:35:06 GMT
Content-Type: text/html
Content-Length: 169
Connection: keep-alive
Location: https://bee-network.nl/
2 Likes

Besides the issue with your very much broken nginx configuration like @rg305 already pointed out, I'd like to mention that Mozilla currently advices to actually use specific pre-generated DH parameters from RFC 7919: Security/Server Side TLS - MozillaWiki (linking to the Wiki history, because since then they've removed the additional information on that page with now just the recommendations with with very short argumentation.)

3 Likes

I'd skip using any cipher that requires a dhparam.pem file - but that's just me :slight_smile:

2 Likes

Agreed, I also don't have any regular DH ciphers in my ciphers list, only EDHE.

3 Likes

Hello,

Thanks for your reply.
So I started over clean, using this link : https://github.com/nginx-proxy/docker-letsencrypt-nginx-proxy-companion

So removed all the volumes and containers and created the scripts as described.

NGINX Script
docker run --detach
--name nginx-proxy
--publish 80:80
--publish 443:443
--volume /etc/nginx/certs
--volume /etc/nginx/vhost.d
--volume /usr/share/nginx/html
--volume /var/run/docker.sock:/tmp/docker.sock:ro
jwilder/nginx-proxy

NGINX Logs
WARNING: /etc/nginx/dhparam/dhparam.pem was not found. A pre-generated dhparam.pem will be used for now while a new one
is being generated in the background. Once the new dhparam.pem is in place, nginx will be reloaded.
forego | starting dockergen.1 on port 5000
forego | starting nginx.1 on port 5100
dockergen.1 | 2020/09/22 18:46:16 Generated '/etc/nginx/conf.d/default.conf' from 5 containers
dockergen.1 | 2020/09/22 18:46:16 Running 'nginx -s reload'
dockergen.1 | 2020/09/22 18:46:16 Watching docker events
dockergen.1 | 2020/09/22 18:46:16 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification 'nginx -s reload'
2020/09/22 18:46:19 [notice] 47#47: signal process started
Generating DH parameters, 2048 bit long safe prime, generator 2
This is going to take a long time
dhparam generation complete, reloading nginx

LETSENCRYPT Script
docker run --detach
--name nginx-proxy-letsencrypt
--volumes-from nginx-proxy
--volume /var/run/docker.sock:/var/run/docker.sock:ro
--env "DEFAULT_EMAIL=jaap.bij@home.nl"
jrcs/letsencrypt-nginx-proxy-companion

LETSENCRYPT Logs
........................++++
writing new private key to '/etc/nginx/certs/default.key.new'

Info: a default key and certificate have been created at /etc/nginx/certs/default.key and /etc/nginx/certs/default.crt.
Info: Creating Diffie-Hellman group in the background.
A pre-generated Diffie-Hellman group will be used for now while the new one
is being created.
Generating DH parameters, 2048 bit long safe prime, generator 2
Reloading nginx proxy (nginx-proxy)...
2020/09/22 18:48:29 Generated '/etc/nginx/conf.d/default.conf' from 6 containers
2020/09/22 18:48:29 [notice] 72#72: signal process started
2020/09/22 18:48:29 Generated '/app/letsencrypt_service_data' from 6 containers
2020/09/22 18:48:29 Running '/app/signal_le_service'
2020/09/22 18:48:29 Watching docker events
2020/09/22 18:48:29 Contents of /app/letsencrypt_service_data did not change. Skipping notification '/app/signal_le_service'
Sleep for 3600s
This is going to take a long time
Info: Diffie-Hellman group creation complete, reloading nginx.
Reloading nginx proxy (nginx-proxy)...
2020/09/22 18:48:40 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification ''
2020/09/22 18:48:40 [notice] 95#95: signal process started

And in NGINX Logs
2020/09/22 18:48:40 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification ''
2020/09/22 18:48:40 [notice] 95#95: signal process started

Script Grafana
docker run --detach
--name grafana
--env "VIRTUAL_HOST=bee-network.nl"
--env "VIRTUAL_PORT=3000"
--env "LETSENCRYPT_HOST=bee-network.nl"
--env "LETSENCRYPT_EMAIL=jaap.bij@home.nl"
grafana/grafana

NGINX Log
dockergen.1 | 2020/09/22 18:48:26 Received event start for container a93881ce118d
dockergen.1 | 2020/09/22 18:48:26 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification 'nginx -s reload'
dockergen.1 | 2020/09/22 18:54:07 Received event start for container 9c5822523acc
dockergen.1 | 2020/09/22 18:54:07 Generated '/etc/nginx/conf.d/default.conf' from 7 containers
dockergen.1 | 2020/09/22 18:54:07 Running 'nginx -s reload'
nginx.1 | bee-network.nl 52.28.236.88 - - [22/Sep/2020:18:54:27 +0000] "GET /.well-known/acme-challenge/MLxXFnBZoI91bmxaV8p-wAzcSYS5iM2tA69ea9tIgLw HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
nginx.1 | bee-network.nl 3.128.26.105 - - [22/Sep/2020:18:54:27 +0000] "GET /.well-known/acme-challenge/MLxXFnBZoI91bmxaV8p-wAzcSYS5iM2tA69ea9tIgLw HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
nginx.1 | bee-network.nl 64.78.149.164 - - [22/Sep/2020:18:54:28 +0000] "GET /.well-known/acme-challenge/MLxXFnBZoI91bmxaV8p-wAzcSYS5iM2tA69ea9tIgLw HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
nginx.1 | bee-network.nl 34.211.6.84 - - [22/Sep/2020:18:54:28 +0000] "GET /.well-known/acme-challenge/MLxXFnBZoI91bmxaV8p-wAzcSYS5iM2tA69ea9tIgLw HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

LETSENCRYPT Log
2020/09/22 18:54:07 Received event start for container 9c5822523acc
2020/09/22 18:54:22 Debounce minTimer fired
2020/09/22 18:54:22 Generated '/app/letsencrypt_service_data' from 7 containers
2020/09/22 18:54:22 Running '/app/signal_le_service'
/etc/nginx/certs/bee-network.nl /app
Reloading nginx proxy (nginx-proxy)...
2020/09/22 18:54:22 Generated '/etc/nginx/conf.d/default.conf' from 7 containers
2020/09/22 18:54:22 [notice] 123#123: signal process started
Creating/renewal bee-network.nl certificates... (bee-network.nl)
2020-09-22 18:54:23,419:INFO:simp_le:1359: Generating new account key
2020-09-22 18:54:25,098:INFO:simp_le:1387: By using simp_le, you implicitly agree to the CA's terms of service: https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf
2020-09-22 18:54:25,497:INFO:simp_le:1450: Generating new certificate private key
2020-09-22 18:54:30,752:INFO:simp_le:401: Saving account_key.json
2020-09-22 18:54:30,753:INFO:simp_le:401: Saving account_reg.json
2020-09-22 18:54:30,754:INFO:simp_le:401: Saving key.pem
2020-09-22 18:54:30,755:INFO:simp_le:401: Saving chain.pem
2020-09-22 18:54:30,755:INFO:simp_le:401: Saving fullchain.pem
2020-09-22 18:54:30,756:INFO:simp_le:401: Saving cert.pem
/app
Reloading nginx proxy (nginx-proxy)...
2020/09/22 18:54:31 Generated '/etc/nginx/conf.d/default.conf' from 7 containers
2020/09/22 18:54:31 [notice] 147#147: signal process started
Sleep for 3600s

So far so good is seems

Now when I try to go to www.bee-network.nl I get:
nginx.1 | www.bee-network.nl 192.168.178.1 - - [22/Sep/2020:19:00:17 +0000] "GET / HTTP/1.1" 503 197 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_0) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15”\

Nginx message on page : 503 Service Temporary Unavailable

And when I go to bee-network.nl (which the browser then turns into https://bee-network.nl) I get:
nginx.1 | _ 192.168.178.1 - - [22/Sep/2020:19:02:10 +0000] "\x16\x03\x01\x00\x94\x01\x00\x00\x90\x03\x01W\xF3T\x1EV\xA1,O\xF6j\x9F\x89wguM\xCA\x89\x8B\x22\xDF\xB2\xF0\x1F\xEB\xBAI^\xC3\x18\x1Ea\x00\x00\x14\xC0" 400 157 "-" "-"

So I am totally at a loss here and have absolutely no clue what it is that is wrong.
And again, I had a working situation that at some point stopped working

Your curl commands now give this:
[root@localhost nginx-letsencrypt]# curl -Iki https://bee-network.nl/
curl: (35) SSL received a record that exceeded the maximum permissible length.

[root@localhost nginx-letsencrypt]# curl -Iki http://bee-network.nl:443/
HTTP/1.1 301 Moved Permanently
Server: nginx/1.17.6
Date: Tue, 22 Sep 2020 19:17:43 GMT
Content-Type: text/html
Content-Length: 169
Connection: keep-alive
Location: https://bee-network.nl/

2 Likes

I have no experience with "docker" at all (and frankly, I don't want to have any to be honest..), so it's very hard to debug this for me personally. But perhaps you can answer some questions:

Does that "nginx-proxy" container contain a fully functioning nginx? Because it sounds like it.... wait for it.. proxies to something using nginx.. What destination are you using?

Also, I'm getting a "HTTP/1.1 503 Service Temporarily Unavailable" error from your nginx..

2 Likes

I see some potential... issues:

  1. one has WWW, the other doesn't.
  1. no TLS(SSL) on port 443.
2 Likes

Hello,

Thanks again for your reply. I am very happy using docker, as it makes things easier when you are playing around with al sorts of programs and do not want any f..up to kill your server.
And as my server runs several things that are not to be disturbed, docker is one way of achieving this.

Anyway, as I understand, you are running Nginx and lets encrypt on the os directly?
That was my next plan to do. Luckily we have a test server, so I can first play around with it before I let it loose on the main server :slight_smile:

I will keep you posted.

2 Likes

I suggest you concentrate on fixing your docker setup, not blindly falling down another rabbit hole.

2 Likes

So I followed a nice tutorial again, and installed Nginx on docker, and it worked like a charm.
But as soon as I get lets encrypt in the mix, it looks like it is working for a short moment, and then I get the jibberish I had before . . .

nginx.1 | _ 207.148.93.136 - - [02/Oct/2020:13:41:11 +0000] "\x16\x03\x01\x02\x00\x01\x00\x01\xFC\x03\x03\xD5\xC6Y\xFD\x1FM6\x5C<\xB2\x1E\x1B)6\x8B@\xCD\x94\x19\x07\x86a\x80\xE5s\xC12Y\xCE\x22\xA6\xD8 \xBE\x9Cwl`\xAD\xD5A\x8A\xA98\x05MD\xB9\xA1\x0B\x92\x84\x86R!c<\x17l \xE6\xA0@rV\x00\x96\x13\x02\x13\x03\x13\x01\xC0,\xC00\xC0+\xC0/\xCC\xA9\xCC\xA8\x00\xA3\x00\x9F\x00\xA2\x00\x9E\xCC\xAA\xC0\xAF\xC0\xAD\xC0$\xC0(\xC0" 400 157 "-" "-"

nginx.1 | _ 207.148.93.136 - - [02/Oct/2020:13:41:11 +0000] "\x16\x03\x01\x02\x00\x01\x00\x01\xFC\x03\x03\xBE!\x87x\x14 \x18\x9F\x862=\xA7\x82\xDD\x9C\xCB\xC2\xE3\x1F\xED\xAB\xCC\xEB!:t\xAE\x08m\x8C\xAA \xF9\x5C\xFDBvq\xFA\x8C\xF8\x89\xE3a\xBE\xE6\xF7\xDD\xCBkD 0\xB5\xE2k\xEF\xB4\xAE\xE7\x9D p\x00\x96\x13\x02\x13\x03\x13\x01\xC0,\xC00\xC0+\xC0/\xCC\xA9\xCC\xA8\x00\xA3\x00\x9F\x00\xA2\x00\x9E\xCC\xAA\xC0\xAF\xC0\xAD\xC0$\xC0(\xC0" 400 157 "-" "-"

And this is exactly the same stuff that keeps happening, no matter what guide I follow.
I cannot imagine that I am the only one that has this happening?

And the message "SSL received a record that exceeded the maximum permissible length.", what does that mean exactly? Does that point to a possible problem in my configuration?

I cannot get rid of the idea that something changed somewhere that is causing this to happen, as we had this succesfully running for a long time . . .

1 Like

Hi @BeeEncrypted

I didn't read the complete topic.

That error says: Your https port is a http port. So it sends the complete content, not only the short Ssl handshake.

curl -Iki http://bee-network.nl:443/

is such a sample.

So your port 443 is http, not https.

2 Likes

Then to me it is unclear where this starts to go wrong.
So Nginx works fine as long as there is no lets encrypt involved

Is this something in Nginx configuration file?

This is annoying me for months now, and no matter which guide I follow, that all say that it is very easy to set up, I always end up with the same problem.

And it has worked for a long time before that, thats even more disturbing . . .

1 Like

There is nothing unclear.

Your port 443 port definition / server block is a http port, not a https port.

Fix it manual or remove that block and let Certbot create a correct port 443 block.

1 Like

Can you show the 443 block in use?

[the results are seen in the logs - the "problem" can only be seen in the config file]

1 Like

This is always a symptom of an HTTP/HTTPS protocol confusion.

Typically, with most servers, each port can only use one protocol or the other. Normally the service listening on port 80 should use HTTP and the service on port 443 should use HTTPS. If they don't, that is definitely a problem in your configuration. This is what other people in this thread have been mentioning.

This could be a problem with your nginx configuration, with your Docker configuration, or, more likely from my current point of view, with the interaction between the two. Consider that Docker is in some way mapping TCP ports as seen by the outside world to TCP ports as seen by a Docker container. If this mapping isn't right, confusion (such as a protocol mismatch) could result.

I guess this particular problem isn't so common because not that many people are using Docker, or their particular Docker solutions match up better with what they're trying to do with Let's Encrypt.

That makes sense, but I guess you also have to be careful with the port number configuration. The isolation between the various containers also means that they might not be exposed, or accessed, in exactly the way that you or they would expect! Every network service—and every Docker configuration that exposes a network service—has certain assumptions, which could create a problem like this if they're violated.

2 Likes

Hello - Would you please be able to advise/ consult on an expired cert renewal? This is an emergency for us and appreciate your service. 917.215.6118 available now.....

1 Like

@David_Axelrad

I pinged @schoen for you.

1 Like

Hello,

I appreciate that you take the time.
My problem is that I have followed several different guides to install Nginx and letsencrypt on docker, and they all end up in the same situation.
None of these guides talk about a configuration change you are referring to. And to be honest, I would not know where to begin . . .
Are you referring to the Nginx config file here?
I will do another brave attempt tonight, and share the Nginx config file.

There was one thing that I found, but this did not solve the problem. I have a ubiquity dream machine, that has some serious security features. One of them was blocking bots.
But as said, switching this of did not help.

Regards,

Jaap

1 Like

Hello,

I have been playing around on my test server for some time now to get this sorted out.

Now what makes this problem very annoying is that it started in the midst of a working situation.
The working situation was realized with a guide, and worked right away.
Important things were to create a network within the docker environment, and have all the virtual environments that needed to be behind nginx and letsencrypt be in that same network.
Worked like a charm. And this situation changed at some point in time, and since then it never worked again . . .

One of the things I found is that my router (Unifi Dream Machine) has security settings that block bots. Switching that off did not change anything.
The only other thing that changed was the OS (CentOS) during updates, which updated docker as well.

I am affraid a am having a very particular situation, which makes it hard to solve, so if you could give some pointers were to start looking first.
From for instance the Wordpress container it is just exposing the ports 80 and 443, so that leaves the nginx config . .

Regards,

Jaap