Let's encrypt for an internal server behind a firewall

I want to encrypt a server, which is accessible only through firewall rule. The internal server hosts Keycloak and its PostgreSQL database. Currently the setup looks like this:

  • The firewall runs on a server with a static IP
  • A firewall rule redirects <firewall IP>:80 -> 10.0.0.7:8080. This is a server, which is in the network of the firewall.
  • On the internal server there are two docker containers running, which are Keycloak and its PostgreSQL database. The containers map ports 8080 to 8080 (Keycloak) and 8443 to 5432 (PostgreSQL)
  • An A DNS record redirects key.xxx.com -> <firewall IP>

My question is, which server should I encrypt - the firewall server or the internal server?

I followed this tutorial to setup let's encrypt, certbot and nginx through docker-compose on the internal server, but the http-01 challange failed with an error:

Detail: <firewall IP>: Fetching http://key.xxx.com/.well-known/acme-challenge/XI0xDGa0cCFjxUyyW32glRv7kbtJ_g60GMjxY3KVoV0: Timeout during connect (likely firewall problem)

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

Hi @mittkkoo, and welcome to the LE community forum :slight_smile:

I would encrypt at the server.

Those two things contradict each other.
[the second says there is no rule allowing it]

4 Likes

Thank you for welcoming! This is indeed my first touch with let's encrypt.

Those two things contradict each other.
[the second says there is no rule allowing it]

The firewall rule works. I can call the firewall IP or the DNS record from everywhere and it will show the landing page of Keycloak which resides on 10.0.0.7:8080.

1 Like

When you opened this thread in the Help section, you should have been provided with a questionnaire. Maybe you didn't get it somehow (which is weird), or you've decided to delete it. In any case, all the answers to this questionnaire are required:


Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:

I ran this command:

It produced this output:

My web server is (include version):

The operating system my web server runs on is (include version):

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don't know):

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

3 Likes

Obviously not from where LE is trying to reach your system.

Please answer the questions above so that we may be able to help you further.

4 Likes

My domain is: xx.isomorph.ch

I ran this command: GitHub - wmnnd/nginx-certbot: Boilerplate configuration for nginx and certbot with docker-compose
It produced this output:

Existing data found for xx.isomorph.ch. Continue and replace existing certificate? (y/N) y
### Creating dummy certificate for xx.isomorph.ch ...
[+] Building 0.0s (0/0)                                                                                                                                                                                                                     
[+] Building 0.0s (0/0)                                                                                                                                                                                                                     
Generating a RSA private key
...........................................................................................................................................................................................++++
.....................................................................................................................................................................................................................................................++++
writing new private key to '/etc/letsencrypt/live/xx.isomorph.ch/privkey.pem'
-----

### Starting nginx ...
[+] Building 0.0s (0/0)                                                                                                                                                                                                                     
[+] Running 1/1
 ✔ Container git_nginx-certbot-nginx-1  Started                                                                                                                                                                                        0.8s 

### Deleting dummy certificate for xx.isomorph.ch ...
[+] Building 0.0s (0/0)                                                                                                                                                                                                                     
[+] Building 0.0s (0/0)                                                                                                                                                                                                                     

### Requesting Let's Encrypt certificate for xx.isomorph.ch ...
[+] Building 0.0s (0/0)                                                                                                                                                                                                                     
[+] Building 0.0s (0/0)                                                                                                                                                                                                                     
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Requesting a certificate for xx.isomorph.ch

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
  Domain: xx.isomorph.ch
  Type:   connection
  Detail: 23.88.x.x: Fetching http://xx.isomorph.ch/.well-known/acme-challenge/K4AUVxS4rCl839vTBmxxxU3IlICAmixxxx: Timeout during connect (likely firewall problem)

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

### Reloading nginx ...
2023/07/11 21:03:59 [emerg] 9#9: cannot load certificate "/etc/letsencrypt/live/xx.isomorph.ch/fullchain.pem": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/letsencrypt/live/xx.isomorph.ch/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)
nginx: [emerg] cannot load certificate "/etc/letsencrypt/live/xx.isomorph.ch/fullchain.pem": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/letsencrypt/live/xx.isomorph.ch/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)

My web server is (include version): nginx/1.15.12
The operating system my web server runs on is (include version): Alpine Linux v3.9
My hosting provider, if applicable, is: Hetzner
I can login to a root shell on my machine (yes or no, or I don't know): Yes
I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No
The version of my client is: certbot 2.6.0

Have you changed your DNS since this post? Because that domain name has the DNS proxied in Cloudflare so uses its CDN service.

I don't see any timeout as described by that error.

I also don't see any cert problem. Can you explain what the issue is?

4 Likes

Is that a Cloudflare IP?

2 Likes

50 % chance it's Hetzner.

Edit: I'm 99 % sure it's Hetzner, all the other stuff in the block are from all kind of non-significant smaller things.

4 Likes

Yes, Hetzner. Is this important?

The domain I want to certify is keyc.isomorph.ch. Currently I managed to do it (to some extend) through a self-signed certificate. You can see it. But the problem persists. The http-01 challenge of certbot still fails.
I also uploaded a dummy file under http://keyc.isomorph.ch/.well-known/acme-challenge/test-challenge.txt. I don't understand why, but I can access it through a browser, but not for example through curl. I think this pretty much replicates the error which certbot experiences.

You should show the certbot log file - it may help us better understand why it fails.

4 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.