Key update via proxy server

Help
1

I'm having trouble updating Letsencrypt's key.
The configuration is connecting to the server tvc.wtnbakr.ntm200.com through a reverse proxy (242.wtnbakr.ntm200.com) and trying to rekey the server.
I was able to get the key with this configuration, but I can't renew it after 3 months.
Looking at it with Wireshark, I found that “404 Not Found (text/html)” was returned, although it was indeed reading the above file from letsencrypt.
Looking at /var/www/html/, the directory /.well-known/acme-challenge is not created in the server.

Please let me know if you have any idea what is causing this.

My domain is: tvc2.wtnbakr.ntm200.com

I ran this command: sudo certbot -v renew --dry-run

It produced this output:
Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/tvc2.wtnbakr.ntm200.com.conf

Certificate is due for renewal, auto-renewing...
Plugins selected: Authenticator webroot, Installer None
Simulating renewal of an existing certificate for tvc2.wtnbakr.ntm200.com
Performing the following challenges:
http-01 challenge for tvc2.wtnbakr.ntm200.com
Using the webroot path /var/www/html for all unmatched domains.
Waiting for verification...
Challenge failed for domain tvc2.wtnbakr.ntm200.com
http-01 challenge for tvc2.wtnbakr.ntm200.com

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
Domain: tvc2.wtnbakr.ntm200.com
Type: unauthorized
Detail: 153.126.137.242: Invalid response from http://tvc2.wtnbakr.ntm200.com/.well-known/acme-challenge/ED_MroTWnNdfbHuP29czSWN9z3oqWgzQmx0lsapb9sI: 404

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

Cleaning up challenges
Failed to renew certificate tvc2.wtnbakr.ntm200.com with error: Some challenges have failed.

All simulated renewals failed. The following certificates could not be renewed:
/etc/letsencrypt/live/tvc2.wtnbakr.ntm200.com/fullchain.pem (failure)

1 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

My web server is (include version): nginx/1.18.0 (ubuntu)

The operating system my web server runs on is (include version): Ubuntu20.04LTS

My hosting provider, if applicable, is:
SAKURA internet

I can login to a root shell on my machine (yes or no, or I don't know): yes.

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no.

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 1.31.0

2

Hi @AkiraWatanabe1, and welcome to the LE community forum :slight_smile:

Please show what the proxy is doing with requests on that FQDN.

3 Likes
3

The answer is below.
Note that tvc.wtnbakr.ntm200.com is an error of tvc2.wtnbakr.ntm200.com.

Definition of terms and system configuration:
tvc2 Application Server that needs a Certificate.
RP Reverse Proxy server.

| tvc2 | ---------- | RP | -------------| letsencrypt |
--------- tunnel ---------- ----------------

tvc2 and RP are connected by our own VPN (tunnel communication).
tvc2 is in the private address area, but thanks to our VPN, RP can query tvc2.
In the DNS settings, the IP address of "tvc2.wtnbakr.ntm200.com" is set to the global address of RP.
In the RP settings, with the apache2 reverse proxy, ProxyPass is set as http://tvc2.wtnbakr.ntm200.com.
If "ntm200.com" is included in the FQDN, the communication automatically becomes a tunnel, and this inquiry will reach tvc2.
This way, we can get letsencrypt's public key certificate even though tvc2 has a private address.
Actually, I obtained a certificate in this way and have operated tvc2.
This is the first time of the key exchange, and I tried to rekey, but the reported error is output and the update is not possible.

4

Please show the actual RP settings/configuration file.

3 Likes
5

The RP (Reverse Proxy server) configuration file is below.
I made a file “vnetreverse.conf” under /etc/apache2/sites-available/.

The contents of vnetreverse.conf:

<VirtualHost *:80>
  ProxyPreserveHost Off
  ServerName tvc2.wtnbakr.ntm200.com
  ProxyPass / http://tvc2.wtnbakr.ntm200.com/
  ProxyPassReverse / http://tvc2.wtnbakr.ntm200.com/
  ErrorLog ${APACHE_LOG_DIR}/wtnbakr_error.log
</VirtualHost>

I imagine the cause is not the RP side, but the tvc side.
This is because tvc2 received the following packet and responded with an error.
The contents of the HTTP packet received by tvc2:
GET /.well-known/acme-challenge/Sk2KCWMrpA1_Zm2be5H0ChDh6Q6r7IhckiZtfl8gkBg HTTP/1.1

tvc2's response:
HTTP/1.1 404 Not Found (text/html)

Looking at the tvc2 files, I cannot find the “.well-known” directory in the following location.
/var/www/html

6
  1. Which server are you trying to run Certbot on? RP or tvc2?
  2. Where is the nginx in all of this? Do you have nginx AND Apache on the RP?
4 Likes
7

I tried to run Certbot on tvc2.
RP has Apache only, and tvc2 has nginx only.

8

I'm lost.

$ curl -i tvc2.wtnbakr.ntm200.com
HTTP/1.1 301 Moved Permanently
Date: Wed, 19 Oct 2022 02:54:42 GMT
Server: nginx/1.18.0 (Ubuntu)
Content-Type: text/html
Content-Length: 178
Location: https://tvc2.wtnbakr.ntm200.com/

If Apache was on the reverse proxy, then we would see Server: Apache in the response headers for the domain.

Am I reading the network diagram backwards?

nginx (tvc2) <-- Apache (RP) <--- Internet

Or is there some kind of routing magic happening at a lower level than HTTP?

3 Likes
9

A confirmation HTTP query will be sent from the Internet to Apache (RP) (tvc.wtnbakr.ntm200.com) (GET /.well-known/---).
The RP settings says that it has to forward to the same FQDN (tvc2.wtnbakr.ntm200.com).
Now our VPN-like application on the RP checks the FQDN and decides that it should go through the tunnel and the HTTP query is sent to the tvc2 side.
This mechanism may be what you call routing magic.
In fact, we have confirmed that HTTP queries are reaching tvc2.
At the time of newly getting the certificate, the same configuration and the same settings worked well.

11

Then how is nginx responding?

curl -Ii tvc2.wtnbakr.ntm200.com/.well-known/acme-challenge/Test_File-1234
HTTP/1.1 404 Not Found
Date: Wed, 19 Oct 2022 04:17:03 GMT
Server: nginx/1.18.0 (Ubuntu)
Content-Type: text/html
Content-Length: 162
2 Likes
12

If the nginx server is where you're requesting the certificate and that's where the HTTP request is landing, then the place you need to be looking is in your nginx configuration.

You'd need to have a virtualhost that has its root configured to /var/www/html, for your current attempt to succeed.

Something approximately like:

server {
    server_name tvc2.wtnbakr.ntm200.com
    listen 80;
    root /var/www/html;

    # Alternatively something like: 
    location /.well-known/acme-challenge/ {
        root /var/www/html;
    }
}

I do worry about the next step though. The VPN routing might work for port 80 HTTP traffic, but for HTTPS traffic on port 443, you'd need to do SNI prereading to have the same effect.

3 Likes
13

The Nginx config file was like this:
tvc2 currently has jitsi-meet (a video conference server application) installed.
Root seems to have been changed to the following.
/usr/share/jitsi-meet.
Could this be the reason why the key cannot be renewed?

File location:
/etc/nginx/sites-enabled/tvc2.wtnbakr.ntm200.com.conf

Contents (partial)
server {
    listen 80;
    listen [::]:80;
    server_name tvc2.wtnbakr.ntm200.com;

    location ^~ /.well-known/acme-challenge/ {
        default_type "text/plain";
        root         /usr/share/jitsi-meet;
    }
    location = /.well-known/acme-challenge/ {
        return 404;
    }
    location / {
        return 301 https://$host$request_uri;
    }
}
server {
    listen 443 ssl;
    listen [::]:443 ssl;
    server_name tvc2.wtnbakr.ntm200.com;
14

Well, that is not great. Try the example location I posted.

3 Likes
15

The Nginx configuration file was rewritten by jitsi-meet.
If I change this setting, jitsi may stop working.
Were there other examples of certain applications changing root?

16

I don't quite follow all that is happening here but the above location is harmless. The = checks only for an exact match of the URI and returns a 404 if matched. No valid request will have only that URI so a 404 is ok in that case. The location is not necessary as nginx would respond 404 anyway.

3 Likes
17

Good catch.

However, we still need /.well-known/acme-challenge/ to be served from /var/www/html, because that's what Certbot is being told. I think adding the location is required here. You can use try_files if you want requests that don't match, to be sent to the Jitsi dcument root.

3 Likes
18

I didn't see this earlier. If this is the case, you can try:

certbot renew --cert-name tvc2.wtnbakr.ntm200.com --webroot -w /usr/share/jitsi-meet
4 Likes
19

Thank you for your various comments.

I asked the jitsi community, and succeeded in renewing by the following method.
The cause seems webroot.
I was taught it is possible to chang the webroot of jitsi with the following command.
ln -s /usr/share/jitsi-meet /var/www/html
After this change, renewing succeeded.

_az taught me how to renew by specifying webroot, so I tried it while the root is /usr/share/jitsi-meet, but I could not do it with the following error.

sudo certbot renew --cert-name tvc2.wtnbakr.ntm200.com --webroot /usr/share/jitsi-meet
usage:
certbot [SUBCOMMAND] [options] [-d DOMAIN] [-d DOMAIN] ...
Certbot can obtain and install HTTPS/TLS/SSL certificates. By default,
it will attempt to use a webserver both for obtaining and installing the
certificate.
certbot: error: unrecognized arguments: /usr/share/jitsi-meet

This time, I got the certificate while the root was /var/www/html.
After that, the root was switched by installing jitsi, but letsencrypt tried to renew without knowing this.
jitsi recommends using letsencrypt during installation, and the certificate can be renewed without any problems by this way.

1 Like
20

Glad you got a cert. I just want to explain this command error. It is missing a -w in front of the webroot folder. Should be like:

sudo certbot renew --cert-name tvc2.wtnbakr.ntm200.com --webroot -w /usr/share/jitsi-meet
3 Likes
21

I understood.
I appreciate your cooperation.

1 Like