Subdomain mess-up with two local IPs

I’m at my wit’s end and have thoroughly messed up my letsencrypt/certbot setup, to the effect that neither my domain nor subdomain work anymore.

The scenario:
I have a domain (velko.men) which I self-host, with DNS services provided by the Norwegian webhost Domeneshop.
The main domain points to one local IP. The machine on this IP serves my Nextcloud, we can call it Box 1 for clarity. This is an Odroid ARM box, with Nextcloudpi installed on top of Armbian Buster. Box 1 has had a working Letsencrypt setup for some weeks.

I thought I’d like to self-host a web radio station on its own subdomain, so I installed regular Ubuntu server on an old laptop (Box 2), installed Azuracast via Docker, changed my DNS setup to add a subdomain (hjarteleg.velko.men) and set up reverse proxy on Box 1 to point to Box 2. I did this by adding a “domain.conf” file in /etc/apache2/sites-available:

<VirtualHost *:80>    
        ServerName hjarteleg.velko.men    
        <Proxy *>
                Order deny,allow
                Allow from all
        </Proxy>    
        <Location />
                ProxyPass http://LAN_IP_HERE/
                ProxyPassReverse http://LAN_IP_HERE/
        </Location>    
</VirtualHost>
<VirtualHost *:443>    
        ServerName hjarteleg.velko.men    
        <Proxy *>
                Order deny,allow
                Allow from all
        </Proxy>    
        <Location />
                ProxyPass https://LAN_IP_HERE/
                ProxyPassReverse https://LAN_IP_HERE/
        </Location>    
</VirtualHost>

I followed this by sudo apache2ctl -k graceful on Box 1 to make Apache load the new config.
I also ran sudo a2enmod proxy_http and sudo a2enmod proxy to make sure Apache would function as a reverse proxy.

Then, on Box2, I ran the recommended certbot command inside Docker:
./docker.sh letsencrypt-create hjarteleg.velko.men
This seems to have been successful, but after this, none of my domains have worked. The Firefox error is SSL_ERROR_RX_RECORD_TOO_LONG.

To overcome the problem, I have tried

  1. Uninstalling Azuracast on Box 2 (and, I hoped, also the certificate I obtained) and reinstalling. No change.
  2. Trying to make a wildcard DNS record on Box 1, using my webhost’s certbot DNS plugin and their API keys they provided. The result seems to have been successful, but none of my domains work.
  3. Checked which is the default domain on Box 1 with apache2ctl -S. The result indicated that my new subdomain, hjarteleg.velko.men, was the new default domain.
  4. Tried to rename the files in /etc/apache2/sites-available on Box 1 to make the default domain appear first, because I read that Apache reads the files in alphabetical order(!), and restarted Apache. The result of apache2ctl -S now omits the subdomain altogether:
    VirtualHost configuration:
    *:80 localhost (/etc/apache2/sites-enabled/000-default.conf:1)
    *:4443 localhost (/etc/apache2/sites-enabled/ncp.conf:2)
    ServerRoot: “/etc/apache2”
    Main DocumentRoot: “/var/www/html”
    Main ErrorLog: “/var/log/apache2/error.log”
    Mutex default: dir="/var/run/apache2/" mechanism=default
    Mutex watchdog-callback: using_defaults
    Mutex rewrite-map: using_defaults
    Mutex ssl-stapling-refresh: using_defaults
    Mutex ssl-stapling: using_defaults
    Mutex proxy: using_defaults
    Mutex ssl-cache: using_defaults
    PidFile: “/var/run/apache2/apache2.pid”
    Define: DUMP_VHOSTS
    Define: DUMP_RUN_CFG
    Define: MODSEC_2.5
    Define: MODSEC_2.9
    User: name=“www-data” id=33
    Group: name=“www-data” id=33

Needless to say, none of my domains work, and I am at loss. Help is appreciated.

1 Like

I see two problems:

  • The apache2ctl -S output doesn’t show your hjarteleg.velko.men configs.
    You may need to enable that config file(s)
  • SSL_ERROR_RX_RECORD_TOO_LONG
    [not sure what the exact fix is in Apache, but in nginx I would try increasing client_max_body_size]
1 Like

http://hjarteleg.velko.men:443/

It's a plain http server.

1 Like

OK so that goes to problem #1:

a2ensite <site> where '<site>' is the name of your site's Virtual Host configuration file, located in /etc/apache2/sites-available/, without the .conf extension.

1 Like

OP @eivind is indeed missing some SSLEngine on directive.

My advice is: remove the port 443 virtualhost block, and let certbot create it anew.

1 Like

Not too sure, because:

So he did manage to get it to do TLS already.
But now somehow broke the whole thing...
Perhaps with (renaming mistake):

1 Like

Not right now: https://velko.men/

@eivind, you have two options:

  • use tls on box2 and a stream proxy on box1
  • do all tls on box1 and go unencrypted from box1 to box2, using an http proxy.

the first mode is a lot more complicated. I’d suggest generating a certificate for box2 on box1, and then going unencrypted on your local network. (edit: you can have both go encrypted if you find a way to make both machines validate, using --apache on box1 should be enough)

I think he tried to do all TLS on box one.
But may have forgot to enable the newly added config file.
[placing a config file in the sites-available folder doesn’t always automatically enable it]
[placing a config file in the sites-enabled folder may have that effect…]
See: a2ensite

doesn't look like it

Thanks all for your suggestions. I have now tried

  1. Removing the https virtualhost block from my (newly renamed) 002-hjarteleg.velko.men.conf file
  2. sudo a2ensite 002-hjarteleg.velko.men
  3. sudo systemctl reload apache2

apache2ctl -S now shows

VirtualHost configuration:
*:80                   is a NameVirtualHost
         default server localhost (/etc/apache2/sites-enabled/000-default.conf:1)
         port 80 namevhost localhost (/etc/apache2/sites-enabled/000-default.conf:1)
         port 80 namevhost hjarteleg.velko.men (/etc/apache2/sites-enabled/002-hjarteleg.velko.men.conf:1)
*:4443                 localhost (/etc/apache2/sites-enabled/ncp.conf:2)
ServerRoot: "/etc/apache2"
Main DocumentRoot: "/var/www/html"
Main ErrorLog: "/var/log/apache2/error.log"
Mutex ssl-stapling: using_defaults
Mutex proxy: using_defaults
Mutex ssl-cache: using_defaults
Mutex default: dir="/var/run/apache2/" mechanism=default 
Mutex watchdog-callback: using_defaults
Mutex rewrite-map: using_defaults
Mutex ssl-stapling-refresh: using_defaults
PidFile: "/var/run/apache2/apache2.pid"
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
Define: MODSEC_2.5
Define: MODSEC_2.9
User: name="www-data" id=33
Group: name="www-data" id=33

The contents of /etc/apache2/sites-available is
-rw-r–r-- 1 root root 378 Mar 3 18:07 000-default.conf
-rw-r–r-- 1 root root 677 Mar 3 18:06 001-nextcloud.conf
-rw-r–r-- 1 root root 332 Mar 20 17:12 002-hjarteleg.velko.men.conf
-rw-r–r-- 1 root root 6.2K Apr 2 2019 default-ssl.conf
-rw-r–r-- 1 root root 495 Mar 3 14:41 ncp-activation.conf
-rw-r–r-- 1 root root 1.1K Mar 3 14:41 ncp.conf

To @9peppe: I had hoped that the certbot commands I issued on Box2 would have lost their effect when I un- and reinstalled Azuracast on it.

did you? ok. should be fine.

now go in ncp.conf and 000-default.conf and set proper ServerName directives

after that, running certbot --apache should solve all your problems.

1 Like

Thanks again! This is very helpful.

The contents of 000-default.conf are now

<VirtualHost _default_:80>
  DocumentRoot /var/www/nextcloud
  <IfModule mod_rewrite.c>
    RewriteEngine On
    RewriteCond %{HTTPS} !=on
    RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1 [R,L]
  </IfModule>
RewriteCond %{SERVER_NAME} =velko.men [OR]
RewriteCond %{SERVER_NAME} =velko.men_
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>

So the second SERVER_NAME directive should be hjarteleg.velko.men?

no. leave that alone.

only edit when you see stuff like ServerName localhost, replace with the actual domain you want the service to respond on.

1 Like

The /etc/sites-available/ncp.conf file is

Listen 4443
<VirtualHost _default_:4443>
  DocumentRoot /var/www/ncp-web
  SSLEngine on
  SSLCertificateFile      /etc/ssl/certs/ssl-cert-snakeoil.pem
  SSLCertificateKeyFile /etc/ssl/private/ssl-cert-snakeoil.key

  # 2 days to avoid very big backups requests to timeout
  TimeOut 172800

  <IfModule mod_authnz_external.c>
    DefineExternalAuth pwauth pipe /usr/sbin/pwauth
  </IfModule>

</VirtualHost>
<Directory /var/www/ncp-web/>

  AuthType Basic
  AuthName "ncp-web login"
  AuthBasicProvider external
  AuthExternal pwauth

  SetEnvIf Request_URI "^" noauth
  SetEnvIf Request_URI "^index\.php$" !noauth
  SetEnvIf Request_URI "^/$" !noauth
  SetEnvIf Request_URI "^/wizard/index.php$" !noauth
  SetEnvIf Request_URI "^/wizard/$" !noauth

  <RequireAll>

   <RequireAny>
      Require host localhost
      Require local
      Require ip 192.168
      Require ip 172
      Require ip 10
      Require ip fe80::/10
      Require ip fd00::/8
   </RequireAny>

   <RequireAny>
      Require env noauth
      Require user ncp
   </RequireAny>

  </RequireAll>

</Directory>

This is the file that seems to give me velko.men:4443, which is the local administration page that is only reachable from the inside. From the outside, velko.men should point to the Nextcloud instance. I would think that /etc/sites-available/001-nextcloud.conf is the file for that:

<IfModule mod_ssl.c>
  <VirtualHost _default_:443>
    DocumentRoot /var/www/nextcloud
ServerName velko.men_
    CustomLog /var/log/apache2/nc-access.log combined
    ErrorLog  /var/log/apache2/nc-error.log
    SSLEngine on
    ServerAlias      velko.men
    SSLCertificateFile /etc/letsencrypt/live/velko.men/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/velko.men/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
  </VirtualHost>
  <Directory /var/www/nextcloud/>
    Options +FollowSymlinks
    AllowOverride All
    <IfModule mod_dav.c>
      Dav off
    </IfModule>
    LimitRequestBody 0
    SSLRenegBufferSize 10486000
  </Directory>
</IfModule>
1 Like

I believe you're right.

Why the underscore? Because it conflicts with ServerAlias? You can remove serveralias and the underscore.

ncp.conf is fine.

now create a new .conf like the one in your first post, but with port 80 only. then run certbot.

1 Like

Thanks a million :slight_smile: Will try once I’m (more) sure I don’t mess up things even more. The underscore in 000-default.conf is there by default, so I can’t tell what it is for.

Note that I am referring to files in /etc/apache2/sites-available, not in …/sites-enabled. I checked that 000-default.conf is the same in the sites-enabled folder.

1 Like

files in sites-enabled are supposed to be symlinks to files in sites-avaliable

1 Like

I tried and cancelled:

$ sudo certbot --apache
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache

Which names would you like to activate HTTPS for?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: hjarteleg.velko.men
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): c

I cancelled because I would, of course, like https for both the main and sub-domain…

1 Like

do you want a certificate for both, or one certificate for each?

it’s fine to tell certbot to only make that, if you want one certificate for each.

otherwise you need to look in the apache config and understand why is certbot not recognizing your domain (probably the underscore) – or you can just type the domains instead of numbers.

2 Likes

PROBLEM SOLVED thanks to @9peppe’s competent and friendly help. Thank you very much! Steps to solve the problem:

$ sudo a2ensite 001-nextcloud
Enabling site 001-nextcloud.
To activate the new configuration, you need to run:
  systemctl reload apache2
$ sudo systemctl reload apache2
$ sudo certbot --apache

https now works! Yay!

1 Like