SSL Cert Renew Process

I am no longer able to access my website remotely and/or internally after a cert renewal request. I host the server at home and seem to have screwed up access by choosing to renew the SSL cert.
I issued the command “certbot-auto -d lab.addmoreroutes.com” and it screwed up my access. I was given the option to choose between two options (Redirect/No Redirect) after inputting that command.
I selected “Redirect” and the website was down shortly afterwards. I am now brought to a default Ubuntu page after choosing that option.Let me know what are the quickest steps to rebuild from scratch or restore from a previous date/time. I am not that great using Linux so please forgive me for any stupid questions.

Overview
url:lab.addmoreroutes.com
What did I do just before it crashed:certbot-auto -d lab.addmoreroutes.com
OS Version:Ubuntu 16.04.6 LTS
The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot): 0.39.0
Can I log into Root Shell ? Yes

*Error Output*
    root@eve-ng:~# certbot-auto  -d lab.addmoreroutes.com
    Upgrading certbot-auto 0.36.0 to 0.39.0...
    Replacing certbot-auto...
    Creating virtual environment...
    Installing Python packages...
    Installation succeeded.
    Saving debug log to /var/log/letsencrypt/letsencrypt.log
    Plugins selected: Authenticator apache, Installer apache
    Cert is due for renewal, auto-renewing...
    Renewing an existing certificate
    Performing the following challenges:
    http-01 challenge for lab.addmoreroutes.com
    Waiting for verification...
    Cleaning up challenges
    Created an SSL vhost at /etc/apache2/sites-available/unetlab-le-ssl.conf
    Deploying Certificate to VirtualHost /etc/apache2/sites-available/unetlab-le-ssl.conf
    Enabling available site: /etc/apache2/sites-available/unetlab-le-ssl.conf

    Please choose whether or not to redirect HTTP traffic to HTTPS, removing HTTP access.
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    1: No redirect - Make no further changes to the webserver configuration.
    2: Redirect - Make all requests redirect to secure HTTPS access. Choose this for
    new sites, or if you're confident your site works on HTTPS. You can undo this
    change by editing your web server's configuration.
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 2
    Added an HTTP->HTTPS rewrite in addition to other RewriteRules; you may wish to check for overall consistency.
    Redirecting vhost in /etc/apache2/sites-enabled/unetlab.conf to ssl vhost in /etc/apache2/sites-available/unetlab-le-ssl.conf

    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    Your existing certificate has been successfully renewed, and the new certificate
    has been installed.

    The new certificate covers the following domains: https://lab.addmoreroutes.com

    You should test your configuration at:
    https://www.ssllabs.com/ssltest/analyze.html?d=lab.addmoreroutes.com
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

    IMPORTANT NOTES:
     - Congratulations! Your certificate and chain have been saved at:
       /etc/letsencrypt/live/lab.addmoreroutes.com/fullchain.pem
       Your key file has been saved at:
       /etc/letsencrypt/live/lab.addmoreroutes.com/privkey.pem
       Your cert will expire on 2020-02-02. To obtain a new or tweaked
       version of this certificate in the future, simply run certbot-auto
       
       again with the "certonly" option. To non-interactively renew *all*
       of your certificates, run "certbot-auto renew"
     - Some rewrite rules copied from
       /etc/apache2/sites-enabled/unetlab.conf were disabled in the vhost
       for your HTTPS site located at
       /etc/apache2/sites-available/unetlab-le-ssl.conf because they have
       the potential to create redirection loops.
     - If you like Certbot, please consider supporting our work by:

       Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
       Donating to EFF:                    https://eff.org/donate-le

    root@*****~# sudo /usr/bin/letsencrypt renew
    sudo: /usr/bin/letsencrypt: command not found

     /etc/apache2/sites-available/unetlab-le-ssl.conf
root@eve-ng:/etc/apache2/sites-available# sudo vi unetlab-le-ssl.conf 

<IfModule mod_ssl.c>
<VirtualHost *:443>
RewriteEngine On
# Some rewrite rules in this file were disabled on your HTTPS site,
# because they have the potential to create redirection loops.

# RewriteCond %{HTTPS} !=on
# RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1 [R,L] 
ServerName lab.addmoreroutes.com
Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateFile /etc/letsencrypt/live/lab.addmoreroutes.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/lab.addmoreroutes.com/privkey.pem
</VirtualHost>
</IfModule>
<IfModule mod_ssl.c>
<VirtualHost *:80>
RewriteEngine On
# Some rewrite rules in this file were disabled on your HTTPS site,
# because they have the potential to create redirection loops.

# RewriteCond %{HTTPS} !=on
# RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1 [R,L] 
</VirtualHost>
</IfModule>
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
Type  :quit<Enter>  to exit Vim

If the default Ubuntu website has taken over your installation, it shouldn’t be too difficult to disable it and get your real site back.

What is the current list of virtualhosts in Apache reported by this command:

apachectl -t -D DUMP_VHOSTS

root@XXXXXXX:~# apachectl -t -D DUMP_VHOSTS
VirtualHost configuration:
*:443 is a NameVirtualHost
default server eve-ng.example.com (/etc/apache2/sites-enabled/eveng-ssl.conf:2)
port 443 namevhost eve-ng.example.com (/etc/apache2/sites-enabled/eveng-ssl.conf:2)
port 443 namevhost lab.addmoreroutes.com (/etc/apache2/sites-enabled/unetlab-le-ssl.conf:2)
*:80 is a NameVirtualHost
default server eve-ng.example.com (/etc/apache2/sites-enabled/unetlab-le-ssl.conf:16)
port 80 namevhost eve-ng.example.com (/etc/apache2/sites-enabled/unetlab-le-ssl.conf:16)
port 80 namevhost eve-ng.example.com (/etc/apache2/sites-enabled/unetlab.conf:24)

Oh. I’m not sure what’s going on there.

It seems like your configuration is lacking the DocumentRoot for the domain? So it’s defaulting to the default Ubuntu Apache one.

You can find backups of your Apache configuration, before Certbot modified it, in /var/lib/letsencrypt/backups/. The directory names are the timestamps of when the backup was generated.

I would be curious to see whether the backups have versions of the config that contain DocumentRoot …

grep -Ri documentroot /var/lib/letsencrypt/backups/

root@eve-ng:~# grep -Ri documentroot /var/lib/letsencrypt/backups/
/var/lib/letsencrypt/backups/1564724652.18/eveng-ssl.conf_0: DocumentRoot /opt/unetlab/html/

Is the website you are expecting to see at “lab.addmoresroutes.com” the “Emulated Virtual Environment” login page?

Or is it a different one?

yes it is the login page

Okay.

I’m not sure why your certificate renewal broke your site, but I can suggest a way to fix it.

Open up /etc/apache2/sites-enabled/unetlab-le-ssl.conf. Where it says:

ServerName eve-ng.example.com

add below it:

ServerAlias lab.addmoreroutes.com

and then:

a2dissite unetlab
a2dissite unetlab-le-ssl

and try reload Apache.

If you do that, you should be able to access the login via https://lab.addmoreroutes.com .

Once that is working, you can reconsider trying the redirect again.

I am having some issues restarting Apache. What is the exact command ?

root@eve-ng:~# sudo /etc/init.d/apache2 reload
Reloading apache2 configuration (via systemctl): apache2.serviceJob for apache2.service failed because the control process exited with error code. See "systemctl status apache2.service" and "journalctl -xe" for details.
failed!

That’s okay. It looks like you have a configuration issue, but you can identify it by running:

apachectl -t

Once we know what it is, we can repair it and reload Apache properly.

root@eve-ng:~# apachectl -t
AH00526: Syntax error on line 11 of /etc/apache2/sites-enabled/unetlab-le-ssl.conf:
Invalid command 'a2dissite', perhaps misspelled or defined by a module not included in the server configuration
Action '-t' failed.
The Apache error log may have more information.

I see. Those a2dissite things were commands I wanted you to run, not lines to add to the configuration file. I can see that the way I posted it, it was really unclear.

Open up /etc/apache2/sites-enabled/unetlab-le-ssl.conf and remove the a2dissite lines.

Then, in your terminal (not in a text editor), run:

a2dissite unetlab-le-ssl

and then try reload Apache.

I removed the a2 lines from the config, ran them on the terminal and reloaded apache. Still no dice

I see that the default Ubuntu page is gone now, but replaced with a forbidden page.

Could you show your vhosts again?

apachectl -t -D DUMP_VHOSTS

root@eve-ng:~# apachectl -t -D DUMP_VHOSTS
VirtualHost configuration:
*:443 eve-ng.example.com (/etc/apache2/sites-enabled/eveng-ssl.conf:2)

I was sure that would work :sweat:. Sorry.

Could you post the contents of /etc/apache2/sites-enabled/eveng-ssl.conf? I thought it would be pointing to the /opt/unetlab/html document root …

<IfModule mod_ssl.c>
    <VirtualHost _default_:443>
        ServerAdmin webmaster@localhost
        DocumentRoot /opt/unetlab/html/
        ErrorLog /opt/unetlab/data/Logs/ssl-error.log
        CustomLog /opt/unetlab/data/Logs/ssl-access.log combined
        Alias /Exports /opt/unetlab/data/Exports
        Alias /Logs /opt/unetlab/data/Logs
        SSLEngine on
        SSLProxyEngine on
        SSLCertificateFile /etc/letsencrypt/live/lab.addmoreroutes.com/fullchain.pem
        SSLCertificateKeyFile /etc/letsencrypt/live/lab.addmoreroutes.com/privkey.pem
        SSLProtocol         all -SSLv3 -TLSv1 -TLSv1.1
        # Many ciphers defined here require a modern version (1.0.1+) of OpenSSL. Some
        # require OpenSSL 1.1.0, which as of this writing was in pre-release.
        SSLCipherSuite      ********LEFT OUT***************************
        SSLHonorCipherOrder on
        SSLCompression      off
        SSLSessionTickets   off
        <FilesMatch "\.(cgi|shtml|phtml|php)$">
                SSLOptions +StdEnvVars
        </FilesMatch>
        <Directory /usr/lib/cgi-bin>
                SSLOptions +StdEnvVars
        </Directory>
        <Location /html5/>
                Order allow,deny
                Allow from all
                ProxyPass http://127.0.0.1:8080/guacamole/ flushpackets=on
                ProxyPassReverse http://127.0.0.1:8080/guacamole/
        </Location>

        <Location /html5/websocket-tunnel>
                Order allow,deny
                Allow from all
                ProxyPass ws://127.0.0.1:8080/guacamole/websocket-tunnel
                ProxyPassReverse ws://127.0.0.1:8080/guacamole/websocket-tunnel

        <FilesMatch "\.(cgi|shtml|phtml|php)$">
                SSLOptions +StdEnvVars
        </FilesMatch>
        <Directory /usr/lib/cgi-bin>
                SSLOptions +StdEnvVars
        </Directory>
        <Location /html5/>
                Order allow,deny
                Allow from all
                ProxyPass http://127.0.0.1:8080/guacamole/ flushpackets=on
                ProxyPassReverse http://127.0.0.1:8080/guacamole/
        </Location>

        <Location /html5/websocket-tunnel>
                Order allow,deny
                Allow from all
                ProxyPass ws://127.0.0.1:8080/guacamole/websocket-tunnel
                ProxyPassReverse ws://127.0.0.1:8080/guacamole/websocket-tunnel
        </Location>
        ProxyPass /janus-ws ws://127.0.0.1:8188/
        ProxyPassReverse /janus-ws ws://127.0.0.1:8188/
        ProxyPass /chat-ws ws://127.0.0.1:9090/
        ProxyPassReverse /chat-ws ws://127.0.0.1:9090/
    </VirtualHost>
</IfModule>

I’m not sure why the default page gives forbidden. You may be able to identify what the cause of that is by running something like:

tail -n 20 /opt/unetlab/data/Logs/ssl-error.log

Visiting https://lab.addmoreroutes.com/html5/ produces a login page, which is a good sign that something is working.

[Tue Nov 05 00:08:27.979248 2019] [authz_core:error] [pid 11262] [client 14.x.x.x:34886] AH01630: client denied by server configuration: /opt/unetlab/html/favicon.ico
[Tue Nov 05 00:08:48.548403 2019] [authz_core:error] [pid 11185] [client 14.x.x.x:34900] AH01630: client denied by server configuration: /opt/unetlab/html/
[Tue Nov 05 00:10:28.097669 2019] [authz_core:error] [pid 14479] [client 207.x.x.x:1376] AH01630: client denied by server configuration: /opt/unetlab/html/