How to force Apache 2.4 (httpd) to read fullchain.pem? certbot seems to work but I have an https:// issue X86_64 GNU/Linux (Linux 2 AMI)

OnEarth · May 1, 2021, 2:27pm

Hi,
Everything seems to work but I do have an error on line 22 (SSLCertificateFile /etc/letsencrypt/live/onearth.studio/fullchain.pem) in httpd-le-ssl.conf file (DocumentRoot "var/www/html") and do not understand why I have a Warning: DocumentRoot [/etc/httpd/var/www/html] does not exist. Why isn't it DocumentRoot [var/www/html]?

My domain is: https://www.onearth.studio

I ran this command:

apachectl -S

It produced this output:

AH00112:Warning: DocumentRoot [/etc/httpd/var/www/html] does not exist
AH00526: Syntax error on line 22 of /etc/httpd/conf/httpd-le-ssl.conf:
SSLCertificateFile: file '/etc/letsencrypt/live/onearth.studio/fullchain.pem' does not exist
or is empty

/etc/lestencrypt/live/onearth.studio/fullchain.pem exists and it is not empty

I ran this command:

sudo certbot renew --dry-run

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/onearth.studio.conf

Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator apache, Installer apache
Simulating renewal of an existing certificate for onearth.studio and www.onearth.studio
Performing the following challenges:
http-01 challenge for onearth.studio
http-01 challenge for www.onearth.studio
Waiting for verification...
Cleaning up challenges

new certificate deployed with reload of apache server; fullchain is
/etc/letsencrypt/live/onearth.studio/fullchain.pem

Congratulations, all simulated renewals succeeded:
/etc/letsencrypt/live/onearth.studio/fullchain.pem (success)

before I deployed http://www.onearth.studio (var/www/html/index.html) without any issue, since https I got the Apache test page instead!

My web server is (include version): Apache/2.4.46

The operating system my web server runs on is (include version): amzn2.x86_64 #1 SMP Mon A
pr 19 23:20:22 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

My hosting provider, if applicable, is: AWS

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 1.14.0

Thank you for your assistance,
Jan

Rip · May 1, 2021, 4:58pm

Hi @OnEarth and welcome to the community!

Crazy path huh?

Would you share the output of:

grep -ERi '/etc/httpd/var/www/html' /etc/apache2/

AND

certbot certificates

That should get us started.

griffin · May 1, 2021, 5:03pm

Welcome to the Let's Encrypt Community, Jan

In addition to what @Rip has requested, it would be great if you provided a couple of other things too.

What are the contents of these files?

/etc/httpd/conf/httpd.conf
/etc/httpd/conf/httpd-le-ssl.conf

What are the outputs of these commands?

sudo apachectl -S
sudo ls -lRa /etc/httpd/conf
sudo ls -lRa /etc/letsencrypt

I know that some of these might be quite long, but they will help to ensure things are clean. Please put three backticks above and below each content and output, like this:

```
content/output
```

OnEarth · May 1, 2021, 5:19pm

Hi Rip,

Thx for your answer, yes crazy path!

I ran this command:
grep -ERi '/etc/httpd/var/www/html' /etc/apache2/

It produced this output:
grep: /etc/apache2/: No such file or directory

I ran this command:
certbot certificates

It produced this output:

Found the following certs:
Certificate Name: onearth.studio
Serial Number: 419-----------------------------c8b1
Key Type: RSA
Domains: onearth.studio www.onearth.studio
Expiry Date: 2021-07-24 18:28:10+00:00 (VALID: 84 days)
Certificate Path: /etc/letsencrypt/live/onearth.studio/fullchain.pem
Private Key Path: /etc/letsencrypt/live/onearth.studio/privkey.pem

...

Thank you for your assistance,
Jan

griffin · May 1, 2021, 5:24pm

From the certificate history of onearth.studio, it looks like there was a Let's Encrypt certificate autorenewing beautifully for quite some time then in April an Amazon certificate appears. I'm guessing that you swapped hosting providers to AWS then the troubles started.

Rip · May 1, 2021, 5:24pm

OK so maybe:

~~grep -ERi '/etc/httpd/var/www/html' /etc/httpd/~~

OnEarth · May 1, 2021, 5:58pm

Thx griffin for your welcome!

content of httpd.conf:

ServerRoot "/etc/httpd"

<VirtualHost *:80>
RewriteEngine On
RewriteCond %{HTTP:X-Forwarded-Proto} =http
RewriteRule .* https://%{HTTP_Host}%{REQUEST_URI} [L,R=permanent]
<Directory "var/www/html">
    Options Indexes FollowSymLinks
    AllowOverride None
    Require all granted
</Directory>
</VirtualHost>

Include conf.modules.d/*.conf

User apache
Group apache

ServerAdmin root@localhost

<Directory />
    AllowOverride none
    Require all denied
</Directory>

<Directory "/var/www">
    AllowOverride None
    # Allow open access:
    Require all granted
</Directory>

<Directory "/var/www/html">
    Options Indexes FollowSymLinks
    AllowOverride None
    Require all granted
</Directory>

<IfModule dir_module>
    DirectoryIndex index.html
</IfModule>

<Files ".ht*">
    Require all denied
</Files>

ErrorLog "logs/error_log"

LogLevel warn
<IfModule log_config_module>
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common

<IfModule logio_module>
 # You need to enable mod_logio.c to use %I and %O
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" co$
</IfModule>
CustomLog "logs/access_log" combined
</IfModule>

<IfModule alias_module>
    ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"
</IfModule>

<Directory "/var/www/cgi-bin">
AllowOverride None
Options None
Require all granted
</Directory>

<IfModule mime_module>
TypesConfig /etc/mime.types
AddType application/x-compress .Z
AddType application/x-gzip .gz .tgz
AddType text/html .shtml
AddOutputFilter INCLUDES .shtml
</IfModule>

AddDefaultCharset UTF-8

<IfModule mime_magic_module>
MIMEMagicFile conf/magic
</IfModule>

EnableSendfile on

<IfModule mod_http2.c>
Protocols h2 h2c http/1.1
</IfModule>

IncludeOptional conf.d/*.conf

Include /etc/httpd/conf/httpd-le-ssl.conf

Content of httpd-le-ssl.conf:

<IfModule mod_ssl.c>
<VirtualHost *:443>
ServerAlias *
DocumentRoot "var/www/html"
RewriteEngine On
<Directory "var/www/html">
Options Indexes FollowSymLinks
AllowOverride None
Require all granted
</Directory>

Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateFile /etc/letsencrypt/live/onearth.studio/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/onearth.studio/privkey.pem
</VirtualHost>
</IfModule>

I ran this command:
sudo apachectl -S

It produced this output:

AH00112: Warning: DocumentRoot [/etc/httpd/var/www/html] does not exist
VirtualHost configuration:
*:80                   ip-17.-..-..-..3.us-east-2.compute.internal (/etc/httpd/conf/http
d.conf:59)
*:443                  is a NameVirtualHost
         default server ip-17.-..-..-..3.us-east-2.compute.internal (/etc/httpd/conf.d/s
sl.conf:56)
         port 443 namevhost ip-17.-..-..-..3.us-east-2.compute.internal (/etc/httpd/conf
.d/ssl.conf:56)
         port 443 namevhost ip-17.-..-..-..3.us-east-2.compute.internal (/etc/httpd/conf
/httpd-le-ssl.conf:2)
                 wild alias *
ServerRoot: "/etc/httpd"
Main DocumentRoot: "/etc/httpd/htdocs"
Main ErrorLog: "/etc/httpd/logs/error_log"
Mutex mpm-accept: using_defaults
Mutex cache-socache: using_defaults
Mutex authdigest-opaque: using_defaults
Mutex watchdog-callback: using_defaults
Mutex proxy-balancer-shm: using_defaults
Mutex rewrite-map: using_defaults
Mutex ssl-stapling-refresh: using_defaults
Mutex authdigest-client: using_defaults
Mutex lua-ivm-shm: using_defaults
Mutex ssl-stapling: using_defaults
Mutex proxy: using_defaults
Mutex authn-socache: using_defaults
Mutex ssl-cache: using_defaults
Mutex default: dir="/run/httpd/" mechanism=default 
PidFile: "/run/httpd/httpd.pid"
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
User: name="apache" id=48
Group: name="apache" id=48

I ran this command:
sudo ls -lRa /etc/httpd/conf

I produced this output:

/etc/httpd/conf:
total 76

drwxr-xr-x 2 root root   135 May  1 14:02 .
drwxr-xr-x 5 root root   105 May  1 14:02 ..
-rwxr-xr-x 1 root root 13218 May  1 17:39 httpd.conf
-rwxr-xr-x 1 root root 12059 Apr  5 15:04 httpd.conf.save
-rwxr-xr-x 1 root root 12231 Apr  9 09:32 httpd.conf.save.1
-rwxr-xr-x 1 root root 12357 Apr  9 11:16 httpd.conf.save.2
-rw-r--r-- 1 root root   786 Apr 30 21:01 httpd-le-ssl.conf
-rw-r--r-- 1 root root 13064 Aug 24  2020 magic

I ran this command:
sudo ls -lRa /etc/letsencrypt

It produced this output:

/etc/letsencrypt:
total 20
drwxr-xr-x  9 root root  190 May  1 17:14 .
drwxr-xr-x 87 root root 8192 Apr 25 17:26 ..
drwx------  4 root root   86 Apr 25 19:31 accounts
drwx------  3 root root   28 Apr 25 19:28 archive
drwxr-xr-x  2 root root  286 Apr 25 19:28 csr
drwx------  2 root root  286 Apr 25 19:28 keys
drwx------  3 root root   42 Apr 25 19:28 live
-rw-r--r--  1 root root  924 Apr 25 17:26 options-ssl-apache.conf
drwxr-xr-x  2 root root   33 Apr 25 19:28 renewal
drwxr-xr-x  5 root root   43 Apr 25 17:26 renewal-hooks
-rw-r--r--  1 root root   64 Apr 25 17:26 .updated-options-ssl-apache-conf-digest.txt

/etc/letsencrypt/accounts:
total 0
drwx------ 4 root root  86 Apr 25 19:31 .
drwxr-xr-x 9 root root 190 May  1 17:14 ..
drwx------ 3 root root  23 Apr 25 19:31 acme-staging-v02.api.letsencrypt.org
drwx------ 3 root root  23 Apr 25 17:26 acme-v02.api.letsencrypt.org

/etc/letsencrypt/accounts/acme-staging-v02.api.letsencrypt.org:
total 0
drwx------ 3 root root 23 Apr 25 19:31 .
drwx------ 4 root root 86 Apr 25 19:31 ..
drwx------ 3 root root 46 Apr 25 19:31 directory

/etc/letsencrypt/accounts/acme-staging-v02.api.letsencrypt.org/directory:
total 0
drwx------ 3 root root 46 Apr 25 19:31 .
drwx------ 3 root root 23 Apr 25 19:31 ..
drwx------ 2 root root 64 Apr 25 19:31 3d104ec8b6970a7532a098ae0583a631

/etc/letsencrypt/accounts/acme-staging-v02.api.letsencrypt.org/directory/3d104ec8b6970a7
532a098ae0583a631:
total 12
drwx------ 2 root root   64 Apr 25 19:31 .
drwx------ 3 root root   46 Apr 25 19:31 ..
-rw-r--r-- 1 root root  103 Apr 25 19:31 meta.json
-r-------- 1 root root 1632 Apr 25 19:31 private_key.json
-rw-r--r-- 1 root root   86 Apr 25 19:31 regr.json

/etc/letsencrypt/accounts/acme-v02.api.letsencrypt.org:
total 0
drwx------ 3 root root 23 Apr 25 17:26 .
drwx------ 4 root root 86 Apr 25 19:31 ..
drwx------ 3 root root 46 Apr 25 17:26 directory

/etc/letsencrypt/accounts/acme-v02.api.letsencrypt.org/directory:
total 0
drwx------ 3 root root 46 Apr 25 17:26 .
drwx------ 3 root root 23 Apr 25 17:26 ..
drwx------ 2 root root 64 Apr 25 17:26 66b7d472dc4d75d3b8ec7111b38fa804

/etc/letsencrypt/accounts/acme-v02.api.letsencrypt.org/directory/66b7d472dc4d75d3b8ec711
1b38fa804:
total 12
drwx------ 2 root root   64 Apr 25 17:26 .
drwx------ 3 root root   46 Apr 25 17:26 ..
-rw-r--r-- 1 root root  103 Apr 25 19:28 meta.json
-r-------- 1 root root 1632 Apr 25 17:26 private_key.json
-rw-r--r-- 1 root root   79 Apr 25 17:26 regr.json

/etc/letsencrypt/archive:
total 0
drwx------ 3 root root  28 Apr 25 19:28 .
drwxr-xr-x 9 root root 190 May  1 17:14 ..
drwxr-xr-x 2 root root  83 Apr 25 19:28 onearth.studio

/etc/letsencrypt/archive/onearth.studio:
total 16
drwxr-xr-x 2 root root   83 Apr 25 19:28 .
drwx------ 3 root root   28 Apr 25 19:28 ..
-rw-r--r-- 1 root root 1866 Apr 25 19:28 cert1.pem
-rw-r--r-- 1 root root 1586 Apr 25 19:28 chain1.pem
-rw-r--r-- 1 root root 3452 Apr 25 19:28 fullchain1.pem
-rw------- 1 root root 1704 Apr 25 19:28 privkey1.pem

/etc/letsencrypt/csr:
total 40
drwxr-xr-x 2 root root 286 Apr 25 19:28 .
drwxr-xr-x 9 root root 190 May  1 17:14 ..
-rw-r--r-- 1 root root 952 Apr 25 17:29 0000_csr-certbot.pem
-rw-r--r-- 1 root root 924 Apr 25 18:03 0001_csr-certbot.pem
-rw-r--r-- 1 root root 928 Apr 25 19:14 0002_csr-certbot.pem
-rw-r--r-- 1 root root 932 Apr 25 19:14 0003_csr-certbot.pem
-rw-r--r-- 1 root root 932 Apr 25 19:15 0004_csr-certbot.pem
-rw-r--r-- 1 root root 928 Apr 25 19:15 0005_csr-certbot.pem
-rw-r--r-- 1 root root 924 Apr 25 19:16 0006_csr-certbot.pem
-rw-r--r-- 1 root root 936 Apr 25 19:26 0007_csr-certbot.pem
-rw-r--r-- 1 root root 952 Apr 25 19:27 0008_csr-certbot.pem
-rw-r--r-- 1 root root 952 Apr 25 19:28 0009_csr-certbot.pem

/etc/letsencrypt/keys:
total 40
drwx------ 2 root root  286 Apr 25 19:28 .
drwxr-xr-x 9 root root  190 May  1 17:14 ..
-rw------- 1 root root 1704 Apr 25 17:29 0000_key-certbot.pem
-rw------- 1 root root 1708 Apr 25 18:03 0001_key-certbot.pem
-rw------- 1 root root 1704 Apr 25 19:14 0002_key-certbot.pem
-rw------- 1 root root 1708 Apr 25 19:14 0003_key-certbot.pem
-rw------- 1 root root 1708 Apr 25 19:15 0004_key-certbot.pem
-rw------- 1 root root 1704 Apr 25 19:15 0005_key-certbot.pem
-rw------- 1 root root 1704 Apr 25 19:16 0006_key-certbot.pem
-rw------- 1 root root 1704 Apr 25 19:26 0007_key-certbot.pem
-rw------- 1 root root 1704 Apr 25 19:27 0008_key-certbot.pem
-rw------- 1 root root 1704 Apr 25 19:28 0009_key-certbot.pem

/etc/letsencrypt/live:
total 4
drwx------ 3 root root  42 Apr 25 19:28 .
drwxr-xr-x 9 root root 190 May  1 17:14 ..
drwxr-xr-x 2 root root  93 Apr 25 19:28 onearth.studio
-rw-r--r-- 1 root root 740 Apr 25 19:28 README

/etc/letsencrypt/live/onearth.studio:
total 4
drwxr-xr-x 2 root root  93 Apr 25 19:28 .
drwx------ 3 root root  42 Apr 25 19:28 ..
lrwxrwxrwx 1 root root  38 Apr 25 19:28 cert.pem -> ../../archive/onearth.studio/cert1.p
em
lrwxrwxrwx 1 root root  39 Apr 25 19:28 chain.pem -> ../../archive/onearth.studio/chain1
.pem
lrwxrwxrwx 1 root root  43 Apr 25 19:28 fullchain.pem -> ../../archive/onearth.studio/fu
llchain1.pem
lrwxrwxrwx 1 root root  41 Apr 25 19:28 privkey.pem -> ../../archive/onearth.studio/priv
key1.pem
-rw-r--r-- 1 root root 692 Apr 25 19:28 README

/etc/letsencrypt/renewal:
total 4
drwxr-xr-x 2 root root  33 Apr 25 19:28 .
drwxr-xr-x 9 root root 190 May  1 17:14 ..
-rw-r--r-- 1 root root 529 Apr 25 19:28 onearth.studio.conf

/etc/letsencrypt/renewal-hooks:
total 0
drwxr-xr-x 5 root root  43 Apr 25 17:26 .
drwxr-xr-x 9 root root 190 May  1 17:14 ..
drwxr-xr-x 2 root root   6 Apr 25 17:26 deploy
drwxr-xr-x 2 root root   6 Apr 25 17:26 post
drwxr-xr-x 2 root root   6 Apr 25 17:26 pre


/etc/letsencrypt/renewal-hooks/deploy:
total 0
drwxr-xr-x 2 root root  6 Apr 25 17:26 .
drwxr-xr-x 5 root root 43 Apr 25 17:26 ..

/etc/letsencrypt/renewal-hooks/post:
total 0
drwxr-xr-x 2 root root  6 Apr 25 17:26 .
drwxr-xr-x 5 root root 43 Apr 25 17:26 ..

/etc/letsencrypt/renewal-hooks/pre:
total 0
drwxr-xr-x 2 root root  6 Apr 25 17:26 .
drwxr-xr-x 5 root root 43 Apr 25 17:26 ..

Thx!
Jan

griffin · May 1, 2021, 6:00pm

Please edit your post to include the backticks as I mentioned. Without them, many things get lost or distorted.

The letsencrypt folders are looking good though.

OnEarth · May 1, 2021, 6:06pm

Griffin, I did swapped hosting providers (from OVH to AWS) before using Let's Encrypt

griffin · May 1, 2021, 6:06pm

Ah... so the Amazon certificates were not related to a hosting swap then.

Rip · May 1, 2021, 6:11pm

ServerRoot + DocumentRoot = /etc/httpd/var/www/html

Getting closer?

OnEarth · May 1, 2021, 6:17pm

I have mistaken dots (...) and backticks (```)!

OnEarth · May 1, 2021, 6:19pm

Rip, do I just need to remove ServerAlias *?

Rip · May 1, 2021, 6:23pm

I'm thinking your Main Document root should look like:

Main DocumentRoot: "/var/www/html"

AND I'd make sure the

ServerName: onearth.studio
ServerAlias:  www.onearth.studio

are explicitly declared in the httpd.conf file (or vhost.conf if you are using one) and remove the current wildcard alias.

OnEarth · May 1, 2021, 6:30pm

I think you are right! I'll change httpd-le-ssl.conf and let you know

Rip · May 1, 2021, 6:46pm

Please update the httpd.conf to match httpd-le-ssl.conf
Both files should declare the same paths and ServerName / ServerAlias info.

OnEarth · May 1, 2021, 6:48pm

Thx Rip, I will do so

Rip · May 1, 2021, 6:59pm

Quoting @rg305...

ONE STEP AT A TIME

Also there are multiple statements declaring port 443 and cert files:
I recommend commenting out the following references:

*:443 is a NameVirtualHost default server ip-17.-..-..-..3.us-east-2.compute.internal (/etc/httpd/conf.d/ssl.conf)
port 443 namevhost ip-17.-..-..-..3.us-east-2.compute.internal (/etc/httpd/conf.d/ssl.conf

They are just getting in the way here... And leave:

port 443 namevhost ip-17.-..-..-..3.us-east-2.compute.internal (/etc/httpd/conf/httpd-le-ssl.conf

In Place.
Then restart your server

apache2ctl -k graceful

OnEarth · May 1, 2021, 7:33pm

Rip, I have changed httpd.conf and httpd-le-ssl.conf then restarted apache I got the following

Job for httpd.service failed because the control process exited with error code. See "sy
stemctl status httpd.service" and "journalctl -xe" for details.

The *:443 have to do with AWS Load Balancers I think, I am not sure what I need to do

Rip · May 1, 2021, 7:36pm

Ok so lets take a look at the output from:

systemctl status httpd.service

AND

journalctl -xe

This will tell us what the complaint is about. And we'll get to the bottom of it
Please post the complete output here and don't forget the backtics!