How to renew my certificate on EC2 Amazon Linux 2 AMI (Apache server)?

OnEarth · July 31, 2021, 12:46pm

Hi everyone,

I received an email telling me that my certificate will expire. What shall I do to renew it when a certificate was already installed?

My domain is: https://onearth.studio/

I ran this command: apachectl -S

It produced this output:AH00526: Syntax error on line 30 of /etc/httpd/conf/httpd-le-ssl.conf: SSLCertificateFile: file '/etc/letsencrypt/live/onearth.studio/fullchain.pem' does not exist or is empty

My web server is (include version): Apache/2.4.48

The operating system my web server runs on is (include version): Linux 4.14.231-173.361.amzn2.x86_64

I can login to a root shell on my machine (yes or no, or I don't know): yes

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): /var not root-owned 48:48

Osiris · July 31, 2021, 2:08pm

Did you run apachectl -S as root? If not, please try sudo apachectl -S first to check if it's not a permission related issue.

OnEarth · July 31, 2021, 2:12pm

Hi Osiris,

Thx for your answer.

I ran this command

sudo apachectl -S

It produced this answer :

VirtualHost configuration:
*:80                   onearth.studio (/etc/httpd/conf/httpd.conf:64)
*:443                  is a NameVirtualHost
         default server ip-172-31-33-253.us-east-2.compute.internal (/etc/httpd/conf.d/ssl.conf:56)
         port 443 namevhost ip-172-31-33-253.us-east-2.compute.internal (/etc/httpd/conf.d/ssl.conf:56)
         port 443 namevhost onearth.studio (/etc/httpd/conf/httpd-le-ssl.conf:3)
                 alias www.onearth.studio
ServerRoot: "/etc/httpd"
Main DocumentRoot: "/etc/httpd/htdocs"
Main ErrorLog: "/etc/httpd/logs/error_log"
Mutex mpm-accept: using_defaults
Mutex cache-socache: using_defaults
Mutex authdigest-opaque: using_defaults
Mutex watchdog-callback: using_defaults
Mutex proxy-balancer-shm: using_defaults
Mutex rewrite-map: using_defaults
Mutex ssl-stapling-refresh: using_defaults
Mutex authdigest-client: using_defaults
Mutex lua-ivm-shm: using_defaults
Mutex ssl-stapling: using_defaults
Mutex proxy: using_defaults
Mutex authn-socache: using_defaults
Mutex ssl-cache: using_defaults
Mutex default: dir="/run/httpd/" mechanism=default 
PidFile: "/run/httpd/httpd.pid"
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
User: name="apache" id=48
Group: name="apache" id=48

Osiris · July 31, 2021, 2:28pm

Ok, so your Apache isn't the problem: you just didn't run it as root previously.

What issue do you have with renewing, if it wasn't the above?

OnEarth · July 31, 2021, 3:01pm

I do not know what to do to renew my certificate when I had one before, I do not want to make any mistake.

When I run this command:

sudo certbot --version

It produced this output:

/var not root-owned 48:48

I ran this command:

sudo certbot --redirect --uir

It produced this output:

/var not root-owned 48:48

Osiris · July 31, 2021, 3:41pm

The usual command to renew is sudo certbot renew, but:

This suggests along the way something terrible has happened on your Linux system: for some reason, the directory /var is owned by a non-root user, which is not standard and certbot seems to refuse to go further as long as /var isn't owned by root.

rg305 · July 31, 2021, 10:00pm

Please show this file:

rg305 · July 31, 2021, 10:17pm

Also: What happened to the certificate issued on April 8 (by Amazon)?
[it is still valid until next May]
See: crt.sh | 4345977464

orangepizza · August 1, 2021, 3:29am

IIRC those certificate are tied to AWS cloudfront and amazon won't give its private key to user.

OnEarth · August 1, 2021, 9:20am

Hi rg305,

This is file /etc/httpd/conf/httpd-le-ssl.conf

<IfModule mod_ssl.c>
#Listen 4242
<VirtualHost *:443>
  #  DocumentRoot: "var/www/html"
 #   ServerAlias *
#Redirect permanent / https://www.onearth.studio
DocumentRoot /var/www/html
ServerName onearth.studio
ServerAlias www.onearth.studio
RewriteEngine On
# Some rewrite rules in this file were disabled on your HTTPS site,
# because they have the potential to create redirection loops.

#     RewriteCond %{HTTP:X-Forwarded-Proto} =http
#     RewriteRule .* https://%{HTTP_Host}%{REQUEST_URI} [L,R=permanent]
<Directory "var/www/html">
Options Indexes FollowSymLinks
AllowOverride None
Require all granted
</Directory>

<Directory "var/www/stripebackend">
Options Indexes FollowSymLinks
AllowOverride None
Require all granted
</Directory>

SSLEngine on
Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateFile /etc/letsencrypt/live/onearth.studio/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/onearth.studio/privkey.pem
#    Header always set Content-Security-Policy upgrade-insecure-requests

SSLProxyEngine on
SSLProxyVerify none
SSLProxyCheckPeerCN off
SSLProxyCheckPeerName off
SSLProxyCheckPeerExpire off
ProxyRequests off
ProxyPreserveHost on
<Proxy https://3.134.106.244:4242>
Require all granted
</Proxy>
ProxyPass /create-checkout-session http://localhost:4242/create-checkout-session/
ProxyPassReverse /create-checkout-session https://localhost:4242/create-checkout-session/
<Location /create-checkout-session>
Require all granted
</Location>
</VirtualHost>
</IfModule>

rg305 · August 1, 2021, 6:29pm

I'm not sure what happened to /var that now blocks certbot from access.
Let's have a look at:
ls -l / | grep -E 'etc|var'
cat /etc/letsencrypt/renewal/*
find /etc/letsencrypt/ -name *.ini

OnEarth · August 1, 2021, 9:42pm

Hi @rg305,

Thx for being here! Very much appreciated!

I ran this command: ls -l / | grep -E 'etc|var'

It produced this output:

drwxr-xr-x  87 root   root   8192 Jul 30 12:02 etc
lrwxrwxrwx   1 root   root     18 Jun 17 21:54 snap -> var/lib/snapd/snap
drwxr-xr-x  21 apache apache  292 Apr 24 19:47 var

I ran this command: cat /etc/letsencrypt/renewal/*

It produced this output:

# renew_before_expiry = 30 days
version = 1.14.0
archive_dir = /etc/letsencrypt/archive/onearth.studio
cert = /etc/letsencrypt/live/onearth.studio/cert.pem
privkey = /etc/letsencrypt/live/onearth.studio/privkey.pem
chain = /etc/letsencrypt/live/onearth.studio/chain.pem
fullchain = /etc/letsencrypt/live/onearth.studio/fullchain.pem

# Options used in the renewal process
[renewalparams]
account = 66b7d472dc4d75d3b8ec7111b38fa804
authenticator = apache
installer = apache
server = https://acme-v02.api.letsencrypt.org/directory

It ran this command: `find /etc/letsencrypt/ -name *.ini

It produced this output:

find: ‘/etc/letsencrypt/accounts’: Permission denied
find: ‘/etc/letsencrypt/keys’: Permission denied
find: ‘/etc/letsencrypt/archive’: Permission denied
find: ‘/etc/letsencrypt/live’: Permission denied

Osiris · August 1, 2021, 9:45pm

I'm pretty sure the /var dir shouldn't be owned by apache.

rg305 · August 1, 2021, 9:46pm

Somehow Apache has taken control of the /var directory.
[unlike /etc which remains owned by root]

Try with sudo:
sudo find /etc/letsencrypt/ -name *.ini

OnEarth · August 1, 2021, 10:00pm

@Osiris, @rg305,

I ran this command: sudo find /etc/letsencrypt/ -name *.ini

It produced nothing

It tried sudo su and I ran this command: find /etc/letsencrypt/ -name *.ini

It produced nothing too

I ran this command next: ls -l / | grep -E 'etc|var'

It produced this output:

drwxr-xr-x  87 root   root   8192 Jul 30 12:02 etc
lrwxrwxrwx   1 root   root     18 Jun 17 21:54 snap -> var/lib/snapd/snap
drwxr-xr-x  21 apache apache  292 Apr 24 19:47 var

rg305 · August 1, 2021, 10:11pm

[note: @OnEarth, when address individuals on this forum don't forget to include the preceding "@"]

Not finding an .ini file is not a problem.
I merely wanted to see the contents (if one existed).

Somehow/somewhere in your journey you have passed the entire /var directory to the sole control/access of the apache user/group.
This is probably going to create issues far outside of certbot - but I shall not digress form the problem at hand.
If you want to use certbot without addressing that ownership issue, you can simply make a new unique dedicated directory (outside of /var) to handle the challenge requests. Then update the .well-known/acme-challenge path to use the newly created folder in your web configs.

OnEarth · August 1, 2021, 10:22pm

@rg305,

Thx for your explanations and valuable advice.

I do not know when the entire /var directory went under apache control/access.

Wouldn't it be better to give back root control/access to /var directory ?

rg305 · August 1, 2021, 10:32pm

I can't say with any certainty.
I don't know what (if anything) now relies on that level of access.
I can only say with certainty that it doesn't make any sense; no single program should own that entire path.

OnEarth · August 1, 2021, 10:39pm

@rg305,

Thx for you explanations, this might appended after I ran Certbot the first time (everything was working fine until now). I will give back root access to /var directory. Then what shall I do next (I do not want to make any mistake)?

rg305 · August 1, 2021, 11:31pm

Well, using --dry-run & -vv, I would continue to test certbot for errors.
[ I suspect that even /etc/letsencrypt/ might now also be owned by apache ]

In any case, I would continue fixing things until certbot passes the -dry-run tests.
[which might be simpler to do by changing the challenge path location - as explained previously]
[but who needs simple - we need certain!]

Unfortunately, I don't know what made the ownership change, so I can't be certain on how to undue it.