Certbot running on EC2 instance failing on some domains

No, but that alone is not what is causing the problem. I was just hoping to isolate where these faulty redirects were coming from.

Do you have an .htaccess file on your staging server anywhere? Because the Apache config you showed does not explain the behavior we see.

Your VirtualHost for port 80 (HTTP) lists all of your domain names except eastern. Which is not best practice but should still work as a default VirtualHost. But, just adding eastern to the ServerAlias list won't help because the HTTP redirects for the other domains are faulty in the same way as for eastern.

Example HTTP requests. I use staging-simpsons here but all your other domains fail in the same way (even eastern). Your production domain names fail identically.

# As expected, http->https
curl -I http://staging.simpsons-place.co.uk
HTTP/1.1 302 Found
Server: Apache/2.4.57 (Ubuntu)
Location: https://staging.simpsons-place.co.uk/

# As expected, http-https preserving the original URI (Test404)
curl -I http://staging.simpsons-place.co.uk/Test404
HTTP/1.1 302 Found
Server: Apache/2.4.57 (Ubuntu)
Location: https://staging.simpsons-place.co.uk/Test404

# Faulty.  Should not redirect to https (per your redirects below)
# And, used index.php instead of the original URI (.well-known...) 
curl -I http://staging.simpsons-place.co.uk/.well-known/acme-challenge/HTTP404
HTTP/1.1 302 Found
Server: Apache/2.4.57 (Ubuntu)
Location: https://staging.simpsons-place.co.uk/index.php

Your redirects from the VirtualHost
Should not redirect .well-known and should preserve URI

  RewriteEngine On
RewriteCond %{HTTPS} !=on
RewriteCond %{REQUEST_URI} !^/\.well-known
RewriteRule ^/(.*) https://%{SERVER_NAME}/$1 [R,L]
3 Likes

If you can't find the reason for the faulty redirects (previous post) we can modify your eastern domain to use the --standalone option.

That is not best since it requires Apache to be out of service while the certs are renewed.

Note we also need to fix the cert which contains maynard-meadows as that is no longer in service (or at least no longer has DNS entries). But, best to wait to resolve this until we finalize the method for authentication (standalone, apache, or webroot)

3 Likes

Hi Mike,

Appreciate your patience on this - I just want to spend some time on this so I can come back to you with a clearer picture.

The DNS for each subdomain points to staging server hosted on AWS.

Inside the AWS instance - var/vhosts are the folders for each site (eg staging.burlington-place.co.uk) :

drwxr-xr-x  4 root     root   4096 Apr 20  2021 staging.alexandra-mansions.co.uk
drwxr-xr-x  4 root     root   4096 Aug  4 10:20 staging.burlington-place.co.uk
drwxrwxr-x  4 www-data ubuntu 4096 Jul 17 07:50 staging.eastern-escape.co.uk
drwxr-xr-x  4 root     root   4096 Jun  3  2021 staging.maynard-meadows.co.uk
drwxr-xr-x  4 root     root   4096 Jun 29 10:08 staging.narrows-place.co.uk
drwxr-xr-x  4 root     root   4096 Aug  4 10:48 staging.simpsons-place.co.uk
drwxr-xr-x  4 root     root   4096 Nov 24  2021 staging.thompson-staithe.co.uk
drwxr-xr-x  4 root     root   4096 Mar 21  2023 staging.tollesbury-house.co.uk

Inside those folders is a 'current' flder which is symlinked to a release:

ubuntu@marketing-staging:/var/vhosts/staging.burlington-place.co.uk$ ls -al
total 20
drwxr-xr-x  4 root root 4096 Aug  4 10:20 .
drwxr-xr-x 10 root root 4096 May 25  2023 ..
lrwxrwxrwx  1 root root   66 Aug  4 10:20 current -> /var/vhosts/staging.burlington-place.co.uk/releases/20230804101903
drwxr-xr-x  6 root root 4096 Aug  4 10:20 releases
drwxr-xr-x  3 root root 4096 Dec 22  2020 shared

Then in the public folder of the release is a .htaccess file

<IfModule mod_rewrite.c>
    <IfModule mod_negotiation.c>
        Options -MultiViews -Indexes
    </IfModule>

    RewriteEngine On

    # Handle Authorization Header
    RewriteCond %{HTTP:Authorization} .
    RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]

    # Redirect Trailing Slashes If Not A Folder...
    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteCond %{REQUEST_URI} (.+)/$
    RewriteRule ^ %1 [L,R=301]

    # Send Requests To Front Controller...
    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteRule ^ index.php [L]
</IfModule>

It seems to me to be very over complicated, given the nature of these sites, but that's outside the scope of this I think.

Again, I appreciate all the help and support here, I'm learning a lot.

Chris.

1 Like

I'm just pointing this out because it stands out (to me):
image

3 Likes

Oof. I do not like .htaccess

But, I think if you modify the last rewrite section to look like below we should be good. Let me know when you've made the change so I can evaluate.

We are adding the line about checking for HTTPS being on so that HTTP requests are not affected by this group. We want all HTTP to go to HTTPS except for the acme challenge. This is done in your VirtualHost so this change below should allow that to work as intended.

    # Send Requests To Front Controller...
    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteCond %{HTTPS} =on
    RewriteRule ^ index.php [L]
3 Likes

Hi Mike,

Thanks for to time and responses on this - sorry I haven't got back to you sooner - but I've had my focus shifted on other projects and have now had time to circle back around to this.

I have tried making the adjustment on www.eastern-escape.co.uk (npot the staging the live version) as this is main one the management was fixed. I changed the .htaccess file in /public to match what you have suggested but I am still getting a failure in the renew:

ubuntu@marketing-vpc:/$ sudo certbot renew
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/burlington-place.co.uk.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not yet due for renewal

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/eastern-escape.co.uk.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for eastern-escape.co.uk
http-01 challenge for www.eastern-escape.co.uk
Waiting for verification...
Challenge failed for domain eastern-escape.co.uk
Challenge failed for domain www.eastern-escape.co.uk
http-01 challenge for eastern-escape.co.uk
http-01 challenge for www.eastern-escape.co.uk
Cleaning up challenges
Attempting to renew cert (eastern-escape.co.uk) from /etc/letsencrypt/renewal/eastern-escape.co.uk.conf produced an unexpected error: Some challenges have failed.. Skipping.
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/eastern-escape.co.uk/fullchain.pem (failure)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

The following certs are not due for renewal yet:
  /etc/letsencrypt/live/burlington-place.co.uk/fullchain.pem expires on 2024-02-12 (skipped)
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/eastern-escape.co.uk/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: eastern-escape.co.uk
   Type:   unauthorized
   Detail: 35.176.14.31: Invalid response from
   https://www.eastern-escape.co.uk/index.php: "<!DOCTYPE html>\n<html
   lang=\"en\" style=\"scroll-behavior: smooth;\">\n    <head>\n
   <meta charset=\"utf-8\">\n        <meta name=\""

   Domain: www.eastern-escape.co.uk
   Type:   unauthorized
   Detail: 35.176.14.31: Invalid response from
   https://www.eastern-escape.co.uk/index.php: "<!DOCTYPE html>\n<html
   lang=\"en\" style=\"scroll-behavior: smooth;\">\n    <head>\n
   <meta charset=\"utf-8\">\n        <meta name=\""

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

Hmmm. Apache is still redirecting the HTTP Challenge to your index.php. This will always fail. I am beginning to understand why someone stopped your Apache to do a standalone method now :slight_smile:

So, let's make sure burlington place cert profile still works and then we'll set eastern to be like that. Please show result of below command

sudo certbot renew --dry-run --cert-name burlington-place.co.uk

And, show the contents of this file (regardless of result of above)

/etc/letsencrypt/renewal/burlington-place.co.uk.conf
3 Likes

Maybe using --webroot would help???
OR
Maybe using a dedicated alias path for the /.well-known/acme-challenge/ requests???

2 Likes

Something is grabbing and redirecting the HTTP Challenges. We can't find it.

Their main cert profile has several domains and uses standalone (and has been). This one that fails is trying to use Apache but at this point I give up trying to find the faulty redirect.

Switching the one cert to use the same method as their main one seems the quickest approach. They can always work on finding that faulty redirect later and then switching their production and staging domains to use that.

3 Likes

.htaccess ???

2 Likes

You can review the thread - we tried that :slight_smile:

3 Likes

Hi Mike,

Here's the output of the dry run:

$ sudo certbot renew --dry-run --cert-name burlington-place.co.uk
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/burlington-place.co.uk.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator standalone, Installer None
Running pre-hook command: service apache2 stop
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for alexandra-mansions.co.uk
http-01 challenge for burlington-place.co.uk
http-01 challenge for narrows-place.co.uk
http-01 challenge for simpsons-place.co.uk
http-01 challenge for thompson-staithe.co.uk
http-01 challenge for tollesbury-house.co.uk
http-01 challenge for www.alexandra-mansions.co.uk
http-01 challenge for www.burlington-place.co.uk
http-01 challenge for www.narrows-place.co.uk
http-01 challenge for www.simpsons-place.co.uk
http-01 challenge for www.thompson-staithe.co.uk
http-01 challenge for www.tollesbury-house.co.uk
Waiting for verification...
Cleaning up challenges

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/burlington-place.co.uk/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

Congratulations, all renewals succeeded. The following certs have been renewed:
  /etc/letsencrypt/live/burlington-place.co.uk/fullchain.pem (success)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Running post-hook command: service apache2 start

and the content of the burlington-place.co.uk.conf file (Trimmed the account ID, I don't know if that's needed to be shown or safe to do so, please correct me if I'm wrong):

# renew_before_expiry = 30 days
version = 0.40.0
archive_dir = /etc/letsencrypt/archive/burlington-place.co.uk
cert = /etc/letsencrypt/live/burlington-place.co.uk/cert.pem
privkey = /etc/letsencrypt/live/burlington-place.co.uk/privkey.pem
chain = /etc/letsencrypt/live/burlington-place.co.uk/chain.pem
fullchain = /etc/letsencrypt/live/burlington-place.co.uk/fullchain.pem

# Options used in the renewal process
[renewalparams]
account = -- Snipped --
authenticator = standalone
server = https://acme-v02.api.letsencrypt.org/directory
pre_hook = service apache2 stop
post_hook = service apache2 start

Interestingly, the .conf file for eastern-escape.co.uk is as follows:

# renew_before_expiry = 30 days
version = 0.27.0
archive_dir = /etc/letsencrypt/archive/eastern-escape.co.uk
cert = /etc/letsencrypt/live/eastern-escape.co.uk/cert.pem
privkey = /etc/letsencrypt/live/eastern-escape.co.uk/privkey.pem
chain = /etc/letsencrypt/live/eastern-escape.co.uk/chain.pem
fullchain = /etc/letsencrypt/live/eastern-escape.co.uk/fullchain.pem

# Options used in the renewal process
[renewalparams]
account = aa8789cc180c09d6a1d09b0b4d1e4124
authenticator = webroot
server = https://acme-v02.api.letsencrypt.org/directory
[[webroot_map]]
eastern-escape.co.uk = /var/vhosts/maynard-meadows.co.uk/current/public
www.eastern-escape.co.uk = /var/vhosts/maynard-meadows.co.uk/current/public

It turns out Maynards is a dead project and the site was taken down a while back

So as a test I changed the webroot map:

[[webroot_map]]
eastern-escape.co.uk = /var/vhosts/burlington-place.co.uk/current/public
www.eastern-escape.co.uk = /var/vhosts/burlington-place.co.uk/current/public

and also tried changing it to eastern-escape.co.uk/current/public (as it's a laravel build)

I get the same response for both changes on the dry run:

** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/eastern-escape.co.uk/fullchain.pem (failure)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Running post-hook command: service apache2 start
1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: eastern-escape.co.uk
   Type:   connection
   Detail: 35.176.14.31: Fetching
   http://eastern-escape.co.uk/.well-known/acme-challenge/zEe4silauSd6deImDJVBzZ_rmkS14zq91XMwKd_GbFQ:
   Connection refused

   Domain: www.eastern-escape.co.uk
   Type:   connection
   Detail: 35.176.14.31: Fetching
   http://www.eastern-escape.co.uk/.well-known/acme-challenge/OWE0Kx57Am2PGh3f5qX0aC059GbHsQzTaJcCMbSD798:
   Connection refused

I feel like we are close here?

Thanks again for the help

1 Like

Is there a renewal config that doesn't start/stop apache and works?

1 Like

No. There is not.

Update: Well, now that I look back we did not try certonly --apache perhaps that would have overridden the faulty redirect we could not locate.

2 Likes

Okay good. But please do not make manual changes to the renewal conf files.

Let's try this

sudo certbot renew --dry-run --cert-name eastern-escape.co.uk --standalone --pre-hook "service apache2 stop" --post-hook "service apache2 start"

If that works run the same command but without the --dry-run

Let us know what happens

3 Likes

Looks like that did the trick.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/eastern-escape.co.uk.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator standalone, Installer None
Running pre-hook command: service apache2 stop
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for eastern-escape.co.uk
http-01 challenge for www.eastern-escape.co.uk
Waiting for verification...
Cleaning up challenges

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/eastern-escape.co.uk/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Congratulations, all renewals succeeded. The following certs have been renewed:
  /etc/letsencrypt/live/eastern-escape.co.uk/fullchain.pem (success)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Running post-hook command: service apache2 start

Thanks Mike.

From what I've picked up over this thread --standalone is not the most desirable way of doing this?

Any advice going forward?

2 Likes

It is not because it requires the Apache server to be stopped. This won't happen often though only when a cert is actually due for renewal so about every 60 days. You have two certs so 2 outages per 60 days. The outage should be brief and you could set your cron job to run off-hours.

I tried to get your --webroot method working which allows apache to stay running. But, something kept redirecting the HTTP Challenge to your index.php. We looked and looked but could not find where this was happening.

It's possible using the --apache plugin along with certonly would work and perhaps we should have tried that. It also uses HTTP Challenge but in a different way so may have overridden the faulty redirect

So, a better solution is to correct the faulty redirect and switch to --webroot or try certonly --apache. And, then apply that to both your certs so all your domains behave the same.

But, you have been living with --standalone for a long time so only you can weight the benefit of no Apache downtime versus the time spent switching to different method.

2 Likes

@MikeMcQ , thanks for the advice, and for going through everything so comprehensively.

I like I said, I have inherited these projects from previous developers, and I've not seen a configuration like what I am seeing in the backend before.

The sites on this server have a short lifespan, so eventually they old configurations will cycle out and as they are replaced with newer projects things should return to 'normal'.

In an ideal world, this would be to fix the redirects and start as we mean to go on, but as they are not high traffic sites so I feel a small outage every 60 days will be fine for getting on with.

2 Likes