Amazon Lightsail virtual host renew fails

denstl · August 29, 2019, 7:26pm

Just tried to renew certificates for 2 domains hosted in a single Amazon Lightsail instance. My original domain is argfam.net. Have renewed this certificate numerous times with nos issues, including today. I also have a WordPress application called acsls.org set up as virtual host. No issue when I added the certificate on June 21, 2019. It is good until Sep 19 so need to get it renewed. When I ran the renew command only argfam.net renewed successfully.

My domain is: acsls.org AND argfam.net (argfam.net is my main, static site. acsls.org is a virtual WordPress site. argfam.net certificate renews fine. acsls.org certificate throws an error when trying to renew it.

I ran this command: sudo ./certbot-auto renew

It produced this output:

Upgrading certbot-auto 0.36.0 to 0.37.2…
Replacing certbot-auto…
Creating virtual environment…
Installing Python packages…
Installation succeeded.
/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/cryptography/hazmat/primitives/constant_t
ime.py:26: CryptographyDeprecationWarning: Support for your Python version is deprecated. The next version of cryptography will remove support. Please upgrade to a 2.7.x release that supports hmac.compare_digest as soon as possible.
utils.PersistentlyDeprecated2018,
Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/acsls.org.conf

Cert is due for renewal, auto-renewing…
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for acsls.org
http-01 challenge for www.acsls.org
Using the webroot path /opt/bitnami/apps/acsls/htdocs for all unmatched domains.
Waiting for verification…
Challenge failed for domain acsls.org
Challenge failed for domain www.acsls.org
http-01 challenge for acsls.org
http-01 challenge for www.acsls.org
Cleaning up challenges
Attempting to renew cert (acsls.org) from /etc/letsencrypt/renewal/acsls.org.conf produced an unexpected error: Some challenges have failed… Skipping.

Processing /etc/letsencrypt/renewal/argfam.net.conf

Cert is due for renewal, auto-renewing…
Plugins selected: Authenticator standalone, Installer None
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for argfam.net
Waiting for verification…
Cleaning up challenges

new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/argfam.net/fullchain.pem

The following certs could not be renewed:
/etc/letsencrypt/live/acsls.org/fullchain.pem (failure)

The following certs were successfully renewed:
/etc/letsencrypt/live/argfam.net/fullchain.pem (success)

The following certs could not be renewed:
/etc/letsencrypt/live/acsls.org/fullchain.pem (failure)

1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:

The following errors were reported by the server:

Domain: acsls.org
Type: connection
Detail: Fetching
http://acsls.org/.well-known/acme-challenge/hrUDayccxtrVhtW4AM-tDyx6LvwlEHBtynvw6qnH0ns:
Connection refused

Domain: www.acsls.org
Type: connection
Detail: Fetching
http://www.acsls.org/.well-known/acme-challenge/LmSAnOrHQcKpp0H_NKQBUZieZMmSoOavBXTisQdQ7uE:
Connection refused

To fix these errors, please make sure that your domain name was
entered correctly and the DNS A/AAAA record(s) for that domain
contain(s) the right IP address. Additionally, please check that
your computer has a publicly routable IP address and that no
firewalls are preventing the server from communicating with the
client. If you’re using the webroot plugin, you should also verify
that you are serving files from the webroot path you provided.

NOTE: the argfam.net certificate renewed with no issues. This was the first time I was renewing the acsls.org certificate as it is a new site. No issues when I originally added the certificate to acsls.org.

My web server is (include version): Apache/2.4.25 (Unix)

The operating system my web server runs on is (include version): Ubuntu 14.04.5 LTS"

My hosting provider, if applicable, is: Amazon Lightsail

I can login to a root shell on my machine (yes or no, or I don’t know): Yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): NO

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot): /usr/lib/python3/dist-packages/requests/init.py:80: RequestsDependencyWarning: urllib3 (1.25.3) o
r chardet (3.0.4) doesn’t match a supported version!
RequestsDependencyWarning)
certbot 0.28.0

JuergenAuer · August 29, 2019, 7:38pm

Hi @denstl

if you use webroot, then your current webserver is used.

But there

you see: Looks like your webserver doesn't answer. So it's impossible to check your domain. Your other domain uses --standalone, so it must stop the running webserver. May be the webserver is already stopped.

But checking your domain there is a second problem - https://check-your-website.server-daten.de/?q=acsls.org

A wrong redirect with a "missing slash":

Domainname	Http-Status	redirect	Sec.	G
• http://acsls.org/
52.15.61.238	302	https://acsls.org	0.250	A

• http://www.acsls.org/
52.15.61.238	302	https://acsls.org	0.250	E

• https://www.acsls.org/
52.15.61.238	301	https://acsls.org/	3.797	B

• https://acsls.org/
52.15.61.238	200		4.360	I

• https://acsls.org	200		4.004	I

• http://acsls.org/.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de
52.15.61.238	302	https://acsls.org.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de	0.246	A
Visible Content: Found The document has moved here .

• http://www.acsls.org/.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de
52.15.61.238	302	https://acsls.org.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de	0.250	E
Visible Content: Found The document has moved here .

• https://acsls.org.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de	-1		0.033	R
NameResolutionFailure - The remote name could not be resolved: 'acsls.org.well-known'
Visible Content:

Result: http + /.well-known/acme-challenge is redirected to the not existing domain acsls.org.well-known - that can't work.

So first step: Check your redirect rules and add the /, then recheck your domain to see if that works.

denstl · August 29, 2019, 10:54pm

I really am not following (sure it’s me just not being clear what this is doing).

I confirmed my A record points to the correct IP address.
When I tried to renew both certificates I did stop all services. Same way I’ve always done this in the past which worked fine for the argfam.net domain. I tried it again without doing that and got the same errors for acsls.org domain (the other it tells me there is no need to renew yet).
Looked at my redirects and don’t really see a missing slash.
I don’t get what all the “acme challenge” or the “well-known” messages are saying.

Why did this all work when I originally installed the certificate back in June? Can I just reinstall the certificate following those same steps again? Worked the first time I installed it. Just not this first time when I try to renew it.

Thanks for your help and patience (if these are ignorant questions).

schoen · August 29, 2019, 11:04pm

Certbot has several different methods for proving your control over a domain name. The --webroot method uses an existing web server to do it (and requires that the existing server be running), while the --standalone method creates a temporary web server to do it (and requires that any existing server not be running). It looks like you happened to choose one of these methods for one certificate and the other for the other certificate.

While both of these would work under the right conditions, it's more challenging to get both them to work permanently together, because they have conflicting assumptions about whether there is a running web server or not. Certbot isn't quite clever enough to say "wait, you have several certificates with different and potentially contradictory assumptions about the environment in which they will be renewed".

In any case, @JuergenAuer's test does show a likely very important problem: if a user (or the Let's Encrypt validation bot) goes to a URL like

http://acsls.org/.well-known/acme-challenge/something

which would happen during the validation process, your site sends a redirect to

https://acsls.org.well-known/acme-challenge/something

instead of the correct

https://acsls.org/.well-known/acme-challenge/something

This is the "missing slash" problem to which @JuergenAuer referred. It would be helpful if you could check over your redirects one more time to try to figure out why this might be happening, since it's a very common problem that often blocks successful certificate renewals. The problem is not at all specific to the Let's Encrypt validation stuff with /.well-known URLs; for example, going to http://acsis.org/anything sends a redirect to https://acsis.organything, rather than the intended https://acsis.org/anything.

JuergenAuer · August 30, 2019, 5:49am

Then the --webroot authentication can't work. And stopping all services produces a simple problem: Your website is offline. So --standalone is normally not an option if you have a running website.

Result: You should switch your authentication method of your argfam.net, so standalone isn't longer used.

Yes, you can. Use the --cert-name to overwrite your existing certificate.

See

https://certbot.eff.org/docs/using.html

PS

Your first redirect

http://acsls.org/ -> https://acsls.org

in your port 80 vHost is wrong, there is no ending slash. Share that configuration, there you must add a /

denstl · August 30, 2019, 2:51pm

Thank you both! I found a port 80 redirect that did not have the virtual host port 80 redirect, I changed “Redirect / https://acsls.org” to “Redirect / https://acsls.org/”. Ran the renew command and certificate was renewed successfully. I did have to restart Apache but now see the certificate is valid thru 11/28/2019.

Not needed, but here are the results of running the " ./certbot-auto renew" command:

Processing /etc/letsencrypt/renewal/acsls.org.conf

Cert is due for renewal, auto-renewing…
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for acsls.org
http-01 challenge for www.acsls.org
Using the webroot path /opt/bitnami/apps/acsls/htdocs for all unmatched domains.
Waiting for verification…
Cleaning up challenges

new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/acsls.org/fullchain.pem

THANK YOU AGAIN FOR THE QUICK AND POINT ON ASSISTANCE!

JuergenAuer · August 30, 2019, 3:57pm

Happy to read that you have found the “missing slash”

That’s a hidden problem of some configurations. Root redirects work, but redirects with subfolders don’t work.

And now you have a new certificate:

CN=acsls.org
	30.08.2019
	28.11.2019
expires in 90 days	acsls.org, www.acsls.org - 2 entries

But you have a small content error - Grade I, you should fix.

Created a screenshot so you can see the problem:

Your source code:

<img 
src="https://argfam.net/acsls/wp-content/uploads/2019/03/cropped-LS-group-1.jpg" 
width="2000" height="990" alt="Ascension Early Childhood Center" 
srcset="https://acsls.org/wp-content/uploads/2019/03/cropped-LS-group-1.jpg 2000w, https://acsls.org/wp-content/uploads/2019/03/cropped-LS-group-1-300x149.jpg 300w, https://acsls.org/wp-content/uploads/2019/03/cropped-LS-group-1-768x380.jpg 768w, https://acsls.org/wp-content/uploads/2019/03/cropped-LS-group-1-1024x507.jpg 1024w" 
sizes="100vw" 
pagespeed_url_hash="3734109468" onload="pagespeed.CriticalImages.checkImageForCriticality(this);"/>

The fallback src attribute -> the file is missing, http status 404 - Not Found.

That’s a difficult error. Because if your browser supports the srcset attribute, you don’t see a “missing image”. But if a browser doesn’t support that attribute, there is nothing.

Rechecked - yep: FireFox doesn’t show a problem. Same with Chrome. The console doesn’t have errors.

And

https://acsls.org/wp-content/uploads/2019/03/cropped-LS-group-1.jpg

is your main picture, so it’s bad if the image isn’t there.

denstl · August 30, 2019, 4:52pm

Thank you again. Must be something either WordPress or the Twenty Seventeen theme adds automatically. I’m new to WordPress so not sure which. Anyway, I uploaded that image again and re-attached to the header and appears to be there now. Looking at the source code I can see it does alter the name of the image I upload and attach (and adds all the images included in the srcset). It will be time to update images soon and I’ll keep an eye on that. Anyway, appreciate your close scrutiny.

system · September 29, 2019, 4:52pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.