Let's Encrypt Renewal Failing on Shared Server

Help
1

My domain is: drupal9test.sinet.ca

I ran this command: N/A

It produced this output :N/A

My web server is (include version): Apache 2.4.54

The operating system my web server runs on is (include version): 5.4.0-163-generic #180-Ubuntu

My hosting provider, if applicable, is: pair.com

I can login to a root shell on my machine (yes or no, or I don't know): no

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): pair.com admin

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): N/A

I have a number of shared hosting servers. My service provider supports Let's Encrypt certificates through their admin platform. When requesting a certificate, I disable the default of adding the www. to the subdomain. Once the certificate has been issued and loaded, my service provider automatically re-directs http:// requests to https://. At this point, I enable Cloudflare proxy for my subdomain using the Full Strict option. Cloudflare also redirects http:// to https://. Everything works as expected

However, it appears that Let's Encrypt is unable to refresh the SSL certificate. About 23 days before it expires, I see daily attempts that originate from Cloudflare proxy servers:

162.158.245.78 - - [29/Nov/2023:18:07:32 -0700] "GET /.well-known/acme-challenge HTTP/1.1" 301 627 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
162.158.245.78 - - [29/Nov/2023:18:07:32 -0700] "GET /.well-known/acme-challenge/ HTTP/1.1" 200 5642 "http://drupal9test.sinet.ca/.well-known/acme-challenge/ "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

If I disabled Cloudflare proxy, the certificate renewal completes successfully. The log shows the same commands, expect I see three pairs of requests (301 and 200) from three IPs, one registered to Internet Security Research Group, the other two to Amazon.

I asked pair.com and was told that:

Cloudflare will not validate when a domain name doesn't use the pair.com IP
address. Our implementation of Cloudflare is built in such a way that it
only uses http/https validation. When the public-facing IP address is
Cloudflare Let's Encrypt will not renew nor successfully install as a new
cert.

Is this a limitation of pair.com's implementation or a feature of Let's Encrypt? I had been successfully renewing Let's Encrypt certificates on an Amazon EC2 instance behind Cloudflare as recently as March 2022 https://forum.virtualmin.com/t/lets-encrypt-renewal-failing-during-secondary-validation-invalid-response-aws-behind-cloudflare/114247. At the time, it seemed to be an issue with Cloudflare switching their proxy servers to IPV6 while my Amazon server was still on IPV4. The recommendation was to use certbot but I could not find a version that supported my rather old Linux version.

2

Hello @nh905, welcome to the Let's Encrypt community. :slightly_smiling_face:

Using the online tool Let's Debug yields these results https://letsdebug.net/drupal9test.sinet.ca/1719501

image
image1183×551 16.4 KB

CloudflareCDN
WARNING
The domain drupal9test.sinet.ca is being served through Cloudflare CDN. Any Let's Encrypt certificate installed on the origin server will only encrypt traffic between the server and Cloudflare. It is strongly recommended that the SSL option 'Full SSL (strict)' be enabled.
https://support.cloudflare.com/hc/en-us/articles/200170416-What-do-the-SSL-options-mean-

Please see https://support.cloudflare.com/hc/en-us/articles/200170416-What-do-the-SSL-options-mean-

1 Like
3

I also see this certificate crt.sh | 11293965725 was issued Dec 2 17:15:03 2023 GMT.

1 Like
4

That's probably issued by cloudflare?

@nh905 the issue probably is cloudflare redirecting http to https instead of letting your server do that. You should tell cloudflare not to do that (or, if you feel adventurous, tell certbot to run http-01 on port 443).

2 Likes
5

Welcome to the Let's Encrypt Community, Norbert!

Here's the easiest solution: :slightly_smiling_face:

4 Likes
6

@griffin Unfortunately, it appears these are private certificates. According to the documentation "Site visitors may see untrusted certificate errors if you pause or disable Cloudflare on subdomains that use Origin CA certificates." Installing a Let's Encrypt certificate on my shared server allows me to enable Cloudflare in Full Strict mode but also protects my site if I need to disable Cloudflare proxy.

7

@9peppe I can try disabling Cloudflare's redirect the next time the Let's Encrypt certificate is up for renewal. However, I also have my shared hosting provider redirect http to https. I think my hosting provider's certbot implementation handles https calls based on what I see in the logs: initially a 301 redirect followed by a 200 OK.

8

Yes, that is a drawback of the Origin CA Cert.

Is there a need to disable the proxy for your "normal" domains? Could that need be satisfied by setting up a separate domain for direct access to your origin. Say, myadmin.drupal9test.sinet.ca

Getting an LE cert for a non-proxied domain is usually pretty easy.

3 Likes
9

@Bruce5051 I am currently running Cloudflare in Full Strict. Cloudflare's proxy servers are communicating with my shared server over https using the shared server's Let's Encrypt certificate. I can disable Cloudflare proxy and still have my server protected by the Let's Encrypt certificate.

I have not seen significant complaints in the Cloudflare forums about Let's Encrypt certificates refusing to renew, outside of specific issues that I do not believe are related to my problem. I did find a Cloudflare tech note that specifically recommended using Let's Encrypt http-01 verification for servers behind Cloudflare.

What may be different in my case is that my shared hosting provider is not replacing the source address (Cloudflare proxy) with the original source IP in the X-Forwarded-Host header - the server log entries all show Cloudflare addresses. On my Amazon EC2 server, I run an Apache module that replaces the Cloudflare proxy address with the true source IP. I do not know enough about how the Let's Encrypt logic works (or should work) to assess whether using the X-Forwarded-Host address could help. Since my hosting provider claims I must turn off Cloudflare proxy, I do not have much leverage unless I can point out that properly implemented Let's Encrypt can survive communicating through a Cloudflare proxy, or for that matter, any other CDN service.

10

@MikeMcQ Cloudflare does not support three levels of subdomain - I cannot set up Cloudflare proxy for myadmin.drupal9test.sinet.ca. I am using Cloudflare as my DNS provider but also use its proxy support for cache, spam/bot blocking, and WAF rules.

It is not a big issue disabling Cloudflare proxy once every two months for my shared hosting subdomains. Unfortunately, if I forget, it means my servers go offline because Cloudflare Full Strict mode requires that my server have a valid SSL certificate.

11

Then you should be able to use an ACME client that can integrate with CF DNS and allow you to obtain certificates via DNS authentication.

3 Likes
12

@rg305 Unfortunately, I have no control over what my hosting provider has implemented or what they have done. They only support http-01. I might be able to convince them to have a look at their code if I can show another shared hosting provider that has implemented Let's Encrypt certificates and the certificates successfully renew for subdomains where Cloudflare proxy is enabled. I see posts about cPanel support but none specifically mention Cloudflare.

1 Like
13

These Cloudflare settings may interest you.

3 Likes
14

@linkp I had set up a Cloudflare rule bypassing http redirect when I first implemented Let's Encrypt on my VPS, but found that the rule did not seem to be required once a Let's Encrypt certificate had been installed. The Let's Encrypt certificate was last renewed on December 1, so it will start renewing on February 6th. I should be able to set up acme.sh on my shared server, manually create a Let's Encrypt certificate, and then manually renew the certificate to determine exactly what conditions cause the renewal to fail. If I cannot get the renewal to fail, it may help me convince my shared hosting provider to take a look at their code.

2 Likes
15

I created a new subdomain on my shared server and followed the procedure to obtain a Let's Encrypt certificate. I then installed the latest acme.sh from git. To simulate what I think the shared server code does, I did not activate Cloudflare proxy. I successfully used acme.sh to create a new certificate using the webroot method. When I checked the logs the next day, I could see the shared server interacting with the Let's Encrypt servers to secure my subdomain and also my subsequent request for a certificate.

I have been able to renew my certificate using acme.sh, but the verification process is skipped with "letsencrypt.sinet.ca is already verified, skip http-01". I found a local .conf file that had an expiry date - resetting the date to yesterday did not help. Running debug suggests that acme.sh is fetching the validated date from the Let's Encrypt servers and skipping the acme-challenge check.

I am going to see if I can decipher acme.sh to verify, but the simplest approach may be to delete the certificate and create a new one with Cloudflare proxy enabled. That does not prove a renewal will work - for that I might have to wait a few months.

16

No, what is probably happening is that Let's Encrypt caches successful validations per domain and account for 30 days (see here).

You could use a fresh account each time to avoid the cache. Certbot has a --dry-run feature to invalidate the cache and also use the Staging system. Sadly, acme.sh has no such feature (that I know of).

2 Likes
17

I will try removing the existing certificate and fetching a new one with a short "--valid-to +".

18
4 Likes
19

No joy - Let's Encrypt only issues 90-day certificates. I set up another subdomain, had my shared hosting provider set up a Let's Encrypt certificate, enabled Cloudflare proxy for the subdomain, and tried to setup a standalone Let's Encrypt certificate using acme.sh. The critical domain validation step failed with an error 403, which is the same error I was seeing on my AWS EC2 server last year. At the time, I suspected that the 403 was returned by Cloudflare, not my server.

To make a long story short, this issue has nothing to do with Cloudflare redirects but was self-inflicted. My AWS servers see a lot of spamming/hacking attempts and high speed site scrapers which I block using Cloudflare IP and class C rules. Starting late on 2021, I also set up rules blocking access from ASNs that seem to harbor a lot of miscreants. In September 2022, AS16509 (amazon.com) got added. Some of Let's Encrypt servers are hosted on amazon.com - with Cloudflare proxy enabled, that meant Cloudflare blocked the final fetches of acme-challenge by Let's Encrypt . I was able to confirm this through the Cloudflare Security/Events log. I removed the AS16509 block and was able to create a local certificate using acme.sh.

Obvious in hindsight, but hindsight is always 20/20. What confused me is that I was seeing Let's Encrypt GET traffic in the logs of failed attempts, just not all of them. In addition, my shared hosting does not show the origin source of requests (all traffic appears to originate from Cloudflare proxy servers).

Thanks to everyone who contributed to the discussion - it encouraged me to investigate the root cause of the problem.

1 Like
20

I don't know if you will be able to implement this on shared hosting, but if you can, it will help.

4 Likes