Let's Encrypt + Rancher Failed Renewal


#1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: wellness.blenderbottle.com
I ran this command: in rancher v1.6.14, we have the service running ‘janeczku/rancher-letsencrypt:v0.5.0’

It produced this output: “Error saving certificate ‘rancher-certs’: Failed to read certificate expiry date: Pem decode did not yield a valid block. Is the certificate in the right format?”

My web server is (include version): rancher

The operating system my web server runs on is (include version):

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don’t know): no, the service keeps restarting

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): Rancher v1.6.14


#2

Hi,

This seems to be a rancher level error…

Please try to open a support ticket at rancher…

(P.S. I have never heard rancher before… So I might be wrong…)

Thank you


#3

@stevenzhu here is some more insight into Let’sEncrypt with Rancher:

The main issue that we are running into is that the renewal for our certs that was set to automatically run is not completing. We are now getting rate-limited(which is not supposed to happen for renewals, per https://letsencrypt.org/docs/rate-limits/ ) which is aggravating the initial renewal issue. We have a dozen or so domains on this renewal process that is completely broken at this point.

In short: it could be partially Rancher which is to blame, but I feel like another equal part of the issues belong to Let’sEncrypt.

Respectfully Yours,

Grant


#4

Hi @gpeterson,

It’s very hard for us to try to diagnose this based on the information that you’ve given, since the only specific error message that you posted is a Rancher one that appears to have to do with how Rancher represents data on disk. Do you think you could provide some more detailed log messages?


#5

Hey @schoen,

So right now we are trying to get certs for multiple domains, I just listed one earlier, but just so you know, that is why the logs may reference a different domain. Here are some:

level=info msg="[INFO][wellness.blenderbottle.com] acme: Could not find solver for: tls-alpn-01" level=info msg="[INFO][wellness.blenderbottle.com] acme: Trying to solve HTTP-01" level=fatal msg="Failed to renew certificate: acme: Error 403 - urn:acme:error:unauthorized - Invalid response from http://box-artwork-generator.corp.blenderbottle.com/.well-known/acme-challenge/H9SsgXUw2tEhdSKEz3e-QcHiwRd9LtkLjtyL0G3csC0 [52.55.165.252]: 503 11/8/2018 12:45:21 PMError Detail: 11/8/2018 12:45:21 PM Validation for box-artwork-generator.corp.blenderbottle.com:80 11/8/2018 12:45:21 PM Resolved to: 11/8/2018 12:45:21 PM 52.55.165.252 11/8/2018 12:45:21 PM 54.144.223.199 11/8/2018 12:45:21 PM Used: 52.55.165.252 11/8/2018 12:45:21 PM 11/8/2018 12:45:21 PM"

Does that help?


#6

Hi,

We could help you with this issue!

The reason this domain fails when obtaining the certificate is because your domain (server ) returns 503 when checking the challenge files.

Could you please try to resolve this issue?

Thank you


#7

Hi @gpeterson

your website is curious. Checked via https://check-your-website.server-daten.de/?q=wellness.blenderbottle.com

Your http redirects to https, your https works. But Letsencrypt wants a file in the subdirectory /.well-known/acme-challenge. After 3 seconds -> a Service Unavailable error.

Your http is a nginx, your https has a header X-Powered-By: Express. But the curious problem is the 503 status in this subdirectory.

Perhaps you should check this nginx - instance.


#8

@JuergenAuer The funny thing is, the website(which is meant to work only on our internal network) is now working just fine. The rest of the sites that we have set to work with Let’sEncrypt on Rancher now work, kind of. They are all using the wellness.blenderbottle.com certificate now, instead of their own respective certs. We’re now looking into additional Rancher configuration, and trying to see why we keep getting stonewalled[re:rate limited] on renewal/registration of our domains.

I will try and post logs and extra information as I can… Rancher, unfortunately, has very poor log retention ;(