Certbot Version & Invalid CA

Help
1

Greetings,

Our domain is:

This issue is time sensitive - any help is much appreciated.

We use 90-day wildcard TLS certs. We automate the renewal process with Certbot and a fairly typical setup. We've had no issues for years. In today's renewal, the script that calls the certbot client ran successfully, using DNS challenges as always.

The resulting cert, once deployed, shows:

Certificate issuer: R3

Certificate chain
DST Root CA X3
R3
*.bitbrew.com

This cert (above) is unrecognized by our RabbitMQ cluster. We had to revert back to the old cert, which expires tomorrow (Feb 18, 2021).

Previously, certs, once deployed, showed:

Certificate issuer: Let's Encrypt Authority X3

Certificate chain
DST Root CA X3
Let's Encrypt Authority X3
*.bitbrew.com

The script that calls certbot runs on macOS and was installed through homebrew. I believe that after upgrading to Big Sur, the certbot version was auto-upgraded by brew. The version is now:

certbot: stable 1.11.0

Looks like the previous version of certbot (last used to renew certs back on Nov 20, 2020) was 0.33.1.

I don't doubt that the new certs from today are valid and use the correct CA and intermediate, but we have no quick fix for RabbitMQ, and need to renew our certs.

Is it possible to install an older version of certbot on macOS?

If yes, will that version generate the cert as it used to be so we can renew and buy some time to investigate RabbitMQ?

Please note that the website cert is managed separately - the issue here concerns our RabbitMQ cluster specifically.

Thanks in advance,

Ben

1 Like
2

Can you explain exactly what you mean by "unrecognized by our RabbitMQ cluster"? What certificates are in its trust store?

My crystal ball says that you have it configured to trust only "Let's Encrypt Authority X3" which is no longer in use, and you should be instead configuring it to trust at least "DST Root CA X3" and "ISRG Root X1", and having whatever connects to it be sure to send its intermediate along with the certificate being used.

If that's the case, it's got nothing to do with the version of certbot, but just that intermediates that Let's Encrypt uses can change at any time and you need to ensure that your systems are configured accordingly to only trust the roots (which change much less often) and handle whatever intermediate is being used.

1 Like
3

Many thanks @petercooperjr - your crystal ball may be quite right. We are investigating now (an older RMQ config it seems). Will follow-up to confirm once we know more. Thanks again.

1 Like
4

@petercooperjr - we're checking the RMQ logs, which have:

TLS server: In state certify received CLIENT ALERT: Fatal - Certificate Unknown

The RMQ config itself does not seem to be hard-coding anything (still investigating though).

Thanks,
Ben

1 Like
5

Check you're using the fullchain pem file and that you're not providing it an intermediate using some other means (some people concatenate using their own scripts etc and end up using the old intermediate) and you're not just giving it your leaf/end-entity cert (which doesn't have the intermediate that signed it).

1 Like
6

@webprofusion - thanks! Yes, we are indeed concatenating the cert and private key and creating a combined.pem (in the script) for use by RabbitMQ, as it mounts the deployed cert as a Kubernetes volume.

1 Like
7

Can you make sure that the combined file has both the end-entity certificate and the intermediate certificate? For example, in Certbot these should be provided together as fullchain.pem.

2 Likes
8

Thanks to you both - checking this now in the script - struggling a bit as I did not write it ...

1 Like
9

Removed old script details as irrelevant.

1 Like
10

Removed old script details as irrelevant.

1 Like
11

Look out also for future use of multiple intermediates, you want to use the fullchain provided by certbot <intermediate 1> => <intermediate 2> => <intermediate n> => <your cert>

If you're not sure what's in the file, open it in a text editor. A fullchain has multiple --certificate-- entries and if the file only has one entry then it's your end-entity cert (i.e. the one for your hostname/domain).

So if rabbitmq only accepts an entry for a key file and a cert file, make sure your cert file has all the component intermediate certs and your end-entity cert. They should appear in the file in order of: end-entity-cert, intermediate that signed it, intermediate that signed that (if any), root cert. How to Create a .pem File for SSL Certificate Installations

You can also use online pem decoders to see what is end each block (or use openssl): Certificate Decoder - Decode certificates to view their contents

1 Like
12

Thanks again @webprofusion - very helpful ... time for some testing.

Thanks as well @schoen - much appreciated!

@schoen - I see the fullchain.pem and as you said, it does indeed have both (end-entity and intermediate) - is this the only file that has the intermediate?

2 Likes
13

No, the intermediate is also found in chain.pem (where fullchain.pem is just the combination of cert.pem and chain.pem).

1 Like
14

Thanks again @schoen

1 Like
15

I've updated the script to include the intermediate cert as part of the combined.pem (that is being used by RabbitMQ). The combined.pem now has:

cert.pem
chain.pem
privkey.pem

however RabbitMQ is still throwing:

TLS server: In state certify received CLIENT ALERT: Fatal - Certificate Unknown

1 Like
16

@schoen we've confirmed a few key pieces of information.

RMQ is not using the combined.pem that we were testing, it only uses:

Removed old config - unneeded for future reading of this thread.

We've updated the script that calls certbot to only build:

tls.crt
tls.nochain.crt
tls.key

We've confirmed that if we use the old (soon to expire later today) certs (those three components above), that RMQ is working and our GCP ingress (load balancers) continue to work. In this test:

tls.crt contains the end-entity cert and the intermediate cert

tls.nochain.crt contains the end-entity cert

tls.key contains the private key

Moving forward, in order to renew, do you think the nochain cert should instead use the intermediate cert?

Also, due to testing, we now receive:

An unexpected error occurred:
There were too many requests of a given type :: Error creating new order :: too many certificates already issued for exact set of domains: ... : see https://letsencrypt.org/docs/rate-limits/

Any tips on that are also appreciated.

1 Like
17

I'm not familiar with RabbitMQ configuration of this specifically, but something labeled "cacert" should probably be holding the roots ("ISRG Root X1" and "DST Root CA X3") rather than anything specific to your hosts.

That means you're hitting the rate limits from making more certificates. Your certificates are fine; making more of them is just adding strain to Let's Encrypt's servers. You just need to configure your system to use the certificates, not make new ones.

1 Like
18

Thanks @petercooperjr - good feedback.

On the main issue with RMQ, yes agreed, their docs indicate that.

ssl_options.cacertfile = /etc/rabbitmq/ssl/tls.crt

is for "Certificate Authority (CA) bundle file path", which makes sense in general.

I'm just not sure how to, as you wrote: get it to "hold the roots ("ISRG Root X1" and "DST Root CA X3") " ...

What certbot command(s) do we need to use for that?

1 Like
19

From the Certificates page:

Download the following files from there:

  • Under Active "ISRG Root X1", download the self-signed "pem" file
  • Under Upcoming "ISRG Root X2" (X2 isn't used yet but I would suggest including it for future compatibility), also download the self-signed "pem" file.
  • And then further down the page, under "Cross Signing", download the "download a copy from us" link

Each of those files should start with -----BEGIN CERTIFICATE----- and end with -----END CERTIFICATE-----. Concatenate them all into one file (I'm assuming that's the format your system is looking for) and use that as your cacertfile.

Why this is more complicated than just, like, using a web browser, is that you're building your own "trust store" of what authorities issue certificates that your system should trust. Most people don't need to worry about this and can just use their system's trust store. (And in fact, it may be possible to find the right file on your system of your system's default trust store and you could just point the cacertfile path to that file, which may be easier, but here this will at least work if you only ever use and want to trust Let's Encrypt certificates.) At some point I want to write up an "intro to WebPKI" and what to consider when building a trust store, since this topic seems to come up a lot here, but I haven't done it yet. :slight_smile:

1 Like
20

Thanks again @petercooperjr for setting us on the right path here.

RMQ can be a bit challenging with respect to using SSL/TLS and there is some added complexity in running this infra application in Kubernetes (of which we are big fans and use extensively) - in terms of how the cluster nodes pickup renewed cert files using a K8s mounted volume and secrets that are base64-encoded (this is fine but makes tracing certs through the system a bit more involved). In one of our non-production environments, we seem to have gotten past the immediate issue (certificate unknown) by slow-rolling the cluster (pod by pod). Clearly something reset at runtime.

We also had some stale logic in our renewal script ... which has been addressed.

For the future, we are going to build a trust store as you noted to ensure reliability.

Everyone on this thread has been a great help, which is very much appreciated!

2 Likes