Deploy certs to IPMI BMC?

The topic is admittedly a little off-topic for Let's Encrypt as such, but as automation is among the core values...

I have a few servers that have a baseboard management controller, an embedded computer that lets me manage the system remotely (including IP KVM capability)--a handy feature for servers that are typically headless. The system gives a web GUI independent of the OS. I have four nodes each of a Dell C6100 and C6220 II, and a couple of Supermicro X9 series motherboards. The Dells force SSL, naturally with a self-signed certificate by default; the Supermicro boards have SSL as an option, but default to HTTP. I'm not aware of any API for any of these systems, though I can't guarantee there's no such thing--but my Google-fu hasn't found anything.

Naturally, it'd be nice to put a trusted cert on these systems, and of course Let's Encrypt is good at providing those--and I already use DNS validation for lots of stuff, so getting the cert won't be a problem.

But the remaining question is, is it possible (and if so, of course, how) to automate the cert updates on these systems? I've found this:

...which may help on the Supermicro side (edit: or not; it seems to only work with X10 and newer series boards, not my X9s). But I haven't yet found anything for the Dell systems. Does anyone know of anything suitable?

2 Likes

I have a feeling you won't find much outside of whatever the manufacturer releases.
And even then, today it might be more of a manual process than one would agree to do on a more than once basis. If so, you might opt for a much longer expiry via your own CA.
But since the only constant in IT is change, it's only a matter of time before they will have that menu driven and fully automated (but that may require an upgrade to their latest/greatest firmware/hardware).

Edit: On reread, I think I haven't really added much to the topic :frowning:
Unfortunately, I come from an environment that everything is behind a firewall (even the firewalls! LOL).
So the iLo and iDRAC TLS certs (for security) aren't quite as relevant.
I will ask around to see if anyone knows anything...

3 Likes

It's something I'm considering. The Smallstep CA wants to issue very short-lived certs, but it could be convinced to make them longer-lasting. Not the way I'd like to do it, but I'm not ruling it out.

3 Likes

Do those IPMI interfaces have a method of installing a certificate through their webinterface at least? If so, it shouldn't be toooo hard to just use something like curl to POST a cert to the interface.. Although perhaps some login issues could arise.

The script you've posted should do nicely for Supermicro though. Not sure what your question is if you already have a solution for half of your servers?

3 Likes

Yes, they do.

So I'd thought, but note my edit--they only work on newer boards than what I have. Though I may be able to adapt it, I suppose.

Even if I did, the other half remain.

3 Likes

This "cry for help" is in practice probably more a request for "How good is your Google FU?" than actually having someone tell us "I did it like this and that, here is my script!" I guess. Chances are very slim someone on this Community has something like that laying around.

And it would probably help very much in the "Google FU" skillz department if we'd know on what we need to search.. For example, if I search on "dell ipmi" I'm getting results like Dell PowerEdge : How to import an externally created custom certificate and private key into the iDRAC, but I don't know if your systems are also using "iDRAC"? Frankly, I have no idea what "iDRAC" actually is? So perhaps you could provide us with more keywords to search for? Also, do you also use the racadm application mentioned in the above documentation?

3 Likes

iDRAC is a remote management system that Dell uses on some, but not all, of their servers (and why on only some of them, I have no idea). It uses IPMI in some way, and provides many (perhaps all) of the same remote monitoring and control capabilities, but also provides additional capabilities that the non-DRAC IPMI BMC doesn't (like, e.g., Active Directory authentication). But it isn't what my systems are using, and indeed it isn't available on them.

I'm really not sure what other keywords might be relevant. Server model? It's a PowerEdge C6220 Gen II. Exclude iDRAC from the Google search? That helps a bit, but what's coming up are still iterations of what I already know--how to do it through the web UI. It's an ASPEED controller, if that's relevant.

Edit: But some further messing around with the Dell system makes it look like you have to generate a CSR through its web interface, get that signed, then upload the resulting certificate--you can't upload just a cert and key. Yuck. Do-able, but ugly. But it will apply the new cert promptly, so I guess that's a win. Whether I can upload a subsequent cert based on the same CSR (vs. creating a new CSR each time) is something I'll have to investigate later--if so, this might not be too painful.

3 Likes

I reckon the CSR download will be one-time, it's unlikely to be generating a new private key every time and your host names will be staying the same, so I think you can re-use the CSR.

The key point with these scripts is they will be brittle if the web UI changes, they need to understand and preserve the login session cookies (so you can login, then post to the certificate update page while authenticated). The linked script looks like exactly the sort of thing you'd need but things like the form fields and exact flow will vary.

Ideally you need a test system to play with, because debugging the script live isn't going to be acceptable usually.

3 Likes

Fortunately (at least for these purposes), updates to the BMC firmware are rare; the last update for the system I'm dealing with right now was in 2015, and I'd be kind of surprised if Dell ever publishes another one (for better or worse). So if I can get this working, I don't need to be too concerned about future breakage.

Not ideal, but less disruptive than I'd seen suggested--some of what I'd found (more on the Supermicro side than the Dell side) had suggested that the system needs to reboot after uploading a new cert, which would be a complete deal-breaker. But as it happens, it's only the BMC that needs to reboot--which still seems like it shouldn't be necessary, but not as bad as I'd thought. But still, not really something I want to be messing with on a production system. For the time being, anyway, I have a "spare" blade in this system that should do.

4 Likes

Turns out that no, I can't do this. If I sign a second cert on the same CSR and try to upload it, I get this:
image

Seems like a thoroughly idiotic design decision. I can kind of understand not wanting to import a private key--this method makes sure the private key never leaves the IPMI controller--but forbidding reuse of the csr is just stupid.

So the process on the Dell system looks like:

  • Generate CSR through the web UI
    • CN, org name, OU, locality, state are all mandatory
  • Sign CSR
  • Upload the signed certificate

Clearly not designed with any thought toward automation. And there's no provision for an intermediate signing cert--there's no separate place to upload one, and if you upload, e.g., fullchain.pem, the intermediate cert is stripped out. Though in Dell's defense (or that of their vendor), the last update for this firmware was in 2015, before Let's Encrypt started issuing certificates and automation started to become important (but not, I think, before certs were routinely signed by intermediate CAs). But I think I need to look into how to script interaction with this web UI.

It actually is. Good for avoiding key reuse, I guess...

3 Likes

Sounds like it will be getting the likes of a 10+ year private cert to me.
[I would NOT put myself through that every 60-90 days!]

2 Likes

Well, in that case you can probably say bye bye to many if not all public CAs? As far as I know, it's even a requirement of the Baseline Requirements to use an intermediate certificate to sign end leaf certs..

4 Likes

Even my local CA signs using an intermediate cert. But browsers behave differently. Firefox apparently has the intermediate cached from other places on my LAN I use a cert from that CA, and gives no complaints, but Chrome still complains. I imagine this would work similarly with a Let's Encrypt cert, in that some browsers would remember the intermediate and others would complain. Certainly not ideal, though.

No, not if I had to do it manually. If it could be effectively automated, it could happen daily for all I care.

But it doesn't look like there's a tool out there to automate it, so if I want one, I'll need to make it myself. And that may be a bit beyond my skill level. Is there a tool available to log what you're doing in the browser, and then show the individual requests and associated data? I'm thinking that implementing this wouldn't be too hard using the Python requests library, but knowing exactly what to implement with that is another story.

3 Likes

Chrome (and Chrome based Edge) has a nice feature in their Dev Tools (press F12 to open) that lets you copy individual browser requests out as a CLI command in various languages (curl, PowerShell, Node, etc).

Go to the Network tab and do whatever you need to do on the site and you'll see the various requests the browser is making in a list. Right click one of them, open the Copy sub-menu, and choose an option like Copy as PowerShell, then paste the result into a text editor.

I haven't actually used it much, so I don't know how well it would work for something complicated like a file upload. But it's worth a shot.

4 Likes

I don't know either about Chrome/Chromium, but curl can of course do such a thing easily, so in theory it shouldn't be too hard for Chrome/Chromium to implement that copy thingy for file uploads.

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.