HP iLO4 SSL circular reference?

Hi! First-time poster. I've googled my way to a fairly old forum thread How-to: Automatic certificate install on an HP iLO4 management processor that I'm partway through, but now stuck on, and I'm hoping that there's a forum member who can shed some light on the issue I'm facing.

I'm trying to set up secure SSL communication for an HP ILO.

The third point down in the referenced thread suggests that, in order to fetch a CSR, I can enter a command of the form:

hpilo_cli -l -p yourilo4.domain.name certificate_signing_request country= state= locality= organization= organizational_unit= common_name=yourilo4.domain.name

However, when I enter the command I get the error shown in the screenshot below.

This seems to suggest that I need a secure connection first before I can fetch the CSR which I need to generate a certificate so I can set up a secure service. I seem to be chasing my tail here.

What I'm ultimately trying to do is script the renewal of the certificate for the ILO using hpilo_cli
I think what I have to do is, initially, manually perform these steps through the ILO GUI to set up a secure connection the first time before I script subsequent renewals. I just want to double-check this with someone before moving ahead with this approach, just in case I've missed something obvious in the referenced thread.

1 Like

I think it's more to do with the fact that the installation of OpenSSL on your operating system does not support RC4. It's a legacy cipher, it may have been entirely compiled out by the maintainer of your Linux distribution.

The hpilo project assumes that it is available, leading to your crash.

You could try comment out this line and hope that your HP iLO device does not actually require RC4.

Otherwise, you could try run this tool from an older operating system where RC4 hadn't been removed yet. For example, Ubuntu 16.04.

1 Like

I think the error is even before it tries to connect.

You can trace back the error listed at the top to the actual code:

If we read the documentation from the ssl module and read about the SSLError:

We see that the error is actually from OpenSSL itself.

From the clang _ssl module code, we can take the possible values for ssl._DEFAULT_CIPHERS of the hpilo.py code, which actually is PY_SSL_DEFAULT_CIPHER_STRING from the clang code below:

So the string passed to OpenSSL is either:

RC4-SHA:DEFAULT:!aNULL:!eNULL:!MD5:!3DES:!DES:!RC4:!IDEA:!SEED:!aDSS:!SRP:!PSK

or

RC4-SHA:ALL:!COMPLEMENTOFDEFAULT:!eNULL

(from https://github.com/openssl/openssl/blob/master/include/openssl/ssl.h.in#L190-L191)

Funny fact: my OpenSSL doesn't accept the first cipher string too, unless I remove RC4:

osiris@erazer ~ $ openssl ciphers -v 'RC4-SHA:DEFAULT:!aNULL:!eNULL:!MD5:!3DES:!DES:!RC4:!IDEA:!SEED:!aDSS:!SRP:!PSK'
Error in cipher list
140615628351296:error:1410D0B9:SSL routines:SSL_CTX_set_cipher_list:no cipher match:ssl/ssl_lib.c:2558:
osiris@erazer ~ $ openssl ciphers -v 'DEFAULT:!aNULL:!eNULL:!MD5:!3DES:!DES:!RC4:!IDEA:!SEED:!aDSS:!SRP:!PSK'
... list of cipher suits ...

The second version of the cipher suit string strangely enough does result in a lot of cipher suits.. However, it's very much plausible that your Python uses the first cipher suite string as the default.. Resulting in an OpenSSL error.

You can try this out:

openssl ciphers -v 'RC4-SHA:DEFAULT:!aNULL:!eNULL:!MD5:!3DES:!DES:!RC4:!IDEA:!SEED:!aDSS:!SRP:!PSK'

And try it without the RC4-SHA.

If the first one gives an error, but the second one works, it's probably because your OpenSSL version has removed support for RC4. If that's the case, you might be able to get things working by editing /usr/local/lib/python3.7/site-packages/hpilo.py and change:

                self.ssl_context.set_ciphers("RC4-SHA:" + ssl._DEFAULT_CIPHERS)

to:

                self.ssl_context.set_ciphers(ssl._DEFAULT_CIPHERS)

:sob: @_az beat me to it.. Took me too long to look up all the code.. And painfully made it obvious that editing the line isn't really necessary, as Python indeed probably defaults to the default :laughing:

4 Likes

You know, it's weird that it crashes at all. I would have expected "we don't support RC4" to simply cause RC4-SHA to get ignored.

I think the problem is that when _DEFAULT_CIPHERS is

DEFAULT:!aNULL:!eNULL:!MD5:!3DES:!DES:!RC4:!IDEA:!SEED:!aDSS:!SRP:!PSK

prefixing anything to it will result in a crash, because nothing can appear ahead of DEFAULT.

So another "fix" could be:

self.ssl_context.set_ciphers(ssl._DEFAULT_CIPHERS.replace('!RC4', '') + ':RC4')

Of course, if the iLO device doesn't support AES and only supports RC4 (assuming your Linux installation doesn't support RC4), it still won't help. But I read that you might be enable to enable AES in the iLO device control panel.

3 Likes

How the hell...?! I'm gobsmacked! You guys are unbelievable!

OK. I need to back-pedal a little bit. I did not use the suggested template. My bad. I'm running Python 3.7 in a FreeNAS jail (FreeBSD 12.2-RC3).
Anyway, I made the change you both referred to and the error changes:

2 Likes

https://github.com/seveas/python-hpilo/issues/232 seems to have some discussion about that problem. Maybe hpilo isn't compatible with certain iLO devices?

2 Likes

This didn't work for me. OS difference maybe? The last line is a command recall.

Trying just openssl ciphers gives the following output:

TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:DHE-RSA-AES256-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES128-SHA:RSA-PSK-AES256-GCM-SHA384:DHE-PSK-AES256-GCM-SHA384:RSA-PSK-CHACHA20-POLY1305:DHE-PSK-CHACHA20-POLY1305:ECDHE-PSK-CHACHA20-POLY1305:AES256-GCM-SHA384:PSK-AES256-GCM-SHA384:PSK-CHACHA20-POLY1305:RSA-PSK-AES128-GCM-SHA256:DHE-PSK-AES128-GCM-SHA256:AES128-GCM-SHA256:PSK-AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:ECDHE-PSK-AES256-CBC-SHA384:ECDHE-PSK-AES256-CBC-SHA:SRP-RSA-AES-256-CBC-SHA:SRP-AES-256-CBC-SHA:RSA-PSK-AES256-CBC-SHA384:DHE-PSK-AES256-CBC-SHA384:RSA-PSK-AES256-CBC-SHA:DHE-PSK-AES256-CBC-SHA:AES256-SHA:PSK-AES256-CBC-SHA384:PSK-AES256-CBC-SHA:ECDHE-PSK-AES128-CBC-SHA256:ECDHE-PSK-AES128-CBC-SHA:SRP-RSA-AES-128-CBC-SHA:SRP-AES-128-CBC-SHA:RSA-PSK-AES128-CBC-SHA256:DHE-PSK-AES128-CBC-SHA256:RSA-PSK-AES128-CBC-SHA:DHE-PSK-AES128-CBC-SHA:AES128-SHA:PSK-AES128-CBC-SHA256:PSK-AES128-CBC-SHA

Still no go.

Possibly, but I'm surprised as the HP Gen8 microserver the ILO (iLO4 not iLO3 as suggested in the issue) is embedded in isn't that old and I wouldn't designate yet as legacy hardware. Still, you may be right. I might raise and issue with the developer, reference this thread and see what he suggests.

1 Like

Sorry, as a new member, my responses are being vetted.

1 Like

Has HP released any updates to the firmware/bios that covers their iLO?

READERS: Get involved and participate: If you read something you like, then click to like it :heart:

3 Likes

Apply the following patch:
--- /usr/local/lib/python3.7/site-packages/hpilo.py~ 2020-10-06 19:52:33.000000000 +0200
+++ /usr/local/lib/python3.7/site-packages/hpilo.py 2020-10-29 08:41:25.082982000 +0100
@@ -400,7 +400,7 @@
# Even more sadly, some iLOs are still using RC4-SHA
# which was dropped from the default cipher suite in
# Python 2.7.10 and Python 3.4.4. Add it back here :frowning:

  •            self.ssl_context.set_ciphers("RC4-SHA:" + ssl._DEFAULT_CIPHERS)
    
  •            self.ssl_context.set_ciphers(ssl._DEFAULT_CIPHERS)
           return self.ssl_context.wrap_socket(
               sock, server_hostname=self.hostname)
       except ssl.SSLError as exc:
    

The problem is that RC4-SHA is no longer available in openssl. Thus this call will fail. Your iLO doesn't need RC4-SHA (and you have upgraded to 2.75?)

regards, Matthias

3 Likes

I'm at the latest 2.75 release.

Altering the python script as suggested and rerunning hpilo_cli:

root@ilo:/usr/local/lib/python3.7/site-packages # hpilo_cli -l Administrator -i ilo.udance.com.au certificate_signing_request country= state= locality= organization= organizational_unit= common_name=ilo.udance.com.au
Password for Administrator@ilo.udance.com.au:
Traceback (most recent call last):
  File "/usr/local/bin/hpilo_cli", line 334, in <module>
    main()
  File "/usr/local/bin/hpilo_cli", line 214, in main
    results = [getattr(ilo, method)(**params)]
  File "/usr/local/lib/python3.7/site-packages/hpilo.py", line 877, in certificate_signing_request
    return self._control_tag('RIB_INFO', 'CERTIFICATE_SIGNING_REQUEST', elements=elements)
  File "/usr/local/lib/python3.7/site-packages/hpilo.py", line 764, in _control_tag
    header, message = self._request(root)
  File "/usr/local/lib/python3.7/site-packages/hpilo.py", line 238, in _request
    self._detect_protocol()
  File "/usr/local/lib/python3.7/site-packages/hpilo.py", line 278, in _detect_protocol
    header, data = self._communicate(b'<RIBCL VERSION="2.0"></RIBCL>', ILO_HTTP, save=False)
  File "/usr/local/lib/python3.7/site-packages/hpilo.py", line 509, in _communicate
    if header['transfer-encoding'] == 'chunked':
KeyError: 'transfer-encoding'
root@ilo:/usr/local/lib/python3.7/site-packages #

@_az may be right. I'll check with the developer to see if my ILO is affected.

2 Likes

Issue raised here https://github.com/seveas/python-hpilo/issues/241

3 Likes

Your iLO (2.75 on a hp microserver g8) is ok. I'm running the same version and with my patch I can upload a certificate.

regarsd, Matthias

2 Likes

I'm closing the issue at https://github.com/seveas/python-hpilo/issues/241 while I investigate further.

2 Likes

I've made some progress, but have now come up against another stumbling block. So this is what's unfolded so far...

First, I might have got further if I had my local DNS server resolve ilo.udance.com.au to the IP address of the ILO :roll_eyes: Once past that point, I managed to fetch the CSR from the ILO.

root@ilo:~ # hpilo_cli -l Administrator -i ilo.udance.com.au certificate_signing_request country= state= locality= organization= organizational_unit= common_name=ilo.udance.com.au
Password for Administrator@ilo.udance.com.au:
Traceback (most recent call last):
  File "/usr/local/bin/hpilo_cli", line 334, in <module>
    main()
  File "/usr/local/bin/hpilo_cli", line 214, in main
    results = [getattr(ilo, method)(**params)]
  File "/usr/local/lib/python3.7/site-packages/hpilo.py", line 877, in certificate_signing_request
    return self._control_tag('RIB_INFO', 'CERTIFICATE_SIGNING_REQUEST', elements=elements)
  File "/usr/local/lib/python3.7/site-packages/hpilo.py", line 764, in _control_tag
    header, message = self._request(root)
  File "/usr/local/lib/python3.7/site-packages/hpilo.py", line 253, in _request
    message = self._parse_message(data)
  File "/usr/local/lib/python3.7/site-packages/hpilo.py", line 618, in _parse_message
    raise subclass(message, status)
hpilo.IloGeneratingCSR: The iLO subsystem is currently generating a Certificate Signing Request(CSR), run script after 10 minutes or more to receive the CSR.

Repeating the command a few minutes later:

root@ilo:~ # hpilo_cli -l Administrator -i ilo.udance.com.au certificate_signing_request country= state= locality= organization= organizational_unit= common_name=ilo.udance.com.au > ilo.csr
Password for Administrator@ilo.udance.com.au:

Contents of ilo.csr

>>> print(my_ilo.certificate_signing_request(country="", state="", locality="", organization="", organizational_unit="", common_name="ilo.udance.com.au"))
-----BEGIN CERTIFICATE REQUEST-----
[REDACTED]
-----END CERTIFICATE REQUEST-----

Using the acme.sh client and Cloudflare DNS API, I then issued the CSR to Let's Encrypt to generate a certificate.

root@ilo:~ # acme.sh --signcsr --csr ilo.csr --dns dns_cf
[Fri Oct 30 04:39:47 AWST 2020] Copy csr to: /root/.acme.sh/ilo.udance.com.au/ilo.uda
[Fri Oct 30 04:39:48 AWST 2020] Using CA: https://acme-v02.api.letsencrypt.org/direct
[Fri Oct 30 04:39:49 AWST 2020] Create account key ok.
[Fri Oct 30 04:39:49 AWST 2020] Registering account: https://acme-v02.api.letsencrypt
[Fri Oct 30 04:39:52 AWST 2020] Registered
[Fri Oct 30 04:39:52 AWST 2020] ACCOUNT_THUMBPRINT='ZAit-0gx4lTGwxDuPXXCDGBdepiJaKpbR
[Fri Oct 30 04:39:52 AWST 2020] Signing from existing CSR.
[Fri Oct 30 04:39:52 AWST 2020] Getting domain auth token for each domain
[Fri Oct 30 04:39:54 AWST 2020] Getting webroot for domain='ilo.udance.com.au'
[Fri Oct 30 04:39:54 AWST 2020] Adding txt value: 5C9-HoPF-69bT1Y0kNqQoFJaQFmG4DkbDK0omain:  _acme-challenge.ilo.udance.com.au
[Fri Oct 30 04:39:56 AWST 2020] Adding record
[Fri Oct 30 04:39:56 AWST 2020] Added, OK
[Fri Oct 30 04:39:56 AWST 2020] The txt record is added: Success.
[Fri Oct 30 04:39:56 AWST 2020] Let's check each DNS record now. Sleep 20 seconds fir
[Fri Oct 30 04:40:17 AWST 2020] Checking ilo.udance.com.au for _acme-challenge.ilo.ud
[Fri Oct 30 04:40:18 AWST 2020] Domain ilo.udance.com.au '_acme-challenge.ilo.udance.s.
[Fri Oct 30 04:40:18 AWST 2020] All success, let's return
[Fri Oct 30 04:40:18 AWST 2020] Verifying: ilo.udance.com.au
[Fri Oct 30 04:40:22 AWST 2020] Pending
[Fri Oct 30 04:40:26 AWST 2020] Pending
[Fri Oct 30 04:40:29 AWST 2020] Pending
[Fri Oct 30 04:40:32 AWST 2020] Pending
[Fri Oct 30 04:40:36 AWST 2020] Pending
[Fri Oct 30 04:40:39 AWST 2020] Pending
[Fri Oct 30 04:40:42 AWST 2020] Success
[Fri Oct 30 04:40:42 AWST 2020] Removing DNS records.
[Fri Oct 30 04:40:42 AWST 2020] Removing txt: 5C9-HoPF-69bT1Y0kNqQoFJaQFmG4DkbDK0qxESn: _acme-challenge.ilo.udance.com.au
[Fri Oct 30 04:40:44 AWST 2020] Removed: Success
[Fri Oct 30 04:40:44 AWST 2020] Verify finished, start to sign.
[Fri Oct 30 04:40:44 AWST 2020] Lets finalize the order.
[Fri Oct 30 04:40:44 AWST 2020] Le_OrderFinalize='https://acme-v02.api.letsencrypt.ore/100737052/5937392711'
[Fri Oct 30 04:40:46 AWST 2020] Downloading cert.
[Fri Oct 30 04:40:46 AWST 2020] Le_LinkCert='https://acme-v02.api.letsencrypt.org/acmbb41a491c15b91b4ba45343098972'
[Fri Oct 30 04:40:47 AWST 2020] Cert success.
-----BEGIN CERTIFICATE-----
[REDACTED]
-----END CERTIFICATE-----
[Fri Oct 30 04:40:47 AWST 2020] Your cert is in  /root/.acme.sh/ilo.udance.com.au/ilo.cer
[Fri Oct 30 04:40:47 AWST 2020] The intermediate CA cert is in  /root/.acme.sh/ilo.ud.cer
[Fri Oct 30 04:40:47 AWST 2020] And the full chain certs is there:  /root/.acme.sh/ilu/fullchain.cer

Attempting to install the resulting certificate...

root@ilo:~/.acme.sh/ilo.udance.com.au # hpilo_cli -l Administrator -i ilo.udance.com.au import_certificate certificate=ilo.udance.com.au.cer
Password for Administrator@ilo.udance.com.au:
Traceback (most recent call last):
  File "/usr/local/bin/hpilo_cli", line 334, in <module>
    main()
  File "/usr/local/bin/hpilo_cli", line 214, in main
    results = [getattr(ilo, method)(**params)]
  File "/usr/local/lib/python3.7/site-packages/hpilo.py", line 1390, in import_certificate
    return self._control_tag('RIB_INFO', 'IMPORT_CERTIFICATE', text=certificate)
  File "/usr/local/lib/python3.7/site-packages/hpilo.py", line 764, in _control_tag
    header, message = self._request(root)
  File "/usr/local/lib/python3.7/site-packages/hpilo.py", line 253, in _request
    message = self._parse_message(data)
  File "/usr/local/lib/python3.7/site-packages/hpilo.py", line 619, in _parse_message
    raise IloError(message, status)
hpilo.IloError: Error: Line #1: syntax error near "ilo.cer".. You may have tried to use a feature this iLO version or firmware version does not support.

I'm so close, but I'm now stumped again.

EDIT: I've finally got it working. The syntax was incorrect. Created a short script to test with the lines...

CERTFILE="/root/.acme.sh/ilo.udance.com.au/ilo.udance.com.au.cer"
hpilo_cli -l Administrator -p [REDACTED] ilo.udance.com.au import_certificate certificate="$(cat $CERTFILE)"

...and the import succeeded!

Thank you @xyzzy and to everybody else who helped me along the way to try to figure out a way to get ahead. :wave:

2 Likes

:partying_face:

Congratulations!

2 Likes

When things change so quickly in the IT world, it's a pleasant surprise to find that a three-year-old
'How-To' thread mostly still works. It just needed a gentle nudge from @xyzzy. I'll reopen the issue with the developer to see if he would like to consider amending the python script.

2 Likes

Now if I can just secure my Commodore 64... :wink:

2 Likes

Hey! My Vic 20 was inline first - LOL

2 Likes