HP iLO4 SSL circular reference?

basilhendroff · October 28, 2020, 8:49pm

Hi! First-time poster. I've googled my way to a fairly old forum thread How-to: Automatic certificate install on an HP iLO4 management processor that I'm partway through, but now stuck on, and I'm hoping that there's a forum member who can shed some light on the issue I'm facing.

I'm trying to set up secure SSL communication for an HP ILO.

The third point down in the referenced thread suggests that, in order to fetch a CSR, I can enter a command of the form:

hpilo_cli -l -p yourilo4.domain.name certificate_signing_request country= state= locality= organization= organizational_unit= common_name=yourilo4.domain.name

However, when I enter the command I get the error shown in the screenshot below.

This seems to suggest that I need a secure connection first before I can fetch the CSR which I need to generate a certificate so I can set up a secure service. I seem to be chasing my tail here.

What I'm ultimately trying to do is script the renewal of the certificate for the ILO using hpilo_cli
I think what I have to do is, initially, manually perform these steps through the ILO GUI to set up a secure connection the first time before I script subsequent renewals. I just want to double-check this with someone before moving ahead with this approach, just in case I've missed something obvious in the referenced thread.

_az · October 28, 2020, 9:29pm

I think it's more to do with the fact that the installation of OpenSSL on your operating system does not support RC4. It's a legacy cipher, it may have been entirely compiled out by the maintainer of your Linux distribution.

The hpilo project assumes that it is available, leading to your crash.

You could try comment out this line and hope that your HP iLO device does not actually require RC4.

Otherwise, you could try run this tool from an older operating system where RC4 hadn't been removed yet. For example, Ubuntu 16.04.

Osiris · October 28, 2020, 9:47pm

I think the error is even before it tries to connect.

You can trace back the error listed at the top to the actual code:

github.com

seveas/python-hpilo/blob/main/hpilo.py#L397-L407


      
          try:
              if not self.ssl_context:
                  self.ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLS)
                  # Even more sadly, some iLOs are still using RC4-SHA
                  # which was dropped from the default cipher suite in
                  # Python 2.7.10 and Python 3.4.4. Add it back here :(
                  self.ssl_context.set_ciphers(ssl._DEFAULT_CIPHERS + ":RC4-SHA")
              return self.ssl_context.wrap_socket(
                  sock, server_hostname=self.hostname)
          except ssl.SSLError as exc:
              raise IloCommunicationError("Cannot establish ssl session with %s:%d: %s" % (self.hostname, self.port, str(exc)))

If we read the documentation from the ssl module and read about the SSLError:

We see that the error is actually from OpenSSL itself.

From the clang _ssl module code, we can take the possible values for ssl._DEFAULT_CIPHERS of the hpilo.py code, which actually is PY_SSL_DEFAULT_CIPHER_STRING from the clang code below:

github.com

python/cpython/blob/main/Modules/_ssl.c#L292-L310


      
               * We have to maintain our own copy. OpenSSL's hostflags default to 0.
               */
              unsigned int hostflags;
              int protocol;
          #ifdef TLS1_3_VERSION
              int post_handshake_auth;
          #endif
              PyObject *msg_cb;
              PyObject *keylog_filename;
              BIO *keylog_bio;
              /* Cached module state, also used in SSLSocket and SSLSession code. */
              _sslmodulestate *state;
              PyObject *psk_client_callback;
              PyObject *psk_server_callback;
          } PySSLContext;
          
          typedef struct {
              int ssl; /* last seen error from SSL */
              int c; /* last seen error from libc */

So the string passed to OpenSSL is either:

RC4-SHA:DEFAULT:!aNULL:!eNULL:!MD5:!3DES:!DES:!RC4:!IDEA:!SEED:!aDSS:!SRP:!PSK

or

RC4-SHA:ALL:!COMPLEMENTOFDEFAULT:!eNULL

(from https://github.com/openssl/openssl/blob/master/include/openssl/ssl.h.in#L190-L191)

Funny fact: my OpenSSL doesn't accept the first cipher string too, unless I remove RC4:

osiris@erazer ~ $ openssl ciphers -v 'RC4-SHA:DEFAULT:!aNULL:!eNULL:!MD5:!3DES:!DES:!RC4:!IDEA:!SEED:!aDSS:!SRP:!PSK'
Error in cipher list
140615628351296:error:1410D0B9:SSL routines:SSL_CTX_set_cipher_list:no cipher match:ssl/ssl_lib.c:2558:
osiris@erazer ~ $ openssl ciphers -v 'DEFAULT:!aNULL:!eNULL:!MD5:!3DES:!DES:!RC4:!IDEA:!SEED:!aDSS:!SRP:!PSK'
... list of cipher suits ...

The second version of the cipher suit string strangely enough does result in a lot of cipher suits.. However, it's very much plausible that your Python uses the first cipher suite string as the default.. Resulting in an OpenSSL error.

You can try this out:

openssl ciphers -v 'RC4-SHA:DEFAULT:!aNULL:!eNULL:!MD5:!3DES:!DES:!RC4:!IDEA:!SEED:!aDSS:!SRP:!PSK'

And try it without the RC4-SHA.

If the first one gives an error, but the second one works, it's probably because your OpenSSL version has removed support for RC4. If that's the case, you might be able to get things working by editing /usr/local/lib/python3.7/site-packages/hpilo.py and change:

                self.ssl_context.set_ciphers("RC4-SHA:" + ssl._DEFAULT_CIPHERS)

to:

                self.ssl_context.set_ciphers(ssl._DEFAULT_CIPHERS)

@_az beat me to it.. Took me too long to look up all the code.. And painfully made it obvious that editing the line isn't really necessary, as Python indeed probably defaults to the default

_az · October 28, 2020, 9:59pm

You know, it's weird that it crashes at all. I would have expected "we don't support RC4" to simply cause RC4-SHA to get ignored.

I think the problem is that when _DEFAULT_CIPHERS is

DEFAULT:!aNULL:!eNULL:!MD5:!3DES:!DES:!RC4:!IDEA:!SEED:!aDSS:!SRP:!PSK

prefixing anything to it will result in a crash, because nothing can appear ahead of DEFAULT.

So another "fix" could be:

self.ssl_context.set_ciphers(ssl._DEFAULT_CIPHERS.replace('!RC4', '') + ':RC4')

Of course, if the iLO device doesn't support AES and only supports RC4 (assuming your Linux installation doesn't support RC4), it still won't help. But I read that you might be enable to enable AES in the iLO device control panel.

basilhendroff · October 28, 2020, 10:21pm

How the hell...?! I'm gobsmacked! You guys are unbelievable!

OK. I need to back-pedal a little bit. I did not use the suggested template. My bad. I'm running Python 3.7 in a FreeNAS jail (FreeBSD 12.2-RC3).
Anyway, I made the change you both referred to and the error changes:

_az · October 28, 2020, 10:32pm

https://github.com/seveas/python-hpilo/issues/232 seems to have some discussion about that problem. Maybe hpilo isn't compatible with certain iLO devices?

basilhendroff · October 28, 2020, 10:47pm

This didn't work for me. OS difference maybe? The last line is a command recall.

Trying just openssl ciphers gives the following output:

TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:DHE-RSA-AES256-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES128-SHA:RSA-PSK-AES256-GCM-SHA384:DHE-PSK-AES256-GCM-SHA384:RSA-PSK-CHACHA20-POLY1305:DHE-PSK-CHACHA20-POLY1305:ECDHE-PSK-CHACHA20-POLY1305:AES256-GCM-SHA384:PSK-AES256-GCM-SHA384:PSK-CHACHA20-POLY1305:RSA-PSK-AES128-GCM-SHA256:DHE-PSK-AES128-GCM-SHA256:AES128-GCM-SHA256:PSK-AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:ECDHE-PSK-AES256-CBC-SHA384:ECDHE-PSK-AES256-CBC-SHA:SRP-RSA-AES-256-CBC-SHA:SRP-AES-256-CBC-SHA:RSA-PSK-AES256-CBC-SHA384:DHE-PSK-AES256-CBC-SHA384:RSA-PSK-AES256-CBC-SHA:DHE-PSK-AES256-CBC-SHA:AES256-SHA:PSK-AES256-CBC-SHA384:PSK-AES256-CBC-SHA:ECDHE-PSK-AES128-CBC-SHA256:ECDHE-PSK-AES128-CBC-SHA:SRP-RSA-AES-128-CBC-SHA:SRP-AES-128-CBC-SHA:RSA-PSK-AES128-CBC-SHA256:DHE-PSK-AES128-CBC-SHA256:RSA-PSK-AES128-CBC-SHA:DHE-PSK-AES128-CBC-SHA:AES128-SHA:PSK-AES128-CBC-SHA256:PSK-AES128-CBC-SHA

Still no go.

Possibly, but I'm surprised as the HP Gen8 microserver the ILO (iLO4 not iLO3 as suggested in the issue) is embedded in isn't that old and I wouldn't designate yet as legacy hardware. Still, you may be right. I might raise and issue with the developer, reference this thread and see what he suggests.

basilhendroff · October 28, 2020, 10:54pm

Sorry, as a new member, my responses are being vetted.

rg305 · October 29, 2020, 12:53am

Has HP released any updates to the firmware/bios that covers their iLO?

READERS: Get involved and participate: If you read something you like, then click to like it

xyzzy · October 29, 2020, 7:44am

Apply the following patch:
--- /usr/local/lib/python3.7/site-packages/hpilo.py~ 2020-10-06 19:52:33.000000000 +0200
+++ /usr/local/lib/python3.7/site-packages/hpilo.py 2020-10-29 08:41:25.082982000 +0100
@@ -400,7 +400,7 @@
# Even more sadly, some iLOs are still using RC4-SHA
# which was dropped from the default cipher suite in
# Python 2.7.10 and Python 3.4.4. Add it back here

           self.ssl_context.set_ciphers("RC4-SHA:" + ssl._DEFAULT_CIPHERS)

           self.ssl_context.set_ciphers(ssl._DEFAULT_CIPHERS)
       return self.ssl_context.wrap_socket(
           sock, server_hostname=self.hostname)
   except ssl.SSLError as exc:

The problem is that RC4-SHA is no longer available in openssl. Thus this call will fail. Your iLO doesn't need RC4-SHA (and you have upgraded to 2.75?)

regards, Matthias

basilhendroff · October 29, 2020, 9:10am

I'm at the latest 2.75 release.

Altering the python script as suggested and rerunning hpilo_cli:

root@ilo:/usr/local/lib/python3.7/site-packages # hpilo_cli -l Administrator -i ilo.udance.com.au certificate_signing_request country= state= locality= organization= organizational_unit= common_name=ilo.udance.com.au
Password for Administrator@ilo.udance.com.au:
Traceback (most recent call last):
  File "/usr/local/bin/hpilo_cli", line 334, in <module>
    main()
  File "/usr/local/bin/hpilo_cli", line 214, in main
    results = [getattr(ilo, method)(**params)]
  File "/usr/local/lib/python3.7/site-packages/hpilo.py", line 877, in certificate_signing_request
    return self._control_tag('RIB_INFO', 'CERTIFICATE_SIGNING_REQUEST', elements=elements)
  File "/usr/local/lib/python3.7/site-packages/hpilo.py", line 764, in _control_tag
    header, message = self._request(root)
  File "/usr/local/lib/python3.7/site-packages/hpilo.py", line 238, in _request
    self._detect_protocol()
  File "/usr/local/lib/python3.7/site-packages/hpilo.py", line 278, in _detect_protocol
    header, data = self._communicate(b'<RIBCL VERSION="2.0"></RIBCL>', ILO_HTTP, save=False)
  File "/usr/local/lib/python3.7/site-packages/hpilo.py", line 509, in _communicate
    if header['transfer-encoding'] == 'chunked':
KeyError: 'transfer-encoding'
root@ilo:/usr/local/lib/python3.7/site-packages #

@_az may be right. I'll check with the developer to see if my ILO is affected.

basilhendroff · October 29, 2020, 9:27am

Issue raised here https://github.com/seveas/python-hpilo/issues/241

xyzzy · October 29, 2020, 10:32am

Your iLO (2.75 on a hp microserver g8) is ok. I'm running the same version and with my patch I can upload a certificate.

regarsd, Matthias

basilhendroff · October 29, 2020, 6:29pm

I'm closing the issue at https://github.com/seveas/python-hpilo/issues/241 while I investigate further.

basilhendroff · October 29, 2020, 11:32pm

I've made some progress, but have now come up against another stumbling block. So this is what's unfolded so far...

First, I might have got further if I had my local DNS server resolve ilo.udance.com.au to the IP address of the ILO Once past that point, I managed to fetch the CSR from the ILO.

root@ilo:~ # hpilo_cli -l Administrator -i ilo.udance.com.au certificate_signing_request country= state= locality= organization= organizational_unit= common_name=ilo.udance.com.au
Password for Administrator@ilo.udance.com.au:
Traceback (most recent call last):
  File "/usr/local/bin/hpilo_cli", line 334, in <module>
    main()
  File "/usr/local/bin/hpilo_cli", line 214, in main
    results = [getattr(ilo, method)(**params)]
  File "/usr/local/lib/python3.7/site-packages/hpilo.py", line 877, in certificate_signing_request
    return self._control_tag('RIB_INFO', 'CERTIFICATE_SIGNING_REQUEST', elements=elements)
  File "/usr/local/lib/python3.7/site-packages/hpilo.py", line 764, in _control_tag
    header, message = self._request(root)
  File "/usr/local/lib/python3.7/site-packages/hpilo.py", line 253, in _request
    message = self._parse_message(data)
  File "/usr/local/lib/python3.7/site-packages/hpilo.py", line 618, in _parse_message
    raise subclass(message, status)
hpilo.IloGeneratingCSR: The iLO subsystem is currently generating a Certificate Signing Request(CSR), run script after 10 minutes or more to receive the CSR.

Repeating the command a few minutes later:

root@ilo:~ # hpilo_cli -l Administrator -i ilo.udance.com.au certificate_signing_request country= state= locality= organization= organizational_unit= common_name=ilo.udance.com.au > ilo.csr
Password for Administrator@ilo.udance.com.au:

Contents of ilo.csr

>>> print(my_ilo.certificate_signing_request(country="", state="", locality="", organization="", organizational_unit="", common_name="ilo.udance.com.au"))
-----BEGIN CERTIFICATE REQUEST-----
[REDACTED]
-----END CERTIFICATE REQUEST-----

Using the acme.sh client and Cloudflare DNS API, I then issued the CSR to Let's Encrypt to generate a certificate.

root@ilo:~ # acme.sh --signcsr --csr ilo.csr --dns dns_cf
[Fri Oct 30 04:39:47 AWST 2020] Copy csr to: /root/.acme.sh/ilo.udance.com.au/ilo.uda
[Fri Oct 30 04:39:48 AWST 2020] Using CA: https://acme-v02.api.letsencrypt.org/direct
[Fri Oct 30 04:39:49 AWST 2020] Create account key ok.
[Fri Oct 30 04:39:49 AWST 2020] Registering account: https://acme-v02.api.letsencrypt
[Fri Oct 30 04:39:52 AWST 2020] Registered
[Fri Oct 30 04:39:52 AWST 2020] ACCOUNT_THUMBPRINT='ZAit-0gx4lTGwxDuPXXCDGBdepiJaKpbR
[Fri Oct 30 04:39:52 AWST 2020] Signing from existing CSR.
[Fri Oct 30 04:39:52 AWST 2020] Getting domain auth token for each domain
[Fri Oct 30 04:39:54 AWST 2020] Getting webroot for domain='ilo.udance.com.au'
[Fri Oct 30 04:39:54 AWST 2020] Adding txt value: 5C9-HoPF-69bT1Y0kNqQoFJaQFmG4DkbDK0omain:  _acme-challenge.ilo.udance.com.au
[Fri Oct 30 04:39:56 AWST 2020] Adding record
[Fri Oct 30 04:39:56 AWST 2020] Added, OK
[Fri Oct 30 04:39:56 AWST 2020] The txt record is added: Success.
[Fri Oct 30 04:39:56 AWST 2020] Let's check each DNS record now. Sleep 20 seconds fir
[Fri Oct 30 04:40:17 AWST 2020] Checking ilo.udance.com.au for _acme-challenge.ilo.ud
[Fri Oct 30 04:40:18 AWST 2020] Domain ilo.udance.com.au '_acme-challenge.ilo.udance.s.
[Fri Oct 30 04:40:18 AWST 2020] All success, let's return
[Fri Oct 30 04:40:18 AWST 2020] Verifying: ilo.udance.com.au
[Fri Oct 30 04:40:22 AWST 2020] Pending
[Fri Oct 30 04:40:26 AWST 2020] Pending
[Fri Oct 30 04:40:29 AWST 2020] Pending
[Fri Oct 30 04:40:32 AWST 2020] Pending
[Fri Oct 30 04:40:36 AWST 2020] Pending
[Fri Oct 30 04:40:39 AWST 2020] Pending
[Fri Oct 30 04:40:42 AWST 2020] Success
[Fri Oct 30 04:40:42 AWST 2020] Removing DNS records.
[Fri Oct 30 04:40:42 AWST 2020] Removing txt: 5C9-HoPF-69bT1Y0kNqQoFJaQFmG4DkbDK0qxESn: _acme-challenge.ilo.udance.com.au
[Fri Oct 30 04:40:44 AWST 2020] Removed: Success
[Fri Oct 30 04:40:44 AWST 2020] Verify finished, start to sign.
[Fri Oct 30 04:40:44 AWST 2020] Lets finalize the order.
[Fri Oct 30 04:40:44 AWST 2020] Le_OrderFinalize='https://acme-v02.api.letsencrypt.ore/100737052/5937392711'
[Fri Oct 30 04:40:46 AWST 2020] Downloading cert.
[Fri Oct 30 04:40:46 AWST 2020] Le_LinkCert='https://acme-v02.api.letsencrypt.org/acmbb41a491c15b91b4ba45343098972'
[Fri Oct 30 04:40:47 AWST 2020] Cert success.
-----BEGIN CERTIFICATE-----
[REDACTED]
-----END CERTIFICATE-----
[Fri Oct 30 04:40:47 AWST 2020] Your cert is in  /root/.acme.sh/ilo.udance.com.au/ilo.cer
[Fri Oct 30 04:40:47 AWST 2020] The intermediate CA cert is in  /root/.acme.sh/ilo.ud.cer
[Fri Oct 30 04:40:47 AWST 2020] And the full chain certs is there:  /root/.acme.sh/ilu/fullchain.cer

Attempting to install the resulting certificate...

root@ilo:~/.acme.sh/ilo.udance.com.au # hpilo_cli -l Administrator -i ilo.udance.com.au import_certificate certificate=ilo.udance.com.au.cer
Password for Administrator@ilo.udance.com.au:
Traceback (most recent call last):
  File "/usr/local/bin/hpilo_cli", line 334, in <module>
    main()
  File "/usr/local/bin/hpilo_cli", line 214, in main
    results = [getattr(ilo, method)(**params)]
  File "/usr/local/lib/python3.7/site-packages/hpilo.py", line 1390, in import_certificate
    return self._control_tag('RIB_INFO', 'IMPORT_CERTIFICATE', text=certificate)
  File "/usr/local/lib/python3.7/site-packages/hpilo.py", line 764, in _control_tag
    header, message = self._request(root)
  File "/usr/local/lib/python3.7/site-packages/hpilo.py", line 253, in _request
    message = self._parse_message(data)
  File "/usr/local/lib/python3.7/site-packages/hpilo.py", line 619, in _parse_message
    raise IloError(message, status)
hpilo.IloError: Error: Line #1: syntax error near "ilo.cer".. You may have tried to use a feature this iLO version or firmware version does not support.

I'm so close, but I'm now stumped again.

EDIT: I've finally got it working. The syntax was incorrect. Created a short script to test with the lines...

CERTFILE="/root/.acme.sh/ilo.udance.com.au/ilo.udance.com.au.cer"
hpilo_cli -l Administrator -p [REDACTED] ilo.udance.com.au import_certificate certificate="$(cat $CERTFILE)"

...and the import succeeded!

Thank you @xyzzy and to everybody else who helped me along the way to try to figure out a way to get ahead.

griffin · October 30, 2020, 12:42am

Congratulations!

basilhendroff · October 30, 2020, 12:58am

When things change so quickly in the IT world, it's a pleasant surprise to find that a three-year-old
'How-To' thread mostly still works. It just needed a gentle nudge from @xyzzy. I'll reopen the issue with the developer to see if he would like to consider amending the python script.

griffin · October 30, 2020, 12:59am

Now if I can just secure my Commodore 64...

rg305 · October 30, 2020, 2:00am

Hey! My Vic 20 was inline first - LOL

system · November 29, 2020, 2:00am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
How-to: Automatic certificate install on an HP iLO4 management processor Server	1	15035	April 30, 2017
Installing SSL Certificates on Dell DRAC/HP iLO Management Cards Server	18	11965	March 30, 2017
Deploying certs to HPE iLO Client dev	5	1134	May 12, 2024
Cli.ini and key-path or csr parameter Server	2	1692	June 4, 2016
Manual certificate renewal Help	2	584	January 3, 2023

HP iLO4 SSL circular reference?

Related topics