OpenSSL bug information

During Boulder development we encountered a bug in certain versions of the OpenSSL API (our testing indicates this bug was fixed in the 1.0.2 release, although this is not definite) which causes parsing errors in the newer, stricter Golang 1.6 ASN.1 parser. This bug is triggered if a CSR or certificate is created using the OpenSSL API without explicitly setting the X509 version. In this case OpenSSL will not populate the version field with a zero value which creates a malformed ASN.1 integer definition.

While we were originally optimistic we could come up with a short-term solution to whitelist requests from existing users who had exhibited the issue this turned out to be far more complex than we originally expected. Since we need to migrate to Golang 1.6 relatively soon to continue getting the benefit of upstream security fixes we have decided to instead move to a hard cutoff date of November 2017.

It should be noted that you are unlikely to notice anything when we implement the cutoff, based on our internal metrics only 0.34% of all active users exhibit this issue.

For users

Typically updating your ACME client will fix this problem. For instance, Certbot (formerly known as letsencrypt or letsencrypt-auto) has fixed this problem as of version 0.4.1. If your client does not have a newer version available, you should update to the most recent release of OpenSSL.

This should be all you have to do to fix this issue and prepare your systems for the hard cutoff. If you would like to test your system we will be making changes to our staging environment in the upcoming weeks to allow users to explicitly test against a Golang 1.6 version of Boulder which will reject the malformed requests.

Iā€™ll update this post with further information as it becomes available.

For client developers

Make sure if you use a OpenSSL language binding to generate CSRs and certificates for tls-sni type challenges that you explicitly set the X509 version before signing and using the resulting CSR/certificate. The certificate must have a serial number.

2 Likes

Does certbot have this fixed and if so, since which version? (I couldnā€™t find an ā€˜issueā€™ op Github.)

Yup, certbot has been fixed since 0.4.1.

1 Like

How to check if a csr has this problem?

I tried openssl req -text -noout -in my.csr and see:

Certificate Request:
    Data:
        Version: 0 (0x0)
        Subject: ...

Does the "Version: 0" mean that this CSR exhibits the problem?

OpenSSL will itself compensate for the issue if you are using a buggy version. Instead you want to use openssl asn1parse which will produce an output that looks like this

$ openssl asn1parse -inform pem -in test.csr
    0:d=0  hl=4 l= 602 cons: SEQUENCE          
    4:d=1  hl=4 l= 322 cons: SEQUENCE          
    8:d=2  hl=2 l=   0 prim: INTEGER           :00
   10:d=2  hl=2 l=  22 cons: SEQUENCE          
   12:d=3  hl=2 l=  20 cons: SET               
   14:d=4  hl=2 l=  18 cons: SEQUENCE          
   16:d=5  hl=2 l=   3 prim: OBJECT            :commonName
   21:d=5  hl=2 l=  11 prim: PRINTABLESTRING   :example.com
   34:d=2  hl=4 l= 290 cons: SEQUENCE          
   38:d=3  hl=2 l=  13 cons: SEQUENCE          
   40:d=4  hl=2 l=   9 prim: OBJECT            :rsaEncryption
   51:d=4  hl=2 l=   0 prim: NULL              
   53:d=3  hl=4 l= 271 prim: BIT STRING        
  328:d=2  hl=2 l=   0 cons: cont [ 0 ]        
  330:d=1  hl=2 l=  13 cons: SEQUENCE          
  332:d=2  hl=2 l=   9 prim: OBJECT            :sha256WithRSAEncryption
  343:d=2  hl=2 l=   0 prim: NULL              
  345:d=1  hl=4 l= 257 prim: BIT STRING

The third line shows that this CSR exhibits the issue, it has a INTEGER with l=0 (l meaning length).

3 Likes

It looks like neither Debian 7 ā€œWheezyā€ (with openssl 1.0.1e-2+deb7u20) nor Debian 8 ā€œJessieā€ (openssl 1.0.1k-3+deb8u4) have this problem. Both show:

8:d=2 hl=2 l= 1 prim: INTEGER :00

Good to know. Thank you.

1 Like

Out of interest how did you generate this CSR?

Our investigation showed that CSRs generated using the openssl command line tool were properly formatted but using a language binding (e.g. calling one of the OpenSSL libraries from Python or Ruby) caused the malformed CSRs when the version was not explicitly set.

1 Like

Hello,

sorry my english is not the best and well...so i need to know if this issue is releated to the openSSL Bug

thanks

It was generated by Neilpangā€™s acme.sh shell script.

To clarify, using openssl CLI from Debian Wheezy (specifically 1.0.1e-2+deb7u21, in my case) doesnā€™t produce the explicit x509 version:

openssl asn1parse -inform pem -in test.csr | grep INTEGER
8:d=2 hl=2 l= 1 prim: INTEGER :00

Versions of openssl >= 1.0.2 are available from Jessie Backports and onwards.

The client that I use, simp_le, was updated to take account of this bug in openssl, so that should work. However, acme.sh has an open bug on this issue.

At the top of this page the cutoff date is specified as November 2017 but the email I received entitled ā€œLetā€™s Encrypt Malformed CSR Advisoryā€ stated the date was November 17th, 2016.

@roland

I tested the above command in docker for many platforms, But it seems that none of them has such issue.

Please correct me if I misunderstood.

The test code:

openssl version
openssl genrsa 2048 2>/dev/null > mykey.key
openssl req -new -sha256 -key mykey.key -subj "/CN=my.com" | openssl asn1parse -inform pem | head -3 | tail -1 | grep "l=   1 prim:"

The test results:

[Sun Sep 18 07:27:18 CEST 2016] Running ubuntu:14.04, this may take a few minutes, please wait.
OpenSSL 1.0.1f 6 Jan 2014
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
ubuntu:14.04 [PASS]
[Sun Sep 18 07:27:19 CEST 2016] Running ubuntu:15.04, this may take a few minutes, please wait.
OpenSSL 1.0.1f 6 Jan 2014
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
ubuntu:15.04 [PASS]
[Sun Sep 18 07:27:20 CEST 2016] Running ubuntu:16.04, this may take a few minutes, please wait.
OpenSSL 1.0.2g-fips  1 Mar 2016
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
ubuntu:16.04 [PASS]
[Sun Sep 18 07:27:21 CEST 2016] Running ubuntu:latest, this may take a few minutes, please wait.
OpenSSL 1.0.2g-fips  1 Mar 2016
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
ubuntu:latest [PASS]
[Sun Sep 18 07:27:21 CEST 2016] Running debian:7, this may take a few minutes, please wait.
OpenSSL 1.0.1e 11 Feb 2013
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
debian:7 [PASS]
[Sun Sep 18 07:27:22 CEST 2016] Running debian:8, this may take a few minutes, please wait.
OpenSSL 1.0.1k 8 Jan 2015
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
debian:8 [PASS]
[Sun Sep 18 07:27:23 CEST 2016] Running debian:latest, this may take a few minutes, please wait.
OpenSSL 1.0.1k 8 Jan 2015
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
debian:latest [PASS]
[Sun Sep 18 07:27:23 CEST 2016] Running centos:5, this may take a few minutes, please wait.
OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
centos:5 [PASS]
[Sun Sep 18 07:27:24 CEST 2016] Running centos:6, this may take a few minutes, please wait.
OpenSSL 1.0.1e-fips 11 Feb 2013
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
centos:6 [PASS]
[Sun Sep 18 07:27:25 CEST 2016] Running centos:7, this may take a few minutes, please wait.
OpenSSL 1.0.1e-fips 11 Feb 2013
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
centos:7 [PASS]
[Sun Sep 18 07:27:26 CEST 2016] Running centos:latest, this may take a few minutes, please wait.
OpenSSL 1.0.1e-fips 11 Feb 2013
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
centos:latest [PASS]
[Sun Sep 18 07:27:26 CEST 2016] Running fedora:21, this may take a few minutes, please wait.
OpenSSL 1.0.1k-fips 8 Jan 2015
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
fedora:21 [PASS]
[Sun Sep 18 07:27:27 CEST 2016] Running fedora:22, this may take a few minutes, please wait.
OpenSSL 1.0.1k-fips 8 Jan 2015
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
fedora:22 [PASS]
[Sun Sep 18 07:27:28 CEST 2016] Running fedora:23, this may take a few minutes, please wait.
OpenSSL 1.0.2h-fips  3 May 2016
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
fedora:23 [PASS]
[Sun Sep 18 07:27:29 CEST 2016] Running fedora:latest, this may take a few minutes, please wait.
OpenSSL 1.0.2h-fips  3 May 2016
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
fedora:latest [PASS]
[Sun Sep 18 07:27:29 CEST 2016] Running opensuse:13.2, this may take a few minutes, please wait.
OpenSSL 1.0.1k-fips 8 Jan 2015
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
opensuse:13.2 [PASS]
[Sun Sep 18 07:27:30 CEST 2016] Running opensuse:42.1, this may take a few minutes, please wait.
OpenSSL 1.0.1i-fips 6 Aug 2014
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
opensuse:42.1 [PASS]
[Sun Sep 18 07:27:31 CEST 2016] Running opensuse:latest, this may take a few minutes, please wait.
OpenSSL 1.0.1i-fips 6 Aug 2014
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
opensuse:latest [PASS]
[Sun Sep 18 07:27:32 CEST 2016] Running alpine:3.1, this may take a few minutes, please wait.
OpenSSL 1.0.1t  3 May 2016
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
alpine:3.1 [PASS]
[Sun Sep 18 07:27:32 CEST 2016] Running alpine:3.2, this may take a few minutes, please wait.
OpenSSL 1.0.2h  3 May 2016
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
alpine:3.2 [PASS]
[Sun Sep 18 07:27:33 CEST 2016] Running alpine:3.3, this may take a few minutes, please wait.
OpenSSL 1.0.2h  3 May 2016
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
alpine:3.3 [PASS]
[Sun Sep 18 07:27:34 CEST 2016] Running alpine:latest, this may take a few minutes, please wait.
OpenSSL 1.0.2h  3 May 2016
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
alpine:latest [PASS]
[Sun Sep 18 07:27:34 CEST 2016] Running oraclelinux:6, this may take a few minutes, please wait.
OpenSSL 1.0.1e-fips 11 Feb 2013
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
oraclelinux:6 [PASS]
[Sun Sep 18 07:27:35 CEST 2016] Running oraclelinux:7, this may take a few minutes, please wait.
OpenSSL 1.0.1e-fips 11 Feb 2013
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
oraclelinux:7 [PASS]
[Sun Sep 18 07:27:36 CEST 2016] Running oraclelinux:latest, this may take a few minutes, please wait.
OpenSSL 1.0.1e-fips 11 Feb 2013
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
oraclelinux:latest [PASS]
[Sun Sep 18 07:27:37 CEST 2016] Running kalilinux/kali-linux-docker, this may take a few minutes, please wait.
OpenSSL 1.0.2h  3 May 2016 (Library: OpenSSL 1.0.2g  1 Mar 2016)
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
kalilinux/kali-linux-docker [PASS]
[Sun Sep 18 07:27:37 CEST 2016] Running base/archlinux, this may take a few minutes, please wait.
OpenSSL 1.0.2h  3 May 2016
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
base/archlinux [PASS]
[Sun Sep 18 07:27:38 CEST 2016] Running mageia, this may take a few minutes, please wait.
OpenSSL 1.0.2h  3 May 2016
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
mageia [PASS]


1 Like

@roland @Neilpang

According to Rolandā€™s description earlier in the thread all the tests you did using openssl v 1.0.1 failed as the ā€œl=ā€ length is null.

@mivk had mentioned that he wasnā€™t experiencing the error on his system and the implication was that acme.sh was correcting the bug. The open bug on acme.sh suggests that isnā€™t the case.

My understanding was that the bug was there when l= 0. All Neilpangā€™s tests and my own have l= 1, meaning it should be fine.

Besides, rolandā€™s original message suggests that this may only be a problem when using an API, not when calling the openssl binary directly from the shell. Or did I completely misunderstand?

Apologies @mivk @Neilpang , my mistake. I can even see that the copy I pasted from my own system shows ā€œl= 1ā€ but that was after I upgraded openssl. My LE failure happened when using a previous version of openssl.

Currently, all my certs are up to date but Iā€™ll switch over another domain and test.

My reading of what @roland wrote was that the LE clients are calling openssl using the API. The client I use, simp_le, has a wrapper to use the system openssl.

Has the staging CA servers been updated to reject this?
If we test successfully against staging can we be sure that everything will work in production?

@roland, I received the Malformed CSR Advisory email, but I tested my CSRs and they look correct:

> openssl asn1parse -inform pem -in 2013.albuquerque.wordcamp.org.crt
    0:d=0  hl=4 l=1279 cons: SEQUENCE          
    4:d=1  hl=4 l= 999 cons: SEQUENCE          
    8:d=2  hl=2 l=   3 cons: cont [ 0 ]        
   10:d=3  hl=2 l=   1 prim: INTEGER           :02
   [...]

(full output)

Are there any other causes that would have triggered that email? Are there other things I should test for?

Hi @iandunn,

It looks like you are parsing your certificate, rather than your CSR. The CSR or Certificate Signing Request, is generated by your client as part of the issuance process. Depending on your client it may or may not be saved to disk long-term. What client, and what version, are you using?

Doh, you're right, that was a dumb mistake, sorry for that. It's a custom client that a former teammate wrote, and this is my first time needing to make any changes. I'll try to dump the CSR to a file and parse the output.

Thanks :slight_smile:

Great. In case it helps, here is the one-line fix for Certbot: https://github.com/certbot/certbot/pull/2529/files. You can also fix the issue by upgrading OpenSSL.

1 Like