OpenSSL bug information

Neilpang · September 18, 2016, 5:25am

I tested the above command in docker for many platforms, But it seems that none of them has such issue.

Please correct me if I misunderstood.

The test code:

openssl version
openssl genrsa 2048 2>/dev/null > mykey.key
openssl req -new -sha256 -key mykey.key -subj "/CN=my.com" | openssl asn1parse -inform pem | head -3 | tail -1 | grep "l=   1 prim:"

The test results:

[Sun Sep 18 07:27:18 CEST 2016] Running ubuntu:14.04, this may take a few minutes, please wait.
OpenSSL 1.0.1f 6 Jan 2014
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
ubuntu:14.04 [PASS]
[Sun Sep 18 07:27:19 CEST 2016] Running ubuntu:15.04, this may take a few minutes, please wait.
OpenSSL 1.0.1f 6 Jan 2014
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
ubuntu:15.04 [PASS]
[Sun Sep 18 07:27:20 CEST 2016] Running ubuntu:16.04, this may take a few minutes, please wait.
OpenSSL 1.0.2g-fips  1 Mar 2016
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
ubuntu:16.04 [PASS]
[Sun Sep 18 07:27:21 CEST 2016] Running ubuntu:latest, this may take a few minutes, please wait.
OpenSSL 1.0.2g-fips  1 Mar 2016
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
ubuntu:latest [PASS]
[Sun Sep 18 07:27:21 CEST 2016] Running debian:7, this may take a few minutes, please wait.
OpenSSL 1.0.1e 11 Feb 2013
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
debian:7 [PASS]
[Sun Sep 18 07:27:22 CEST 2016] Running debian:8, this may take a few minutes, please wait.
OpenSSL 1.0.1k 8 Jan 2015
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
debian:8 [PASS]
[Sun Sep 18 07:27:23 CEST 2016] Running debian:latest, this may take a few minutes, please wait.
OpenSSL 1.0.1k 8 Jan 2015
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
debian:latest [PASS]
[Sun Sep 18 07:27:23 CEST 2016] Running centos:5, this may take a few minutes, please wait.
OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
centos:5 [PASS]
[Sun Sep 18 07:27:24 CEST 2016] Running centos:6, this may take a few minutes, please wait.
OpenSSL 1.0.1e-fips 11 Feb 2013
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
centos:6 [PASS]
[Sun Sep 18 07:27:25 CEST 2016] Running centos:7, this may take a few minutes, please wait.
OpenSSL 1.0.1e-fips 11 Feb 2013
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
centos:7 [PASS]
[Sun Sep 18 07:27:26 CEST 2016] Running centos:latest, this may take a few minutes, please wait.
OpenSSL 1.0.1e-fips 11 Feb 2013
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
centos:latest [PASS]
[Sun Sep 18 07:27:26 CEST 2016] Running fedora:21, this may take a few minutes, please wait.
OpenSSL 1.0.1k-fips 8 Jan 2015
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
fedora:21 [PASS]
[Sun Sep 18 07:27:27 CEST 2016] Running fedora:22, this may take a few minutes, please wait.
OpenSSL 1.0.1k-fips 8 Jan 2015
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
fedora:22 [PASS]
[Sun Sep 18 07:27:28 CEST 2016] Running fedora:23, this may take a few minutes, please wait.
OpenSSL 1.0.2h-fips  3 May 2016
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
fedora:23 [PASS]
[Sun Sep 18 07:27:29 CEST 2016] Running fedora:latest, this may take a few minutes, please wait.
OpenSSL 1.0.2h-fips  3 May 2016
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
fedora:latest [PASS]
[Sun Sep 18 07:27:29 CEST 2016] Running opensuse:13.2, this may take a few minutes, please wait.
OpenSSL 1.0.1k-fips 8 Jan 2015
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
opensuse:13.2 [PASS]
[Sun Sep 18 07:27:30 CEST 2016] Running opensuse:42.1, this may take a few minutes, please wait.
OpenSSL 1.0.1i-fips 6 Aug 2014
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
opensuse:42.1 [PASS]
[Sun Sep 18 07:27:31 CEST 2016] Running opensuse:latest, this may take a few minutes, please wait.
OpenSSL 1.0.1i-fips 6 Aug 2014
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
opensuse:latest [PASS]
[Sun Sep 18 07:27:32 CEST 2016] Running alpine:3.1, this may take a few minutes, please wait.
OpenSSL 1.0.1t  3 May 2016
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
alpine:3.1 [PASS]
[Sun Sep 18 07:27:32 CEST 2016] Running alpine:3.2, this may take a few minutes, please wait.
OpenSSL 1.0.2h  3 May 2016
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
alpine:3.2 [PASS]
[Sun Sep 18 07:27:33 CEST 2016] Running alpine:3.3, this may take a few minutes, please wait.
OpenSSL 1.0.2h  3 May 2016
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
alpine:3.3 [PASS]
[Sun Sep 18 07:27:34 CEST 2016] Running alpine:latest, this may take a few minutes, please wait.
OpenSSL 1.0.2h  3 May 2016
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
alpine:latest [PASS]
[Sun Sep 18 07:27:34 CEST 2016] Running oraclelinux:6, this may take a few minutes, please wait.
OpenSSL 1.0.1e-fips 11 Feb 2013
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
oraclelinux:6 [PASS]
[Sun Sep 18 07:27:35 CEST 2016] Running oraclelinux:7, this may take a few minutes, please wait.
OpenSSL 1.0.1e-fips 11 Feb 2013
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
oraclelinux:7 [PASS]
[Sun Sep 18 07:27:36 CEST 2016] Running oraclelinux:latest, this may take a few minutes, please wait.
OpenSSL 1.0.1e-fips 11 Feb 2013
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
oraclelinux:latest [PASS]
[Sun Sep 18 07:27:37 CEST 2016] Running kalilinux/kali-linux-docker, this may take a few minutes, please wait.
OpenSSL 1.0.2h  3 May 2016 (Library: OpenSSL 1.0.2g  1 Mar 2016)
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
kalilinux/kali-linux-docker [PASS]
[Sun Sep 18 07:27:37 CEST 2016] Running base/archlinux, this may take a few minutes, please wait.
OpenSSL 1.0.2h  3 May 2016
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
base/archlinux [PASS]
[Sun Sep 18 07:27:38 CEST 2016] Running mageia, this may take a few minutes, please wait.
OpenSSL 1.0.2h  3 May 2016
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
mageia [PASS]

iainm · September 18, 2016, 9:07am

@roland @Neilpang

According to Roland’s description earlier in the thread all the tests you did using openssl v 1.0.1 failed as the “l=” length is null.

@mivk had mentioned that he wasn’t experiencing the error on his system and the implication was that acme.sh was correcting the bug. The open bug on acme.sh suggests that isn’t the case.

mivk · September 18, 2016, 11:04am

My understanding was that the bug was there when l= 0. All Neilpang’s tests and my own have l= 1, meaning it should be fine.

Besides, roland’s original message suggests that this may only be a problem when using an API, not when calling the openssl binary directly from the shell. Or did I completely misunderstand?

iainm · September 18, 2016, 12:24pm

Apologies @mivk @Neilpang , my mistake. I can even see that the copy I pasted from my own system shows “l= 1” but that was after I upgraded openssl. My LE failure happened when using a previous version of openssl.

Currently, all my certs are up to date but I’ll switch over another domain and test.

My reading of what @roland wrote was that the LE clients are calling openssl using the API. The client I use, simp_le, has a wrapper to use the system openssl.

arf · September 18, 2016, 8:10pm

Has the staging CA servers been updated to reject this?
If we test successfully against staging can we be sure that everything will work in production?

iandunn · October 7, 2016, 8:40pm

@roland, I received the Malformed CSR Advisory email, but I tested my CSRs and they look correct:

> openssl asn1parse -inform pem -in 2013.albuquerque.wordcamp.org.crt
    0:d=0  hl=4 l=1279 cons: SEQUENCE          
    4:d=1  hl=4 l= 999 cons: SEQUENCE          
    8:d=2  hl=2 l=   3 cons: cont [ 0 ]        
   10:d=3  hl=2 l=   1 prim: INTEGER           :02
   [...]

(full output)

Are there any other causes that would have triggered that email? Are there other things I should test for?

jsha · October 12, 2016, 6:33pm

Hi @iandunn,

It looks like you are parsing your certificate, rather than your CSR. The CSR or Certificate Signing Request, is generated by your client as part of the issuance process. Depending on your client it may or may not be saved to disk long-term. What client, and what version, are you using?

iandunn · October 12, 2016, 8:24pm

Doh, you’re right, that was a dumb mistake, sorry for that. It’s a custom client that a former teammate wrote, and this is my first time needing to make any changes. I’ll try to dump the CSR to a file and parse the output.

Thanks

jsha · October 12, 2016, 9:19pm

Great. In case it helps, here is the one-line fix for Certbot: https://github.com/certbot/certbot/pull/2529/files. You can also fix the issue by upgrading OpenSSL.

iandunn · October 13, 2016, 10:22pm

Implementing the fix has actually been the easy part for me, the hard part has been verifying it.

I’ve dumped the CSRs out to files before and after applying the fix, and I can see that the value of the version field is changing, but it looks like it’s always a proper int, regardless of whether or not I explicitly set the version.

> openssl asn1parse -inform pem -in /tmp/2013.albuquerque.wordcamp.org.csr.unversioned
    0:d=0  hl=4 l= 621 cons: SEQUENCE          
    4:d=1  hl=4 l= 341 cons: SEQUENCE          
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
    [...]

> openssl asn1parse -inform pem -in /tmp/2013.albuquerque.wordcamp.org.csr.versioned
    0:d=0  hl=4 l= 621 cons: SEQUENCE          
    4:d=1  hl=4 l= 341 cons: SEQUENCE          
    8:d=2  hl=2 l=   1 prim: INTEGER           :02
    [...]

Maybe that’s just because OpenSSL.crypto.dump_certificate() is silently fixing it, which wouldn’t happen to the copy in memory that gets sent to LE, though.

I’m tempted to just deploy the change and hope for the best, since it seems like it should work, but I’m a bit leery of relying on that. The client does try to renew certs at 60 days, though, so worst case I’d have a month to fix it for existing sites, and could manually create certs for the handful of new sites that would need them in that period.

Anyway, thanks for your help @jsha. I’ll post back here if I make any more progress, just in case the info helps anyone else.

jsha · October 13, 2016, 10:24pm

I would recommend testing against the staging server, https://acme-staging.api.letsencrypt.org/directory. The staging server already enforces the malformed CSR check, so if you are able to issue a certificate against staging, the fix worked.

iandunn · October 14, 2016, 9:39pm

Huh, that’s what I have been testing against, and I’ve never gotten a failure, even before updating the code to explicitly set the version.

If I introduce an artificial bug into the client – like not signing the CSR, or not setting the subject CN – then I do get an error, so it seems like the client is detecting errors properly.

I thought maybe only some of the CSRs were malformed, so I checked them all with find output/certificates/*.csr |xargs -L1 openssl asn1parse -inform pem -in |grep INTEGER |grep "l= 0" and didn’t find anything. ¹

So that leads me back to thinking maybe the email I got was a false-positive? I guess at this point it doesn’t really matter, since things are working against the staging server.

¹: Note that Discourse isn't preserving the consequence spaces, but the actual command does grep for 3 of them, not 1. It finds the expected number of results when I grep for `l = 1`, so I think it's working as expected.

jsha · October 14, 2016, 11:00pm

Try this: https://play.golang.org/p/39uXCEkc9X. If you hit “run,” you should get the error message: asn1: structure error: empty integer. Then, take the base64-encoder der form of your existing CSRs, and paste them in place of the base64 there. Do you get the same error?

Alternately, feel free to simply attach one of your old CSRs (from before the fix) here, and I’ll check it. I’m interested in tracking down false positives.

iandunn · October 17, 2016, 8:40pm

Huh, I must be doing something wrong. I got the expected error when running it against the default input, but when pasted in my string, I got illegal base64 data at input byte 144.

Here’s what I did:

Dumped the CSR to a .der file with OpenSSL.crypto.dump_certificate_request( OpenSSL.crypto.FILETYPE_ASN1, csr ),
Ran base64 output/certificates/2013.albuquerque.wordcamp.org.der in a terminal
Stripped the newlines from the output and pasted that into the base64.RawURLEncoding.DecodeString() param

I’ve attached the .der file in case you can see what’s going on.

jsha · October 17, 2016, 9:02pm

Can you provide a link on play.golang.org to the result?

iandunn · October 17, 2016, 9:49pm

Sure: https://play.golang.org/p/hgWTxS3o5T

jsha · October 17, 2016, 10:18pm

I see the problem: the link I gave you used RawURLEncoding, but you actually want StdEncoding. I fixed it, and now this code properly parses your CSR with no errors: https://play.golang.org/p/5NI2UmHozJ.

So that suggests that the CSR you provided doesn’t have this particular problem. Is that the only CSR you submitted in the last ~4 months? @roland, can you take a look and see if this CSR would have triggered the code you used to look for empty version numbers?

roland · October 17, 2016, 10:34pm

The attached CSR doesn’t exhibit the issue and wouldn’t trigger the test as the bytes in offsets 8 and 9 have values 02 and 01 which are valid.

iandunn · October 17, 2016, 11:36pm

Huh, that’s odd. wordcamp.org has about ~650 sites using LE, and they’re set to renew every 60 days, but I don’t have any of the past CSRs.

jsha · October 18, 2016, 5:39pm

@iandunn: Are they all under the wordcamp.org domain? Have you issued any other LE certificates you could be getting email about? If you PM your email address to @roland, he can cross-check the data and scripts he used to send the warning email and figure out which certificate requests triggered it.