OpenSSL bug information

Issuance Policy
21

Implementing the fix has actually been the easy part for me, the hard part has been verifying it.

I’ve dumped the CSRs out to files before and after applying the fix, and I can see that the value of the version field is changing, but it looks like it’s always a proper int, regardless of whether or not I explicitly set the version.

> openssl asn1parse -inform pem -in /tmp/2013.albuquerque.wordcamp.org.csr.unversioned
    0:d=0  hl=4 l= 621 cons: SEQUENCE          
    4:d=1  hl=4 l= 341 cons: SEQUENCE          
    8:d=2  hl=2 l=   1 prim: INTEGER           :00
    [...]

> openssl asn1parse -inform pem -in /tmp/2013.albuquerque.wordcamp.org.csr.versioned
    0:d=0  hl=4 l= 621 cons: SEQUENCE          
    4:d=1  hl=4 l= 341 cons: SEQUENCE          
    8:d=2  hl=2 l=   1 prim: INTEGER           :02
    [...]

Maybe that’s just because OpenSSL.crypto.dump_certificate() is silently fixing it, which wouldn’t happen to the copy in memory that gets sent to LE, though.

I’m tempted to just deploy the change and hope for the best, since it seems like it should work, but I’m a bit leery of relying on that. The client does try to renew certs at 60 days, though, so worst case I’d have a month to fix it for existing sites, and could manually create certs for the handful of new sites that would need them in that period.

Anyway, thanks for your help @jsha. I’ll post back here if I make any more progress, just in case the info helps anyone else.

22

I would recommend testing against the staging server, https://acme-staging.api.letsencrypt.org/directory. The staging server already enforces the malformed CSR check, so if you are able to issue a certificate against staging, the fix worked.

1 Like
23

Huh, that’s what I have been testing against, and I’ve never gotten a failure, even before updating the code to explicitly set the version.

If I introduce an artificial bug into the client – like not signing the CSR, or not setting the subject CN – then I do get an error, so it seems like the client is detecting errors properly.

I thought maybe only some of the CSRs were malformed, so I checked them all with find output/certificates/*.csr |xargs -L1 openssl asn1parse -inform pem -in |grep INTEGER |grep "l= 0" and didn’t find anything. 1

So that leads me back to thinking maybe the email I got was a false-positive? I guess at this point it doesn’t really matter, since things are working against the staging server.

1: Note that Discourse isn't preserving the consequence spaces, but the actual command does grep for 3 of them, not 1. It finds the expected number of results when I grep for `l = 1`, so I think it's working as expected.
24

Try this: https://play.golang.org/p/39uXCEkc9X. If you hit “run,” you should get the error message: asn1: structure error: empty integer. Then, take the base64-encoder der form of your existing CSRs, and paste them in place of the base64 there. Do you get the same error?

Alternately, feel free to simply attach one of your old CSRs (from before the fix) here, and I’ll check it. I’m interested in tracking down false positives.

25

Huh, I must be doing something wrong. I got the expected error when running it against the default input, but when pasted in my string, I got illegal base64 data at input byte 144.

Here’s what I did:

  1. Dumped the CSR to a .der file with OpenSSL.crypto.dump_certificate_request( OpenSSL.crypto.FILETYPE_ASN1, csr ),
  2. Ran base64 output/certificates/2013.albuquerque.wordcamp.org.der in a terminal
  3. Stripped the newlines from the output and pasted that into the base64.RawURLEncoding.DecodeString() param

I’ve attached the .der file in case you can see what’s going on.

26

Can you provide a link on play.golang.org to the result?

27

Sure: https://play.golang.org/p/hgWTxS3o5T

28

I see the problem: the link I gave you used RawURLEncoding, but you actually want StdEncoding. I fixed it, and now this code properly parses your CSR with no errors: https://play.golang.org/p/5NI2UmHozJ.

So that suggests that the CSR you provided doesn’t have this particular problem. Is that the only CSR you submitted in the last ~4 months? @roland, can you take a look and see if this CSR would have triggered the code you used to look for empty version numbers?

29

The attached CSR doesn’t exhibit the issue and wouldn’t trigger the test as the bytes in offsets 8 and 9 have values 02 and 01 which are valid.

30

Huh, that’s odd. wordcamp.org has about ~650 sites using LE, and they’re set to renew every 60 days, but I don’t have any of the past CSRs.

31

@iandunn: Are they all under the wordcamp.org domain? Have you issued any other LE certificates you could be getting email about? If you PM your email address to @roland, he can cross-check the data and scripts he used to send the warning email and figure out which certificate requests triggered it.

32

Almost all are subdomains of wordcamp.org, mostly in the format year.city.wordcamp.org and city.wordcamp.org (e.g., 2016.seattle.wordcamp.org).

The exceptions:

As far as I know, there shouldn’t be any other certs or activity tied to the email address used during the request. I PM’d the address to Roland.

33

Our data shows two malformed CSRs that were both submitted on the 10th of July this year by the registration ID associated with your email, for the names 2016.fayetteville.wordcamp.org and 2016.manchester.wordcamp.org.

Given you are now producing valid CSRs this was most likely either a software bug that was found and fixed (on purpose or inadvertently), you changed the client you were using at some point, or that you updated the version of OpenSSL (I didn’t test specially patched versions provided by distros like the version you seem to be using from Debian) or the Python bindings since those two malformed CSRs were originally generated.

34

Further searching indicate there were also CSRs exhibiting this issue submitted on both the 9th and 11th of July which is about as far back as we looked at the time I assembled this data set.

35

Ah, ok, thanks for checking! I don’t think we made any changes to the client, so I’m guessing it was a backported OpenSSL update.

36

This change was enabled in production today. If you are getting the error below, follow the instructions from the original post in this thread.

CSR generated using a pre-1.0.2 OpenSSL with a client that doesn't properly specify the CSR version. See https://community.letsencrypt.org/t/openssl-bug-information/19591

37

I’ve added a small edit to the end of the “For client developers” section mentioning the tls-sni-01 challenge certificate must have a serial number set. Like the X509 version number this was identified as a compatibilty breaking change for old OpenSSL users and Go 1.6+. (Thanks to the Github user “phillipp” :clap:)

38

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.