HPKP best practices if you choose to implement

#1

HPKP pinning carries an inherent risk of bricking your site and requires great care. For that reason, it is not presently supported by the Let’s Encrypt client, and will not be until we have a lot of time to perform quality control on an implementation, if ever.

But @intchloe asked a question on GitHub about how to roll their own HPKP pins:

Will Let’s Encrypt always use ‘Let’s Encrypt Authority X1’ in its cert chain? So the hash “YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg=” can be used on all websites issued by Let’s Encrypt?

I’m going to try to answer this here. But first, some general points about HPKP:

  • First, don’t use HPKP unless you have read and understood the specifications, and are extremely careful. Know that if you get something wrong, you can render your domain unusable.
  • Never pin exclusively to Let’s Encrypt (or any single CA). Make sure you have at a couple of backups / alternatives that you can work with if something catastrophic or simply buggy happens to your first choice
  • If the domain is of any value to you, test a deployment on a less valuable domain, and test your backup CA with it.
  • Use the -Report-Only header and the report-uri field before you actually deploy HPKP, to see how many visitors report breakage.
  • Pinning works on public keys, not certificates, and a pinned key can occur anywhere in a cert chain: root, intermediate, or leaf.

Now, to attempt to answer the question. Since this advice has not been tested, only try this on domains you don’t care about, and are willing to see permanently broken.

  • First, generate a backup keypair, and include the SPKI hash of the public key in your HPKP header. Keep this keypair somewhere safe. You will use it in an emergency to request a leaf certificate from some other CA in case you are unable to issue from any of your pinned CAs for some reason.
  • Let’s Encrypt will not always use the current Let’s Encrypt Authority X1 cross-signed intermediate cert, so you shouldn’t pin exclusively to it. We currently have a Let’s Encrypt Authority X2 intermediate CA as well, and will have other intermediate certs in the future.
  • Always include the public keys hashes from the ISRG Root X1 and DST Root X3 certs in your pin set.
  • Somewhat surprisingly, clients validate the trust path they build, which doesn’t necessarily have anything to do with the intermediate certificates your server provides in the handshake. This trust path varies based on what intermediates the client has cached, and what cross-signatures the intermediates and/or roots have. Because of this it’s possible for pinning to break even without any changes on your server. To avoid that, it’s highly advisable to pin one of the two public key hashes that is guaranteed to be in the built chain: the one from your leaf cert (cert.pem) or the one from the intermediate that issued it (chain.pem). Make sure to update this pin with each renewal, since intermediates can change without notice.
  • Remember to pin to at least two non-Let’s Encrypt CAs!

The above advice has not been tested and may well contain errors. Perhaps those who are willing to test them on sacrificial domain names can report back and share knowledge in this thread.

[Edit: fixed some inaccuracies]
[Edit: changed title]

HTTP Public Key Pinning (HPKP)
Support for HPKP - HTTP Public Key Pinning?
Let's Encrypt shouldn't rely on a centralized CA; it should be decentralized
Pinning and SHA1 migration
[Help needed] Windows XP support
Certbot reusing private key
Official HPKP support from Let's Encrypt
Official HPKP support from Let's Encrypt
Official HPKP support from Let's Encrypt
New rate limit question
Pros and cons of 90-day certificate lifetimes
Is it possible to renew a certificate using the X1 intermediate certificate?
iOS Public Key Pinning Question
#2

@pde I think to say it is inherently dangerous because the idea and the technical background is not dangerous, only if not correctly been used.
And personal i do not like to pin an CA. My personal hint is to create two backup spare keys.
One for RSA one for EC and pin these two backup keys currently not used and safe them offline.
(Maybe even print out as QR code for recovery).
Than you can change to any CA and nobody can push an unauthorized Cert to the user, not even an Good-CA acting on an NSL (National Security Letter).

#3

same here that’s what i do as well http://centminmod.com/http-public-key-pinning.html but i am only testing with short max-age values for now out of the box at 7 days

Didn’t really want to pin against other CAs, but I guess there’s use case if you switch CA SSL certs. But currently, all my in use production Wildcard SSL certs are with 3yr expiries so wasn’t any issue unlike 90 day expiries :slight_smile:

#4
#5

I have a few questions:

Just to be clear on exactly what the recommended set of keys to pin on is:
a) All of the CA’s root keys
b) All of the CA’s intermediate keys
c) Same as “a” and “b” for a secondary trusted CA
d) Same as “a” and “b” for a third trusted CA
e) A pair of backup keys which are kept secret and ideally offline - One RSA and one EC
f) Each leaf key that has been issued for as long as they are valid

Is this correct ? It seems a little excessive - Why are the second and third CAs needed if you have your own backup leaf keys ?

It looks like “b” and “f” are suggested as either / or, where “b” could be used if you don’t know what the leaf keys are going to be (For example, in a load balanced environment where each server generates it’s own key and doesn’t know about the other servers / keys) and “f” is a better choice if the leaf keys are managed separately ?

Do you have (Or plan to have) a public feed which can be used to subscribe to root or intermediate key changes ?

What is the advice on the pinning expiry ? The specification suggests to start at 60 days, but this looks to have been written when year-long certificates were the norm - What is the suggested expiry based on the 90 day expiry of LE’s leaf certificates ? Should the duration be set to the duration of the shortest certificate in-use ? I.e. If the leaf key is expected to be refreshed in 5 days, should 5 days be used for the pin ?

#6

Not Correct ideally you pin exactly 3 or more Public Keys:

  1. The currently used public key of your certificate (more iff you have more than one certificate in use)
  2. pair of backup keys which are kept secret and ideally offline - One RSA and one EC (Also here can be more if you have different EC-Curves as backup)
    This guarantees that ONLY YOU can create accepted certificates.
    Even an NSL (National Security Letter) that force an CA to sign an malicious cert is not accepted since the PIN limits to your keys and not the ca keys.
    With these setup i would say you can pin >3 months.
#7

a) is not needed if you already do b) - it may even be disadvantageous.

Also not needed, because you already pin the intermediates.

Indeed a nice idea to also backup an EC one.

Because you maybe also loose access to your backup keys…
(or just create more than one backup leaf key and store each of them - securely - at a different place.

Yes and if you pin the intermediates you can follow these “long term” suggestions as the intermediates won’t change in the nearer future.

If you only pin your current leaf cert then yes - theoretically.
However if you really only want to pin leaf certs you may better either only renew them (so you can use the cert pin longer than 90days) and(/or) already pre-generate (but obviously not sign them by LE) the certs you intent to use afterwards, so that you can switch them (and the according HPKP header) one by one without having problems.

Related:

(bottom line :wink: )

But generally the best suggestion IMHO is: Read about the implications of each way you can do, weight the risks up and decide by yourself what best suites you/your webserver/your visitors. This may depend on many factors.
A good article as a first read about HPKP was written by @ScottHelme:
Guidance on setting up HPKP

#8

And your users will be faced with a big large “This certificate was signed by an untrusted issuer” error :question:

#9

Thank you all. I always knew pinning was a minefiled, but the more I learn about it the more it seems like a minefield inside of another minefield !

@Osiris: This page shows how the HPKP validation failure errors look in various browsers:

Unless you’re asking asking how the the backup keys are trusted by browsers ? When you need to use them, you create a certificate signing request using your backup key and get that signed by a commercial CA such as Let’s Encrypt - The keys are generated and controlled by you, but the chain of trust still exists.

#10

Ah, thanks, that’s what he ment… I was thinking about keeping keys and what the heck should you do with them in the event of trouble? Play your own CA? That’s not gonna work now is it? But nooo, make a CSR and send it to another trusted CA, d’oh… :stuck_out_tongue: It’s late, time for sleep :wink:

#11

I have a new blog coming on how I now use Let’s Encrypt and HPKP. I’m not using the LE client due to the problems with key rotation but there is a simple solution. I will update once it’s published.

As @rugk says, you need to make sure you fully understand HPKP before deploying it, much like any new feature I guess. Always deploy in report-only mode and I’d recommend implementing reporting (https://report-uri.io) and to test your policy thoroughly (https://report-uri.io/home/tools).

*Full disclosure, the linked site is my service.

#12

I read elsewhere on the forum that clients don’t trace the chain of signatures beyond the first trusted certificate. So if the intermediates are trusted by the client, it won’t trace the chain to the root and therefore, the HPKP verification will fail if the intermediates aren’t defined as valid.

#13

From RFC 7469, section 2.6

To perform Pin Validation, the UA will compute the SPKI Fingerprints for each certificate in the Pinned Host’s validated certificate chain, using each supported hash algorithm for each certificate. (As described in Section 2.4, certificates whose SPKI cannot be taken in isolation cannot be pinned.) The UA MUST ignore superfluous certificates in the chain that do not form part of the validating chain. The UA will then check that the set of these SPKI Fingerprints intersects the set of SPKI Fingerprints in that Pinned Host’s Pinning Metadata. If there is set intersection, the UA continues with the connection as normal. Otherwise, the UA MUST treat this Pin Validation failure as a non-recoverable error. Any procedure that matches the results of this Pin Validation procedure is considered equivalent.

So the UA should check the whole chain, not just the first trusted certificate. Just one valid Pin is enough for the validation to succeed.

#14

And this is the practice for all/most clients that support HPKP? (I hope it is)

#15

Well, the source code for Chromium does state a lot of HashesIntersect calls :stuck_out_tongue: over a variable hashes… Not quite sure where that variable comes from though… Not yet anyway.

Ah, found something:

// PublicKeyPins contains a number of SubjectPublicKeyInfo hashes for a site.
// The validated certificate chain for the site must not include any of
// |excluded_hashes| and must include one or more of |required_hashes|.
struct PublicKeyPins {
  const char* const* required_hashes;
  const char* const* excluded_hashes;
};

Looks like they also check the whole validated chain indeed. According to the comment anyway :wink:

#16

@Osiris yes, the UA will check the whole chain against pinned keys, you can pin either at the leaf, intermediate or root level. Sites like Github currently pin at the root level ( https://report-uri.io/home/pkp_analyse/https%3A%2F%2Fgithub.com ) so that only those 2 certificate authorities can issue for their domain.

The browser will check every SPKI hash in the served chain against every cached pin, you only need 1 match for the connection to succeed.

I’ve written a few articles on HPKP if you’d like some further reading: https://scotthelme.co.uk/tag/hpkp/

#17

Here’s the article I mentioned: https://scotthel.me/LE

#18

Thanks for the write up Scott, I have stayed away from HPKP but after reading that I will take a non important domain of mine and play with HPKP for a while and see how it goes before I dare to venture into using it on anything important… if ever.

Thanks also for your super awesome https://securityheaders.io/ one of the most invaluable resources which I use uber frequently.

I was randomly scanning some “supposedly” super secure sites today just for fun, or shall I say, sites that should be secure and they all scored a miserable F, tested all of them on Qualsys SSL Labs too and found so many that failed there too … bloody scary … some real idiots out there who get paid fat salaries and do not seem to know diddly about security and these are banks, government sites and online shopping sites I was testing.

#19

My guess is that those “F” sites still do all to allow customers running Windowx XP and to do so still accept RC4 (aka. clear text)

#20

accepting XP does not correlate to RC4, XP runs fine with 3DES which is iirc still more or less secure.