New staging certs

new staging certs are failing because I cannot find the new staging certs to download anywhere?

They are not here

or here

Looking for this cert
CA /C=US/O=(STAGING) Let's Encrypt/CN=(STAGING) Counterfeit Cashew R10 (a368102c)

Yeah, they haven't updated the staging environment documentation yet. (I made a Github ticket for it a few days ago, but they've probably been too busy getting their new prod intermediates online.)

You can find one way to get the staging certs from this post:

So "Counterfeit Cashew R10" should be available from http://stg-r10.i.lencr.org/

But when you say "are failing", usually systems shouldn't care about the intermediates, since the server gets them from the ACME server when getting a cert and then sends them as part of the chain to the clients. If you're more specific about what you're doing, people here may be able to give you a better way. The intermediate you get can change with each certificate one gets.

6 Likes

Sorry for the inconvenience. You're right it's been very busy getting everything going. I'll get the staging documentation updated soon. We have a batch of doc updates going out shortly now that the launch has happened.

(But as others have said, you shouldn't be in a situation where you're relying on our documentation that leads to failures)

6 Likes

I'm curious what would be failing and why exactly?

3 Likes

Any idea when the new staging documentation is going to be available? My pipeline is down and we can't produce a new release of our product.

I tried both http://stg-r10.i.lencr.org/ and http://stg-r11.i.lencr.org/ to the get the DER files and added to my system the same way I did the old staging certs. Still getting (STAGING) Counterfeit Cashew R10 or (STAGING) Wannabe Watercress R11 not found when attempting to generate the certificate.

Why would there be a need to include intermediate certificates to your system?

5 Likes

Especially intermediate staging certificates? Usually a server sends the intermediates to the client, and the client uses them to check against a root trust store. If you're building a custom root trust store that has the staging root certificates, I could see doing that but it's pretty unusual. Needing to include intermediates directly somewhere in that process sounds really broken. If you can explain exactly what your "pipeline" is trying to do, we can probably help you figure out the right way of doing it because intermediates can and will regularly change.

6 Likes

As others have already said, the ACME server gives you the intermediates, you don't need any external documentation to get them. Just for reference, here are all current staging intermediate certs in PEM format (not including cross signs, only default chains):

s:C = US, O = (STAGING) Let's Encrypt, CN = (STAGING) Counterfeit Cashew R10
i:C = US, O = (STAGING) Internet Security Research Group, CN = (STAGING) Pretend Pear X1
-----BEGIN CERTIFICATE-----
MIIFTTCCAzWgAwIBAgIRAIEbJZNMRe313pPFbPDf1AQwDQYJKoZIhvcNAQELBQAw
ZjELMAkGA1UEBhMCVVMxMzAxBgNVBAoTKihTVEFHSU5HKSBJbnRlcm5ldCBTZWN1
cml0eSBSZXNlYXJjaCBHcm91cDEiMCAGA1UEAxMZKFNUQUdJTkcpIFByZXRlbmQg
UGVhciBYMTAeFw0yNDAzMTMwMDAwMDBaFw0yNzAzMTIyMzU5NTlaMFoxCzAJBgNV
BAYTAlVTMSAwHgYDVQQKExcoU1RBR0lORykgTGV0J3MgRW5jcnlwdDEpMCcGA1UE
AxMgKFNUQUdJTkcpIENvdW50ZXJmZWl0IENhc2hldyBSMTAwggEiMA0GCSqGSIb3
DQEBAQUAA4IBDwAwggEKAoIBAQCa8zRfthw5T4/n9kt2iuNxb4zt78rR3ZygeZvp
GGx8nMdU7jqCLeOqkuqhKBvBV823fvA0bYg3JaCiFsyB6Idry8eVZLFJp3BtBZFX
pbAb7+QCYLAZrUl5rX3G9VoTG/x4Q8a9pN57CI34bmJlKaaefSLoeeAOArk8fcV3
/MkTQHWG+heh9ex0ogr3kDQQOSm+dI14hz75eHuiV26kory+tDDBN1re76Qf8RRt
NxnngYZFrbJ9IOovFJa28weGOVXekOL4JJ7/VdFMMceXQze8M+0qnkaaj+HNRYQx
z0N1geD3clT/xrme3gL0y5xAKjX6eq4dXdJuau0h90VFTc3DAgMBAAGjggEAMIH9
MA4GA1UdDwEB/wQEAwIBhjAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUHAwEw
EgYDVR0TAQH/BAgwBgEB/wIBADAdBgNVHQ4EFgQUpFJG6lioj2jYt7GQ0UpCSo9r
KHEwHwYDVR0jBBgwFoAUtfNl8v6wCpIf+zx980SgrGMlwxQwNgYIKwYBBQUHAQEE
KjAoMCYGCCsGAQUFBzAChhpodHRwOi8vc3RnLXgxLmkubGVuY3Iub3JnLzATBgNV
HSAEDDAKMAgGBmeBDAECATArBgNVHR8EJDAiMCCgHqAchhpodHRwOi8vc3RnLXgx
LmMubGVuY3Iub3JnLzANBgkqhkiG9w0BAQsFAAOCAgEAHhpGt8p6QJk8fnM0zOS5
AEUultcQP/20NSCdBpIxAfownd+ylUzH2s1prl8T9rhKzTm4xZ+NBqPEDAd/1p5e
1PYqR8bNnKJcTAAkjBYvKo5br0ng6GUkCQTqJ3atbAvx2bghdfrULF6+u+f4o288
Jo6o1kN2Jf1mTXTKi2GRJ5JerChn0dls28Mzx8QS3mWfFFSu+QuGwtxLty7ySzWE
TH+lNMa8U6MbtSvoWf8OYzNUZ7Bih6JKnuo4ueG7zRir2go8ygNdZoWHhmk84c35
ABH8dKX+fKNDK6xzeygtbguJN2/D1n72RQK8w7gXjT7ptphEoxmxe+ZSQ1XEoKUv
I/Xy2D8F/5+b4Uxg8xSNev/N+E9F/env3Pp7zQJt/L4dhnUMoqqbAOifOE8l/CdN
QHKv1OJkNYiaJ+xfx+Z9wUw4zRvjuD23EK42GjCfsK0JNuHb98LAmDBF+Xfq+PlH
OGZTLlmnM72Jmjvi5IJ/uo5pM2QbmauUMle4tZwq11ipRs0BggZY0fHFK/7AxXj/
j3PQZAbnLSo/9x/YUK6aR1NGZYBScfwFCfbEMNuXB1KJJvn/eQP7TbVSrqsrsHpk
AAzAHvDkhUpctdN8Bfjjylj4p9D0zGpqz7RMd4EKnC+K6waARt3+cY05nzq7uF34
4eamtcAi8f6h3h9vQlcWOAg=
-----END CERTIFICATE-----

s:C = US, O = (STAGING) Let's Encrypt, CN = (STAGING) Wannabe Watercress R11
i:C = US, O = (STAGING) Internet Security Research Group, CN = (STAGING) Pretend Pear X1
-----BEGIN CERTIFICATE-----
MIIFTTCCAzWgAwIBAgIRAOOuDiVgQFyATegPOxfOa5IwDQYJKoZIhvcNAQELBQAw
ZjELMAkGA1UEBhMCVVMxMzAxBgNVBAoTKihTVEFHSU5HKSBJbnRlcm5ldCBTZWN1
cml0eSBSZXNlYXJjaCBHcm91cDEiMCAGA1UEAxMZKFNUQUdJTkcpIFByZXRlbmQg
UGVhciBYMTAeFw0yNDAzMTMwMDAwMDBaFw0yNzAzMTIyMzU5NTlaMFoxCzAJBgNV
BAYTAlVTMSAwHgYDVQQKExcoU1RBR0lORykgTGV0J3MgRW5jcnlwdDEpMCcGA1UE
AxMgKFNUQUdJTkcpIFdhbm5hYmUgV2F0ZXJjcmVzcyBSMTEwggEiMA0GCSqGSIb3
DQEBAQUAA4IBDwAwggEKAoIBAQCVYHwe2gxMv9Jh/d1BgocW3am5E/iBOrLkyENg
RqLmVl6vNRXgoKpPjtP40Mv17Z2LlYhRsneaK9gdSNmNTxJlY8viqs9dk4XwKJxD
Pfl/vJBW4M/nJzFwS30qu7VF8jSfyF82Q7KkLl5NxJ39Y5yvQFzHjL3YCO8coMG5
v72DDgWzc+9wk2ncoXiB3/3mVg4RtD1w1xPpdjtSh5keB7jqgnD2/I75OLBzamy6
NL9nguaEc8cX8CdnpoCCQI8/Fz9qhPtJjP+O/RHCugWVkSptuCRra8tcnJTGHRiN
r/GyYsHZZ4jZ76hbJjxapwf24E8RAVzuMRsw/nS6/hUJ8lqPAgMBAAGjggEAMIH9
MA4GA1UdDwEB/wQEAwIBhjAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUHAwEw
EgYDVR0TAQH/BAgwBgEB/wIBADAdBgNVHQ4EFgQUE8vX9q6d/GlCZNZcfCPMhflH
x7YwHwYDVR0jBBgwFoAUtfNl8v6wCpIf+zx980SgrGMlwxQwNgYIKwYBBQUHAQEE
KjAoMCYGCCsGAQUFBzAChhpodHRwOi8vc3RnLXgxLmkubGVuY3Iub3JnLzATBgNV
HSAEDDAKMAgGBmeBDAECATArBgNVHR8EJDAiMCCgHqAchhpodHRwOi8vc3RnLXgx
LmMubGVuY3Iub3JnLzANBgkqhkiG9w0BAQsFAAOCAgEAeD9pt5sWtXEQyF94RMzo
lF+uip+ODYaNxPEJG/93xtxPc6QC0bsC9QIgv6dBk+s1LUf5ulIlJZQ1UlkLgTH4
HzHi37O4ivcbSHpILcgD0hKnftv0E8o9SNAaIcO7DKNmbeWQ6cPlfOlRtpcIXYoc
95wY9AipN0/ken5KxDxdxTSKuWS7Ae+ZWh7hmLdfDc5C85o2omLvM/6Ovmp0xUSP
U/xFR1fk0kRmM975smehEOXSHOJ24zFaY6BnMUWa67yLawbxUj0AqzGmohdqz7Lu
DhtKd9oXJgRzacFzuMSQluN0ivwKIzirpLR/AJoz4nHXIaDQRyfaA3jr1VYpK+bA
poIXAjgxR/vEFtbEHw7fNO3FKRCddvyJ0vh+OKIrY0nrbGUoyRlPlKF/9XrQ3UJo
Zc708mBBjzqijXdUQ0TB1/irAAQ7O/6i+ThQxElt77gYWzv2/8sSoagzqRKXOqLJ
ckNJQIA39t4HMIFgqxDLRuRNtooFUeoOvocDWTQr80bUp6357cBtAl7ypcOzrPK6
m5cuuyZfQ7zdh5ufcfMdFbgjwfeN8d442EzBnLxoMA9T2Igfn7KOhFl3znZ+WJhS
h+oyBS8PrKUKB2Xo9pnXma6123xyFPDapnuK3iiNpGJfI0sULCqjEodYG+aGyfCz
ZFLWftbU683YxZCOddgYxHI=
-----END CERTIFICATE-----

s:C = US, O = (STAGING) Let's Encrypt, CN = (STAGING) Pseudo Plum E5
i:C = US, O = (STAGING) Internet Security Research Group, CN = (STAGING) Pretend Pear X1
-----BEGIN CERTIFICATE-----
MIIEljCCAn6gAwIBAgIQRzEp1D1mDiVVv4b1zlB56jANBgkqhkiG9w0BAQsFADBm
MQswCQYDVQQGEwJVUzEzMDEGA1UEChMqKFNUQUdJTkcpIEludGVybmV0IFNlY3Vy
aXR5IFJlc2VhcmNoIEdyb3VwMSIwIAYDVQQDExkoU1RBR0lORykgUHJldGVuZCBQ
ZWFyIFgxMB4XDTI0MDMxMzAwMDAwMFoXDTI3MDMxMjIzNTk1OVowUjELMAkGA1UE
BhMCVVMxIDAeBgNVBAoTFyhTVEFHSU5HKSBMZXQncyBFbmNyeXB0MSEwHwYDVQQD
ExgoU1RBR0lORykgUHNldWRvIFBsdW0gRTUwdjAQBgcqhkjOPQIBBgUrgQQAIgNi
AATljbbcV+mqWZa3g+z0bDOuBpZOtbi48iK9rjLtPdRU0WsgVp53MW3nXFU6qVYV
zEYaYd6PSmec0Tj3R5zEp5/F+cuOjTdh3AkTMzYm1tkflocPBN5APHYZ+76WxZad
q+WjggEAMIH9MA4GA1UdDwEB/wQEAwIBhjAdBgNVHSUEFjAUBggrBgEFBQcDAgYI
KwYBBQUHAwEwEgYDVR0TAQH/BAgwBgEB/wIBADAdBgNVHQ4EFgQU/EbRAUNfu3um
PTBorhG64LxtydMwHwYDVR0jBBgwFoAUtfNl8v6wCpIf+zx980SgrGMlwxQwNgYI
KwYBBQUHAQEEKjAoMCYGCCsGAQUFBzAChhpodHRwOi8vc3RnLXgxLmkubGVuY3Iu
b3JnLzATBgNVHSAEDDAKMAgGBmeBDAECATArBgNVHR8EJDAiMCCgHqAchhpodHRw
Oi8vc3RnLXgxLmMubGVuY3Iub3JnLzANBgkqhkiG9w0BAQsFAAOCAgEAAtCGn4iG
cupruhkCTcoDqSIVTFgVR8JJ3GvGL7SYwIc4Fn0As66nQgnkATIzF5+gFb+CXEQD
qR2Jo+R38OeT7lQ1rNDcaJcbY6hL8cNRku3QlcfdYODZ5pgTVH04gTZUJISZKLjD
kMMcQIDZlF7iYqTvmHbn2ISSKorsJ3QKAvWhHwMoJtocSz3VeDJIep5QtbHnoXh1
/dyDx7sp8RuhC0eO9ElTgDtiA2V6JxigLPzqcnibBBR4bFLGtMNE4EvOOD/Fkd0L
hdGDbAMNd+O06n+b0rgmDvg75IgOV6fpDrdZFoiNfCckOEJh9v10uYt4pTc3B6lf
zI/X3EWP1H4VJmsYuy+OA29jPeP831sAObZtd3RWv0LQPrMfx6FCmy4AaeYEMvul
FrF6OX+JbssE+bn83F+sGEMZu/eVBwwKh3db7+2UduMdTOb8DePE3Aqlg9zofS8X
9fJXrrp+PPrdQyvM3e8DxuioWa9GLG30yD9WD6WTlSiiOrdWGOzisWpW4shFoL8u
0EfmeLVU4JVbauhOYZASQXABNeXewe9lqJWwfqaARYpRjyf+jRibn22H5NVK4Vog
l55Iq1rUgjc8r493NaNrlNwG7va7Ztkch5lJ3oL/FEVlVSK4snTbgb0b5qjQz3SA
i7rA/8QRZvOLnKNtdEUlDZNrzkZwHNluLGw=
-----END CERTIFICATE-----
s: C = US, O = (STAGING) Let's Encrypt, CN = (STAGING) False Fennel E6
i:C = US, O = (STAGING) Internet Security Research Group, CN = (STAGING) Pretend Pear X1
-----BEGIN CERTIFICATE-----
MIIElzCCAn+gAwIBAgIQJtklZLQLsngwp0DD4clbajANBgkqhkiG9w0BAQsFADBm
MQswCQYDVQQGEwJVUzEzMDEGA1UEChMqKFNUQUdJTkcpIEludGVybmV0IFNlY3Vy
aXR5IFJlc2VhcmNoIEdyb3VwMSIwIAYDVQQDExkoU1RBR0lORykgUHJldGVuZCBQ
ZWFyIFgxMB4XDTI0MDMxMzAwMDAwMFoXDTI3MDMxMjIzNTk1OVowUzELMAkGA1UE
BhMCVVMxIDAeBgNVBAoTFyhTVEFHSU5HKSBMZXQncyBFbmNyeXB0MSIwIAYDVQQD
ExkoU1RBR0lORykgRmFsc2UgRmVubmVsIEU2MHYwEAYHKoZIzj0CAQYFK4EEACID
YgAE7mIshEAbfnZhXHmCLThagBq8wNKY84YkUynY83vBo8FGIFdBQ7xQV3NrPkhi
dMXyW4aQcADxF/skAnZAsvnjxR+oCnYZAtBXgeE+WFA6FE7/fr/ecbTglsAgp59e
ADZDo4IBADCB/TAOBgNVHQ8BAf8EBAMCAYYwHQYDVR0lBBYwFAYIKwYBBQUHAwIG
CCsGAQUFBwMBMBIGA1UdEwEB/wQIMAYBAf8CAQAwHQYDVR0OBBYEFKF0GgZtULeG
LUoswX60jYhJbM0WMB8GA1UdIwQYMBaAFLXzZfL+sAqSH/s8ffNEoKxjJcMUMDYG
CCsGAQUFBwEBBCowKDAmBggrBgEFBQcwAoYaaHR0cDovL3N0Zy14MS5pLmxlbmNy
Lm9yZy8wEwYDVR0gBAwwCjAIBgZngQwBAgEwKwYDVR0fBCQwIjAgoB6gHIYaaHR0
cDovL3N0Zy14MS5jLmxlbmNyLm9yZy8wDQYJKoZIhvcNAQELBQADggIBAFGjOrc+
GlZYCuxNKS9u+4l6pDux++Cg3FX3hR0QhCZPKORxnua9dJlles5NBcIf5W1Y2OW7
1El+oq+0cCS39bqfG3d+9sRbqnLlqB9xysRfpGPmY2IRQ4YFurIzurFw/TshoGGw
L74VtBgmxeO1fhP8OWJfnMB3EILdz6hxEqbQtCmU/2ejQTQEK6cQtBj33a+MUI7i
YnRP+OTL1M81tmZRkUWY7WItMGDPIabhXfHqvqYXh0huxqUDls0VjC52xpiK6e2F
+ZxouwmBwQlxSL9pJWMvhRGJdFCIBulQPm/0tbt2seBxkFKWIYmouDh+28swkTSl
f8mV0KGmEVmPka5CQkgSg4btNHUoZ1xwt++f4N12AapSfPDYVyJ69c9fw+kbFTFa
9G1keN75SIJovWWhy4XUEcGkInDM4GyyEff1kpeEeZaOQZe2UojnptfmqLf0qDj6
n02ABUdDiHhQT7Za634b0pxKAPXW2pSFflrGZc2e+pKpIG4ui/80uq8AGybeHda1
Dl7iS2Msv4wpBii4j1Qs8gXAb7yI9ozS2weCmu5oqDN4SnYZBYoirT8seUia62aw
XMVzkkQUsjRQz2oqQsTUZxtDuIU0VCNYPY0UgeSLjdzo+cUIbaoO6M8rAK42tq/1
jJ+0RltUFPHxLXxi0LIAV0tFJxEJVUbpv7Nt
-----END CERTIFICATE-----
5 Likes

It seems like @jf043 is doing this in order to create a working end-to-end test involving staging certificates (using them as part of a larger test environment that's as realistic and full-featured as possible).

2 Likes

@schoen Even so, what do intermediates have to do with that?

4 Likes

Oh yeah, that's true, the most realistic test would indeed be to trust staging roots rather than staging intermediates.

8 Likes

Yes, I have several full stack dev builds that have full chain health checks and other apps/clients need to trust the chain and signers as well. If you share and document root and intermediates certificates here:

and

they need to be updated with recent certificates.

Thank you.

1 Like

@jf043 you should explicitly trust the root by including it in a list of trusted roots, then automatically trust intermediates signed by that root. That's the point of signing the intermediates.

Manually trusting intermediates means your chain validation will frequently break as intermediates change. Staging should be able to randomly introduce a new intermediate tomorrow and your trust chain should still work with no intervention.

5 Likes

Ultimately only the roots are important. The intermediates are nice to know.

The Chains of Trust page is up to date and does not contain staging information. While I agree the staging environment page needs updating (there already is an issue opened on the Let's Encrypt website repository for this as mentioned before), but the roots have remained the same (according to Deploying Let's Encrypt's New Issuance Chains - #4 by mcpherrinm only the intermediates have changed).

So with changed intermediates but identical roots your system should not have any issue with these changes. If they do, then your systems are either faulty or should not error out but only warn on intermediate cert changes IMO.

3 Likes