Possible race condition observed with autocert

Hi there! Got an obscure bug I could use a pointer or two developing. Here's the synopsis:

I'm running a frontend using the golang autocert package, in order to provision certs just-in-time for user-supplied domain names.

It mostly works, however I see an issue for the very first request: Chrome on Mac will display the certificate as invalid for the rest of the session; and Safari on Mac will display it as invalid for just that request. Example from Safari:

Inspecting the cert chain in the UI shows everything is valid.

I'm wondering if this comes down to some sort of expected client/server race condition with local time and NotBefore of the issued certs. If that's the case, I am guessing this comes down to the consequences of JIT provisioning & imperfect client clocks. I can think of three possible solutions:

  • Ask LetsEncrypt for an earlier NotBefore timestamp. (Probably not allowed, for good reason..?)
  • Provision the cert as early as possible, ahead of any real customer request (i.e. avoid JIT).
  • Add an artificial extra delay on first response (eg delay 5 seconds = we can tolerate clients whose clocks are up to 5sec slow).

Any general recommendations are welcome. Thanks!

Let's Encrypt already backdates notBefore by 1 hour. I think it's probably something else.

Putting Safari aside for one second, what's the error code in Chrome, when you encounter this? e.g. NET::ERR_CERT_AUTHORITY_INVALID.

If you click that error text in Chrome, it should show the full detail of the certificate which produces that error. It would be helpful to see that.

2 Likes

As far as I know, LE already issues certs with an earlier NotBefore entry, I think it's an hour earlier than the actual issuing date (but could be mistaken in that one hour).

Unfortunately, I can't see in your screenshot why the certificate was marked as invalid, as the valid snippet is pasted over the error..

2 Likes

Wow thanks for the fast responses! Great community here.

The error in Chrome is indeed NET::ERR_CERT_AUTHORITY_INVALID. If you are curious to try reproducing it directly, I've pointed *.taps.live at the frontend; any random host repros it for me.

Edit: And here is the dump (this from Chrome):

Subject: asdasdads.taps.live

Issuer: Let's Encrypt Authority X3

Expires on: Jan 29, 2021

Current date: Oct 31, 2020

PEM encoded chain:
-----BEGIN CERTIFICATE-----
MIIEkTCCA3mgAwIBAgISBEHjDDgXODXIsqQwOp02njf3MA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD
ExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0yMDEwMzExOTE4MTRaFw0y
MTAxMjkxOTE4MTRaMB4xHDAaBgNVBAMTE2FzZGFzZGFkcy50YXBzLmxpdmUwWTAT
BgcqhkjOPQIBBggqhkjOPQMBBwNCAATrrK1O/kkkk5RpEVjNInLylpz9x5D3bc+N
J3/730f9OdpgIxH8lTjoEaV5zn2GchEE1rsKXrH+gVhsMFrglqvLo4ICZjCCAmIw
DgYDVR0PAQH/BAQDAgeAMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAM
BgNVHRMBAf8EAjAAMB0GA1UdDgQWBBSuasjxMiynK4d/fnwX8BuyeAGeHTAfBgNV
HSMEGDAWgBSoSmpjBH3duubRObemRWXv86jsoTBvBggrBgEFBQcBAQRjMGEwLgYI
KwYBBQUHMAGGImh0dHA6Ly9vY3NwLmludC14My5sZXRzZW5jcnlwdC5vcmcwLwYI
KwYBBQUHMAKGI2h0dHA6Ly9jZXJ0LmludC14My5sZXRzZW5jcnlwdC5vcmcvMB4G
A1UdEQQXMBWCE2FzZGFzZGFkcy50YXBzLmxpdmUwTAYDVR0gBEUwQzAIBgZngQwB
AgEwNwYLKwYBBAGC3xMBAQEwKDAmBggrBgEFBQcCARYaaHR0cDovL2Nwcy5sZXRz
ZW5jcnlwdC5vcmcwggECBgorBgEEAdZ5AgQCBIHzBIHwAO4AdQCUILwejtWNbIhz
H4KLIiwN0dpNXmxPlD1h204vWE2iwgAAAXWAT3xhAAAEAwBGMEQCICwAupXe14TO
kVIDaH0hvP/D9Yy8YHr249rVOLx67/5qAiAk6yCkaGgr+F4kRXdGN9PqFhnvvZVf
EGbojgokh+KzvQB1AH0+8viP/4hVaCTCwMqeUol5K8UOeAl/LmqXaJl+IvDXAAAB
dYBPfKIAAAQDAEYwRAIgHx3n5uCAiEE4Gg6sTRGuwGsVDvjsmWC9t6esPnoVuK0C
IGVr1GfFQiNJgprk0jZUHy9JeQJQqt/2EYobgxJ7afCjMA0GCSqGSIb3DQEBCwUA
A4IBAQBy8Ulno9x5HpoxmeCXCsSU8IvM9sx5mV6d87DqwJAHFYYvKZmLVOMOM2Ei
qQAmFDWcGhwETK3VqtaWnYVZmm2IlPN31q61HD5ADU70vQVD9HaTnhl8z3bOORll
MF0J0BJX0chgyzpDjXdBcBYiWiX/zU+x2zEDs8HgtpZeufiMNqQQBSAHX+h/1fmZ
X5ldoiwGBKi9I02LclELgAT9Bu3wD9rDQgk0JV6wzkmYJKfZ7nL+lIRBZ22u/FsJ
WotQ1MeR32nC3hVECv6cP8CH4XCjdV2PvKgYlXygK7aHE3KF73Lp5A/tHeM+xW5G
+rcbcQk1ZbcyLrlsbxk2JkE77UTV
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIEkjCCA3qgAwIBAgIQCgFBQgAAAVOFc2oLheynCDANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTE2MDMxNzE2NDA0NloXDTIxMDMxNzE2NDA0Nlow
SjELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUxldCdzIEVuY3J5cHQxIzAhBgNVBAMT
GkxldCdzIEVuY3J5cHQgQXV0aG9yaXR5IFgzMIIBIjANBgkqhkiG9w0BAQEFAAOC
AQ8AMIIBCgKCAQEAnNMM8FrlLke3cl03g7NoYzDq1zUmGSXhvb418XCSL7e4S0EF
q6meNQhY7LEqxGiHC6PjdeTm86dicbp5gWAf15Gan/PQeGdxyGkOlZHP/uaZ6WA8
SMx+yk13EiSdRxta67nsHjcAHJyse6cF6s5K671B5TaYucv9bTyWaN8jKkKQDIZ0
Z8h/pZq4UmEUEz9l6YKHy9v6Dlb2honzhT+Xhq+w3Brvaw2VFn3EK6BlspkENnWA
a6xK8xuQSXgvopZPKiAlKQTGdMDQMc2PMTiVFrqoM7hD8bEfwzB/onkxEz0tNvjj
/PIzark5McWvxI0NHWQWM6r6hCm21AvA2H3DkwIDAQABo4IBfTCCAXkwEgYDVR0T
AQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8EBAMCAYYwfwYIKwYBBQUHAQEEczBxMDIG
CCsGAQUFBzABhiZodHRwOi8vaXNyZy50cnVzdGlkLm9jc3AuaWRlbnRydXN0LmNv
bTA7BggrBgEFBQcwAoYvaHR0cDovL2FwcHMuaWRlbnRydXN0LmNvbS9yb290cy9k
c3Ryb290Y2F4My5wN2MwHwYDVR0jBBgwFoAUxKexpHsscfrb4UuQdf/EFWCFiRAw
VAYDVR0gBE0wSzAIBgZngQwBAgEwPwYLKwYBBAGC3xMBAQEwMDAuBggrBgEFBQcC
ARYiaHR0cDovL2Nwcy5yb290LXgxLmxldHNlbmNyeXB0Lm9yZzA8BgNVHR8ENTAz
MDGgL6AthitodHRwOi8vY3JsLmlkZW50cnVzdC5jb20vRFNUUk9PVENBWDNDUkwu
Y3JsMB0GA1UdDgQWBBSoSmpjBH3duubRObemRWXv86jsoTANBgkqhkiG9w0BAQsF
AAOCAQEA3TPXEfNjWDjdGBX7CVW+dla5cEilaUcne8IkCJLxWh9KEik3JHRRHGJo
uM2VcGfl96S8TihRzZvoroed6ti6WqEBmtzw3Wodatg+VyOeph4EYpr/1wXKtx8/
wApIvJSwtmVi4MFU5aMqrSDE6ea73Mj2tcMyo5jMd6jmeWUHK8so/joWUoHOUgwu
X4Po1QYz+3dszkDqMp4fklxBwXRsW10KXzPMTZ+sOPAveyxindmjkW8lGy+QsRlG
PfZ+G6Z6h7mjem0Y+iWlkYcV4PIWL1iwBi8saCbGS5jN2p8M+X+Q7UNKEkROb3N6
KOqkqm57TH2H3eDJAkSnh6/DNFu0Qg==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTAwMDkzMDIxMTIxOVoXDTIxMDkzMDE0MDExNVow
PzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1cmUgVHJ1c3QgQ28uMRcwFQYDVQQD
Ew5EU1QgUm9vdCBDQSBYMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AN+v6ZdQCINXtMxiZfaQguzH0yxrMMpb7NnDfcdAwRgUi+DoM3ZJKuM/IUmTrE4O
rz5Iy2Xu/NMhD2XSKtkyj4zl93ewEnu1lcCJo6m67XMuegwGMoOifooUMM0RoOEq
OLl5CjH9UL2AZd+3UWODyOKIYepLYYHsUmu5ouJLGiifSKOeDNoJjj4XLh7dIN9b
xiqKqy69cK3FCxolkHRyxXtqqzTWMIn/5WgTe1QLyNau7Fqckh49ZLOMxt+/yUFw
7BZy1SbsOFU5Q9D8/RhcQPGX69Wam40dutolucbY38EVAjqr2m7xPi71XAicPNaD
aeQQmxkqtilX4+U9m5/wAl0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNV
HQ8BAf8EBAMCAQYwHQYDVR0OBBYEFMSnsaR7LHH62+FLkHX/xBVghYkQMA0GCSqG
SIb3DQEBBQUAA4IBAQCjGiybFwBcqR7uKGY3Or+Dxz9LwwmglSBd49lZRNI+DT69
ikugdB/OEIKcdBodfpga3csTS7MgROSR6cz8faXbauX+5v3gTt23ADq1cEmv8uXr
AvHRAosZy5Q6XkjEGB5YGV8eAlrwDPGxrancWYaLbumR9YbK+rlmM6pZW87ipxZz
R8srzJmwN0jP41ZL9c8PDHIyh8bwRLtTcm1D9SZImlJnt1ir/md2cXjbDaJWFBM5
JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubSfZGL+T0yjWW06XyxV3bqxbYo
Ob8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ
-----END CERTIFICATE-----
1 Like

I can't seem to find anything technically wrong:

openssl s_client -connect ${random}.taps.live:443 -showcerts

Shows a correct chain..

openssl s_client -connect ${random}.taps.live:443 | openssl x509 -noout -text

Shows a correct certificate with the NotBefore timestamp indeed 1 hour before the actual time.

2 Likes

I do have one theory about why this kind of thing could happen ... but I have very little proof so far. (My related post).

I think it could be related to clock skew, but related to the timestamp in the certificate transparency SCT rather than in the certificate itself. The SCT timestamp is not backdated, and browsers are supposed to reject timestamps in the future. Chromium and Safari both enforce SCTs, but I'm not sure whether they have done something to account for this. Safari is closed source and Chromium's SCT validator seems to change constantly.

In your case though, if you're getting a transient ERR_CERT_AUTHORITY_INVALID error (rather than the CT one), I think this it's probably something else.

You could try create the net-export dump as documented in my post anyway, as it creates very detailed logs about the certificate verification process. Could help, who knows.

FWIW I couldn't reproduce the problem with your domain on Linux+Chromium.

2 Likes

I can't reproduce it either on my Chromium browser. But of course, I've got ntpd running :stuck_out_tongue:

1 Like

I can't reproduce it either on my Chromium browser. But of course, I've got ntpd running :stuck_out_tongue:

Hah - forgot to say I did check that as well; no smoking gun, drift of a few microseconds..

You could try create the net-export dump as documented in my post anyway, as it creates very detailed logs about the certificate verification process. Could help, who knows.

I'll give that a shot! I've noticed the repo isn't 100% reliable, I managed to see one domain complete successfully.

1 Like

Update: Tried with a trunk Chromium build and can't reproduce at all (same machine); consistent with your findings.

Okay! Got a capture of a repro with Chrome. If you're interested in taking a look, would you mind DM'ing me? (I don't think my newbie forum rep allows me to initiate, but am guessing I can respond).

I DM'd you, but if that doesn't work, my email address is in that other post as well.

1 Like

:frowning: . Still no closer to figuring it out.

Looks like it is not the SCTs. Chrome's certificate verification process begins around ~350-400ms after the SCT timestamps in the certificate; no problems there.

The actual cause of the ERR_CERT_AUTHORITY_INVALID is unfortunately not surfaced in the logs. But I really appreciate you collecting them.

150: CERT_VERIFIER_TASK
Start Time: 2020-11-01 08:03:03.404

t=4785 [st=  0] +CERT_VERIFIER_TASK  [dt=166]
t=4785 [st=  0]   +CERT_VERIFY_PROC  [dt=166]
                  --> certificates = ["-----BEGIN CERTIFICATE-----\nMIIEkjCCA3qgAwIBAgISA5DoGeITSfN1pmh90def4t79MA0GCSqGSIb3DQEBCwUA\nMEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD\nExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0yMDEwMzEyMDAzMDNaFw0y\nMTAxMjkyMDAzMDNaMB0xGzAZBgNVBAMTEnRlc3Q1NTU1LnRhcHMubGl2ZTBZMBMG\nByqGSM49AgEGCCqGSM49AwEHA0IABC3bYZd2UUMSsuV66cqbNzKQB0lwSKhlCk+8\n6em1iCmxucOS0tcsVZWN3YujUMPEWua5JQxJkeaOBjxUMnUI/3KjggJoMIICZDAO\nBgNVHQ8BAf8EBAMCB4AwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwG\nA1UdEwEB/wQCMAAwHQYDVR0OBBYEFLUWZytwXOnAgGvXNXDQkjrBMQrRMB8GA1Ud\nIwQYMBaAFKhKamMEfd265tE5t6ZFZe/zqOyhMG8GCCsGAQUFBwEBBGMwYTAuBggr\nBgEFBQcwAYYiaHR0cDovL29jc3AuaW50LXgzLmxldHNlbmNyeXB0Lm9yZzAvBggr\nBgEFBQcwAoYjaHR0cDovL2NlcnQuaW50LXgzLmxldHNlbmNyeXB0Lm9yZy8wHQYD\nVR0RBBYwFIISdGVzdDU1NTUudGFwcy5saXZlMEwGA1UdIARFMEMwCAYGZ4EMAQIB\nMDcGCysGAQQBgt8TAQEBMCgwJgYIKwYBBQUHAgEWGmh0dHA6Ly9jcHMubGV0c2Vu\nY3J5cHQub3JnMIIBBQYKKwYBBAHWeQIEAgSB9gSB8wDxAHcAXNxDkv7mq0VEsV6a\n1FbmEDf71fpH3KFzlLJe5vbHDsoAAAF1gHiDlgAABAMASDBGAiEA9XFSBXUL+tFo\nP6Sxptfinl5+cE5XOggVySOyRPUYPJwCIQC4kSXms+fwVqBHE9tEe9IZybXQX5BZ\nsUbF+d7FwaFuiAB2APZclC/RdzAiFFQYCDCUVo7jTRMZM7/fDC8gC8xO8WTjAAAB\ndYB4g7cAAAQDAEcwRQIhAN01sOrpeRco5QqwE573bWOfks/DTK2Q9KnL7qNtf2rW\nAiBz1UkxuLljEv2PdF6E7551Lk03svmphjclgzCb1AbrcTANBgkqhkiG9w0BAQsF\nAAOCAQEAmOP16PSfcx6t19q5v/Ev2yRPpot9y20fcf1MLlhJ9xd3o7zef0Zg98Kj\nocolXq+ncOfuPmM5MJIULsztHGBbEpRDYtulXTtTrvKGeYmQH4zx+CVgQlOgpL7Z\nbYdwCUxFcY9nFlgbyjAoaLK6hydn+cvc4/7ifCmiA3ibnq2iVObDCs+a5yEsBNTc\np94SOLsrXoSOIjWj/dKtCtMq3dOuXoMBJbbWjsIAMkZcyfPa1O/lzHWdUdB3wh2J\nEmkZ5YjRKmXml+JIh67wRVLc2d7ImDpmD3zY0uzu/LJZIYp7NCnRxJmUCIsHcB6S\na+oHodly0DOzOZ/1zE4MWATx5NKFoQ==\n-----END CERTIFICATE-----\n","-----BEGIN CERTIFICATE-----\nMIIEkjCCA3qgAwIBAgIQCgFBQgAAAVOFc2oLheynCDANBgkqhkiG9w0BAQsFADA/\nMSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT\nDkRTVCBSb290IENBIFgzMB4XDTE2MDMxNzE2NDA0NloXDTIxMDMxNzE2NDA0Nlow\nSjELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUxldCdzIEVuY3J5cHQxIzAhBgNVBAMT\nGkxldCdzIEVuY3J5cHQgQXV0aG9yaXR5IFgzMIIBIjANBgkqhkiG9w0BAQEFAAOC\nAQ8AMIIBCgKCAQEAnNMM8FrlLke3cl03g7NoYzDq1zUmGSXhvb418XCSL7e4S0EF\nq6meNQhY7LEqxGiHC6PjdeTm86dicbp5gWAf15Gan/PQeGdxyGkOlZHP/uaZ6WA8\nSMx+yk13EiSdRxta67nsHjcAHJyse6cF6s5K671B5TaYucv9bTyWaN8jKkKQDIZ0\nZ8h/pZq4UmEUEz9l6YKHy9v6Dlb2honzhT+Xhq+w3Brvaw2VFn3EK6BlspkENnWA\na6xK8xuQSXgvopZPKiAlKQTGdMDQMc2PMTiVFrqoM7hD8bEfwzB/onkxEz0tNvjj\n/PIzark5McWvxI0NHWQWM6r6hCm21AvA2H3DkwIDAQABo4IBfTCCAXkwEgYDVR0T\nAQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8EBAMCAYYwfwYIKwYBBQUHAQEEczBxMDIG\nCCsGAQUFBzABhiZodHRwOi8vaXNyZy50cnVzdGlkLm9jc3AuaWRlbnRydXN0LmNv\nbTA7BggrBgEFBQcwAoYvaHR0cDovL2FwcHMuaWRlbnRydXN0LmNvbS9yb290cy9k\nc3Ryb290Y2F4My5wN2MwHwYDVR0jBBgwFoAUxKexpHsscfrb4UuQdf/EFWCFiRAw\nVAYDVR0gBE0wSzAIBgZngQwBAgEwPwYLKwYBBAGC3xMBAQEwMDAuBggrBgEFBQcC\nARYiaHR0cDovL2Nwcy5yb290LXgxLmxldHNlbmNyeXB0Lm9yZzA8BgNVHR8ENTAz\nMDGgL6AthitodHRwOi8vY3JsLmlkZW50cnVzdC5jb20vRFNUUk9PVENBWDNDUkwu\nY3JsMB0GA1UdDgQWBBSoSmpjBH3duubRObemRWXv86jsoTANBgkqhkiG9w0BAQsF\nAAOCAQEA3TPXEfNjWDjdGBX7CVW+dla5cEilaUcne8IkCJLxWh9KEik3JHRRHGJo\nuM2VcGfl96S8TihRzZvoroed6ti6WqEBmtzw3Wodatg+VyOeph4EYpr/1wXKtx8/\nwApIvJSwtmVi4MFU5aMqrSDE6ea73Mj2tcMyo5jMd6jmeWUHK8so/joWUoHOUgwu\nX4Po1QYz+3dszkDqMp4fklxBwXRsW10KXzPMTZ+sOPAveyxindmjkW8lGy+QsRlG\nPfZ+G6Z6h7mjem0Y+iWlkYcV4PIWL1iwBi8saCbGS5jN2p8M+X+Q7UNKEkROb3N6\nKOqkqm57TH2H3eDJAkSnh6/DNFu0Qg==\n-----END CERTIFICATE-----\n"]
                  --> crlset_sequence = 6211
                  --> host = "test5555.taps.live"
                  --> verify_flags = 0
t=4951 [st=166]   -CERT_VERIFY_PROC
                  --> cert_status = 4
                  --> is_issued_by_known_root = true
                  --> net_error = -202 (ERR_CERT_AUTHORITY_INVALID)
                  --> public_key_hashes = ["sha256/Dz13aZC5lYrcM1OkFnMHLz8TA7a9Qq60NzFNtUiCRiA=","sha256/YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg=","sha256/Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="]
                  --> verified_cert = {"certificates":["-----BEGIN CERTIFICATE-----\nMIIEkjCCA3qgAwIBAgISA5DoGeITSfN1pmh90def4t79MA0GCSqGSIb3DQEBCwUA\nMEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD\nExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0yMDEwMzEyMDAzMDNaFw0y\nMTAxMjkyMDAzMDNaMB0xGzAZBgNVBAMTEnRlc3Q1NTU1LnRhcHMubGl2ZTBZMBMG\nByqGSM49AgEGCCqGSM49AwEHA0IABC3bYZd2UUMSsuV66cqbNzKQB0lwSKhlCk+8\n6em1iCmxucOS0tcsVZWN3YujUMPEWua5JQxJkeaOBjxUMnUI/3KjggJoMIICZDAO\nBgNVHQ8BAf8EBAMCB4AwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwG\nA1UdEwEB/wQCMAAwHQYDVR0OBBYEFLUWZytwXOnAgGvXNXDQkjrBMQrRMB8GA1Ud\nIwQYMBaAFKhKamMEfd265tE5t6ZFZe/zqOyhMG8GCCsGAQUFBwEBBGMwYTAuBggr\nBgEFBQcwAYYiaHR0cDovL29jc3AuaW50LXgzLmxldHNlbmNyeXB0Lm9yZzAvBggr\nBgEFBQcwAoYjaHR0cDovL2NlcnQuaW50LXgzLmxldHNlbmNyeXB0Lm9yZy8wHQYD\nVR0RBBYwFIISdGVzdDU1NTUudGFwcy5saXZlMEwGA1UdIARFMEMwCAYGZ4EMAQIB\nMDcGCysGAQQBgt8TAQEBMCgwJgYIKwYBBQUHAgEWGmh0dHA6Ly9jcHMubGV0c2Vu\nY3J5cHQub3JnMIIBBQYKKwYBBAHWeQIEAgSB9gSB8wDxAHcAXNxDkv7mq0VEsV6a\n1FbmEDf71fpH3KFzlLJe5vbHDsoAAAF1gHiDlgAABAMASDBGAiEA9XFSBXUL+tFo\nP6Sxptfinl5+cE5XOggVySOyRPUYPJwCIQC4kSXms+fwVqBHE9tEe9IZybXQX5BZ\nsUbF+d7FwaFuiAB2APZclC/RdzAiFFQYCDCUVo7jTRMZM7/fDC8gC8xO8WTjAAAB\ndYB4g7cAAAQDAEcwRQIhAN01sOrpeRco5QqwE573bWOfks/DTK2Q9KnL7qNtf2rW\nAiBz1UkxuLljEv2PdF6E7551Lk03svmphjclgzCb1AbrcTANBgkqhkiG9w0BAQsF\nAAOCAQEAmOP16PSfcx6t19q5v/Ev2yRPpot9y20fcf1MLlhJ9xd3o7zef0Zg98Kj\nocolXq+ncOfuPmM5MJIULsztHGBbEpRDYtulXTtTrvKGeYmQH4zx+CVgQlOgpL7Z\nbYdwCUxFcY9nFlgbyjAoaLK6hydn+cvc4/7ifCmiA3ibnq2iVObDCs+a5yEsBNTc\np94SOLsrXoSOIjWj/dKtCtMq3dOuXoMBJbbWjsIAMkZcyfPa1O/lzHWdUdB3wh2J\nEmkZ5YjRKmXml+JIh67wRVLc2d7ImDpmD3zY0uzu/LJZIYp7NCnRxJmUCIsHcB6S\na+oHodly0DOzOZ/1zE4MWATx5NKFoQ==\n-----END CERTIFICATE-----\n","-----BEGIN CERTIFICATE-----\nMIIEkjCCA3qgAwIBAgIQCgFBQgAAAVOFc2oLheynCDANBgkqhkiG9w0BAQsFADA/\nMSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT\nDkRTVCBSb290IENBIFgzMB4XDTE2MDMxNzE2NDA0NloXDTIxMDMxNzE2NDA0Nlow\nSjELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUxldCdzIEVuY3J5cHQxIzAhBgNVBAMT\nGkxldCdzIEVuY3J5cHQgQXV0aG9yaXR5IFgzMIIBIjANBgkqhkiG9w0BAQEFAAOC\nAQ8AMIIBCgKCAQEAnNMM8FrlLke3cl03g7NoYzDq1zUmGSXhvb418XCSL7e4S0EF\nq6meNQhY7LEqxGiHC6PjdeTm86dicbp5gWAf15Gan/PQeGdxyGkOlZHP/uaZ6WA8\nSMx+yk13EiSdRxta67nsHjcAHJyse6cF6s5K671B5TaYucv9bTyWaN8jKkKQDIZ0\nZ8h/pZq4UmEUEz9l6YKHy9v6Dlb2honzhT+Xhq+w3Brvaw2VFn3EK6BlspkENnWA\na6xK8xuQSXgvopZPKiAlKQTGdMDQMc2PMTiVFrqoM7hD8bEfwzB/onkxEz0tNvjj\n/PIzark5McWvxI0NHWQWM6r6hCm21AvA2H3DkwIDAQABo4IBfTCCAXkwEgYDVR0T\nAQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8EBAMCAYYwfwYIKwYBBQUHAQEEczBxMDIG\nCCsGAQUFBzABhiZodHRwOi8vaXNyZy50cnVzdGlkLm9jc3AuaWRlbnRydXN0LmNv\nbTA7BggrBgEFBQcwAoYvaHR0cDovL2FwcHMuaWRlbnRydXN0LmNvbS9yb290cy9k\nc3Ryb290Y2F4My5wN2MwHwYDVR0jBBgwFoAUxKexpHsscfrb4UuQdf/EFWCFiRAw\nVAYDVR0gBE0wSzAIBgZngQwBAgEwPwYLKwYBBAGC3xMBAQEwMDAuBggrBgEFBQcC\nARYiaHR0cDovL2Nwcy5yb290LXgxLmxldHNlbmNyeXB0Lm9yZzA8BgNVHR8ENTAz\nMDGgL6AthitodHRwOi8vY3JsLmlkZW50cnVzdC5jb20vRFNUUk9PVENBWDNDUkwu\nY3JsMB0GA1UdDgQWBBSoSmpjBH3duubRObemRWXv86jsoTANBgkqhkiG9w0BAQsF\nAAOCAQEA3TPXEfNjWDjdGBX7CVW+dla5cEilaUcne8IkCJLxWh9KEik3JHRRHGJo\nuM2VcGfl96S8TihRzZvoroed6ti6WqEBmtzw3Wodatg+VyOeph4EYpr/1wXKtx8/\nwApIvJSwtmVi4MFU5aMqrSDE6ea73Mj2tcMyo5jMd6jmeWUHK8so/joWUoHOUgwu\nX4Po1QYz+3dszkDqMp4fklxBwXRsW10KXzPMTZ+sOPAveyxindmjkW8lGy+QsRlG\nPfZ+G6Z6h7mjem0Y+iWlkYcV4PIWL1iwBi8saCbGS5jN2p8M+X+Q7UNKEkROb3N6\nKOqkqm57TH2H3eDJAkSnh6/DNFu0Qg==\n-----END CERTIFICATE-----\n","-----BEGIN CERTIFICATE-----\nMIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/\nMSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT\nDkRTVCBSb290IENBIFgzMB4XDTAwMDkzMDIxMTIxOVoXDTIxMDkzMDE0MDExNVow\nPzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1cmUgVHJ1c3QgQ28uMRcwFQYDVQQD\nEw5EU1QgUm9vdCBDQSBYMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB\nAN+v6ZdQCINXtMxiZfaQguzH0yxrMMpb7NnDfcdAwRgUi+DoM3ZJKuM/IUmTrE4O\nrz5Iy2Xu/NMhD2XSKtkyj4zl93ewEnu1lcCJo6m67XMuegwGMoOifooUMM0RoOEq\nOLl5CjH9UL2AZd+3UWODyOKIYepLYYHsUmu5ouJLGiifSKOeDNoJjj4XLh7dIN9b\nxiqKqy69cK3FCxolkHRyxXtqqzTWMIn/5WgTe1QLyNau7Fqckh49ZLOMxt+/yUFw\n7BZy1SbsOFU5Q9D8/RhcQPGX69Wam40dutolucbY38EVAjqr2m7xPi71XAicPNaD\naeQQmxkqtilX4+U9m5/wAl0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNV\nHQ8BAf8EBAMCAQYwHQYDVR0OBBYEFMSnsaR7LHH62+FLkHX/xBVghYkQMA0GCSqG\nSIb3DQEBBQUAA4IBAQCjGiybFwBcqR7uKGY3Or+Dxz9LwwmglSBd49lZRNI+DT69\nikugdB/OEIKcdBodfpga3csTS7MgROSR6cz8faXbauX+5v3gTt23ADq1cEmv8uXr\nAvHRAosZy5Q6XkjEGB5YGV8eAlrwDPGxrancWYaLbumR9YbK+rlmM6pZW87ipxZz\nR8srzJmwN0jP41ZL9c8PDHIyh8bwRLtTcm1D9SZImlJnt1ir/md2cXjbDaJWFBM5\nJDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubSfZGL+T0yjWW06XyxV3bqxbYo\nOb8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ\n-----END CERTIFICATE-----\n"]}
t=4951 [st=166] -CERT_VERIFIER_TASK

I had the same thing when trying to catch this on Windows. I wonder if there's a difference between the Chromium and Google builds relating to when built-in libraries vs the macOS Security Framework is used to verify.

Doubly annoying because there's a neat cert_verify_tool that you can use debug verifiction at a certain timestamp without constantly recreating certifictes, but you'd need the one from a Google build where the problem is reproducible.

2 Likes

I wonder if there's a difference between the Chromium and Google builds relating to when built-in libraries vs the macOS Security Framework is used to verify.

Hey! I think you might be on to something there. I don't know these systems all that well, but on a lark I popped open console, tried to repro, and dug through all the garbage in syslog when I did. Sure enough:

trustd | SCT is in the future: 1604195641038 > 1604195641000

Damned if that doesn't look like some sort of milisecond-precision clock and microsecond-precision clock getting into a silly disagreement, eh...?

I can also confirm this log entry does not show up for non-repro loads.... :thinking:

4 Likes

Damned if that doesn't look like some sort of milisecond-precision clock and microsecond-precision clock getting into a silly disagreement, eh...?

Correcting myself: Probably right idea, wrong units: The value on the right (trustd's now()?) looks like a seconds-precision timestamp being converted to & compared with millis. In all repros I've found, the value is 0 % 1000. Fresh example:

SCT is in the future: 1604197302365 > 1604197302000
2 Likes

Wow, trustd logs. That's a really nice find, good job! I am also very pleased that my theory actually went somewhere.

Let's say that the immediate issue is that trustd is querying a clock with seconds precision (or is lazy code which only looks at timeval.tv_sec?).

Doesn't this mean that certificate backdating (to fix client clock skew) is basically ineffectual for any UA with a CT policy? I don't think it's possible/practical to backdate SCTs. @jsha have you come across this before?

5 Likes

Interesting! I haven't seen anything like this before. And I think you may be right that backdating to any time earlier than one of the SCTs would be expected to break on clients that enforce "SCT not in future."

One simple fix for ACME clients is to wait a little while before making a certificate available. In general one can't rely on issuance happening within seconds anyhow.

2 Likes

Today I've learned a lot more about CT than I bargained for, but it's been fun! Thanks for all the help and pointers. I have a workaround, and now just some curiosities:

I think you may be right that backdating to any time earlier than one of the SCTs would be expected to break on clients that enforce "SCT not in future."

Which system actually creates this timestamp, and what would be the consequences of back dating it?

Something that's unsatisfying to me -- and maybe it just needs to be -- is that there isn't doesn't seem to be an accommodation for clock skew possible in the CT protocol here (where dialing back NotBefore an hour is the solution for the same real-world problem for cert validity).

In general one can't rely on issuance happening within seconds anyhow.

I wonder if autocert is a bit of an unwitting culprit here, maybe even shifting that expectation: It sure does make it very easy to code up "just in time"-issuing frontends. Which in turn seem like they can achieve a much shorter window between <time cert issued> and <time of first real client request>, leading to this kind of sensitivity.

(I guess the trustd lazy timeval bug is just 'helpfully' simulating up to 999ms of ntp drift..)

One simple fix for ACME clients is to wait a little while before making a certificate available.

I'll definitely be pursuing this as a workaround!

cheers,
mikey

3 Likes

They come from third parties.

Let's Encrypt gets countersignatures from Google + 1 other trusted log operator, for the certificate it intends to issue, in the form of SCTs.

The timestamp itself is generated by that log operator and Let's Encrypt doesn't have any say in its value.

Let's Encrypt then includes those SCTs in the final certificate as proof that the certificate has been publicly logged.

Reading RFC6962-bis ("Certificate Transparency Version 2.0"), there are some changes:

  • The requirement to reject future timestamps is gone and instead it's upto the client.
  • Log entries don't need to be in chronological order

but I don't think these exist to enable backdating.

I also realized that OCSP is probably one more place where you could run into trouble because those responses are not backdated by Let's Encrypt either. However, I think browsers are pretty lax when it comes to OCSP problems.

3 Likes

Just adding this tangent-

On macOS/OSX, Chrome seems to cache invalid certs until you restart the browser application. This has happened to me a few times in the past.

In the cases I have witnessed, it was not because of this time-delay issue, but because of expired certificates. The servers had been restarted with updated certificates and were serving them - even for days - but Chrome kept believing/insisting the old certificates were in use. While I usually experienced this as a Publisher, I experienced this as a consumer this past week, when gitter.im pushed a new release and accidentally deployed an expired certificate.

Perhaps the Chrome bug is not due to caching the certificate, but due to caching the certificate's validation.

1 Like

I think this is true, and in general I strongly discourage writing just-in-time issuing frontends. A couple of reasons:

  • This style of frontend takes a much harder dependency on Let's Encrypt's uptime. If we're down, you're down.
  • We don't guarantee that issuance will happen in seconds. We may very well add validation processes in the future that take longer, which would break the assumptions of this style of clients.

I have also noticed that autocert in particular is prone to a particular failure mode where it spams our service extremely rapidly - up to millions of requests. Because it doesn't log requests or alert its maintainer, this often goes unfixed. And because autocert's example code doesn't set an email address, most autocert users don't set one in their configs. That means it's hard for us to reach out when clients go bad and get the underlying bugs fixed.