How do certificates end up on crt.sh?

[AndrewSav]

Hello,

I created a certificate yesterday, and today I was searching it on crt.sh and to my surprise I could not located it. Searched both by identity and serial. This never happened to me before.

So can someone explain to me how the certificates end up on crt.sh and what can account in principle for mine not listed there?

Cheers,
Andrew

[schoen]

Hi @AndrewSav,

crt.sh is a web interface to a distributed database called the certificate transparency logs.

What is Certificate Transparency? - Certificate Transparency

Let’s Encrypt submits newly issued certificates to certificate transparency logs. It takes time for the certificates to be received by the logs and for crt.sh to download the associated information. But normally I believe this should be on the order of an hour or so, not a day or so. Would you like to share a PEM file so that we can compare?

[AndrewSav]

Here:

openssl s_client -showcerts -connect mail.savinykh.nz:443 </dev/null 2>/dev/null|openssl x509 -outform PEM

[rg305]

There is no way to be excluded from that list - all certs are listed.

[AndrewSav]

I think you are mistaken. This one is not. The one that you linked is completely different cert. @schoen, what do you think?

[schoen]

I agree that it’s not present in either https://crt.sh/?Identity=mail.savinykh.nz&iCAID=16418 or https://transparencyreport.google.com/https/certificates?cert_search_auth=&cert_search_cert=&cert_search=include_expired:false;include_subdomains:false;domain:mail.savinykh.nz&lu=cert_search (the other popular CT web interface). The certificate is about 13 hours old, and that feels like an unusually long delay for visible CT inclusion to me.

@cpu, could you check on this if it’s still missing in the morning?

-----BEGIN CERTIFICATE-----
MIIGWDCCBUCgAwIBAgISAxfGHlh8TZj7H09CBl6tPMi4MA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD
ExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0xNzExMTkxNzQ0NDBaFw0x
ODAyMTcxNzQ0NDBaMBsxGTAXBgNVBAMTEG1haWwuc2F2aW55a2gubnowggIiMA0G
CSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQDUTF0bC6zf0D27O1B5MR1JkLrZQV5T
yT35kEBWMvva8zYIEpv1DMAQXoxyJWzIr9ljQVYLpSV00lM6ioxF/jAyZ7KyNnLr
u2gI1ZcFnOeZcN4l4Ptd3gLktIcQJ0aWTNmapjSf9UgM+aQ02fr2j8NHhtxIFAgc
0s28W80VRPDLqf5mGsnG9r9EM3xBblUebmxqGGA5xY6E//jUJugaSKAjXSDP05SL
rpRlxWY1x6g9chZGigC4Bxa61NkZEEbxzRkr3cgBxoOfApLRncCv8W6YS0YWnUxY
IE0t54e5akUVRJ+ODrFqJyc1ETaDhfXHwsX87XPiMF1n3kP+K7Ls7ROb/eADJaXO
xGpeWJeM2P1D7HO7+Vjfa9s0p01sJzblZgQY6zZuhb4o86Kj+RLrxMT67twdqbUU
4g5+5jMUjQpFNpSOy6mGiYMXtdDqj6mJI91uo+hOeqU6pPE46qAHntmyf35PDZEq
ytfNATLij/j49bjeojauQhALSwHEZfVuni/GjVrKKo2CsAECecTyQOSSSQ8+M+/f
NhAxRKhrbkDO4i520yPR1KRROayLDoTCAbIeXnew3f0N+VAh9YwgzBew2F5CxfAO
GJ6JRvzROy1cjAkTV0kLZF5JjPxKe7lFcHDbEC5M3pwTcXnRnUtxu/VQs5Q10Egf
gP4IK6RHPOJlEwIDAQABo4ICZTCCAmEwDgYDVR0PAQH/BAQDAgWgMB0GA1UdJQQW
MBQGCCsGAQUFBwMBBggrBgEFBQcDAjAMBgNVHRMBAf8EAjAAMB0GA1UdDgQWBBSJ
OmrrEv45joU3GUSZ8Mi/fEgLZDAfBgNVHSMEGDAWgBSoSmpjBH3duubRObemRWXv
86jsoTBvBggrBgEFBQcBAQRjMGEwLgYIKwYBBQUHMAGGImh0dHA6Ly9vY3NwLmlu
dC14My5sZXRzZW5jcnlwdC5vcmcwLwYIKwYBBQUHMAKGI2h0dHA6Ly9jZXJ0Lmlu
dC14My5sZXRzZW5jcnlwdC5vcmcvMHAGA1UdEQRpMGeCEG1haWwuc2F2aW55a2gu
bnqCEm1pbmlvLmJydXRzb2Z0LmNvbYIYcG9zdGZpeGFkbWluLnNhdmlueWtoLm56
ghBzcGFtLnNhdmlueWtoLm56ghN3ZWJtYWlsLnNhdmlueWtoLm56MIH+BgNVHSAE
gfYwgfMwCAYGZ4EMAQIBMIHmBgsrBgEEAYLfEwEBATCB1jAmBggrBgEFBQcCARYa
aHR0cDovL2Nwcy5sZXRzZW5jcnlwdC5vcmcwgasGCCsGAQUFBwICMIGeDIGbVGhp
cyBDZXJ0aWZpY2F0ZSBtYXkgb25seSBiZSByZWxpZWQgdXBvbiBieSBSZWx5aW5n
IFBhcnRpZXMgYW5kIG9ubHkgaW4gYWNjb3JkYW5jZSB3aXRoIHRoZSBDZXJ0aWZp
Y2F0ZSBQb2xpY3kgZm91bmQgYXQgaHR0cHM6Ly9sZXRzZW5jcnlwdC5vcmcvcmVw
b3NpdG9yeS8wDQYJKoZIhvcNAQELBQADggEBAEc/BOS4evv//FJnMzUAmCGgJmCf
PLbteSiDzFnaPtt+FaM8ikokZiQqTmbc4FgYQj155td+FonTpgu8MRyDYsvxH+jK
yKXQnkFowwaJIZfTOGBXBy8PL62FZL0zwo4Oa9LY5z+qjnu6cXzZqIpRQFeQ4iIp
WTOOFDrIU9SaYDNX0nzgfJIoBYj2FUNX6frNCSX++RrFSxmheWI+H6KpUJ8uPeLP
PfaIiJleVgQHssoCjvc+K7+pdooqc6HTYcaluDOmUKnArOIGxgzlwxJbUEtWCJ/P
0NL4qaZzU5/VjLanOJBWCBb0YIOzzAlB2147uKkEEF6X/hyXaRxQx+0X7fE=
-----END CERTIFICATE-----

[_az]

@schoen Actually I seem to see this cert on censys: https://censys.io/certificates/d0faa80eb2a875c6b1159207f761d553504a821eb9709cca89421970e6de0b27

and it definitely came from CT:

 metadata.source	ct

[rg305]

For some reason it just tool longer than normal: https://crt.sh/?id=258885528

[cpu]

@schoen

@cpu, could you check on this if it’s still missing in the morning?

As @rg305 points out, it seems to be present now.

I believe there can often be lag between when Let's Encrypt submits a certificate to the certificate transparency logs and when it appears in crt.sh. In general we have no insight into why this is and inquiries about it should be directed to the crt.sh maintainers.

Like @_az found certificates can often be present in the logs that Let's Encrypt directly submits to before they are present in crt.sh and other log indexers.

Thanks for the all investigation & replies folks

[tialaramex]

There are two relevant delays involved.

One is, individual Certificate Transparency logs are often parallel systems, they need to periodically perform a step to synchronise things so that what they tell the world (about certificates they have been shown) is consistent. They are prohibited from ever saying anything inconsistent (e.g. saying once that they’ve logged a particular certificate and then later that they did not) but they have up to 24 hours (called the “Maximum Merge Delay”) to deliver a new consistent view after logging the certificate. Of course it doesn’t take 24 hours to do the mathematics, but this is the absolute maximum delay, it includes for example if they have a power failure or network interruption, they must recover and publish new consistent records in under 24 hours if they’d signed any SCTs before they lose power / network. The chosen MMD is a compromise, a longer MMD would allow logs to recover from more serious failures, e.g. the whole of Japan is destroyed by a rampaging lizard monster, whereas a shorter one would detect problems more quickly.

The second is that crt.sh itself [run by Comodo] needs to suck in the millions of certificates from the dozens of active logs, it is able to mostly keep up, but it isn’t live. https://crt.sh/monitored-logs shows crt.sh’s view of its own backlog, the difference between the number of certificates it knows it should have record of, and the number it already has downloaded from the logs. It is normal for there to be a significant backlog for the very popular logs used by Let’s Encrypt.

[AndrewSav]

Okay thank you everyone. So the recipe is “wait for 24 hours and if it’s not up there panic, lizard monsters are incoming”. Gotcha. I think we are done here.