Certificate transparency error on Chrome 74 after extending certificate

Help
1

After extending a certificate today, one of my users on Windows, Chrome 74, started seeing a certificate transparency required issue ERR_CERTIFICATE_TRANSPARENCY_REQUIRED. They do not see it on their Ubuntu or Mac machines.

I cannot see it on any of my browsers or devices, and it hasn’t been reported by other users yet.

image
image.png854×551 18.6 KB

Why would this have happened? How can I resolve it?

My domain is: app.developerhub.io

I ran this command: sudo certbot certonly --manual -d *.developerhub.io -d {all-the-domains} --agree-tos --no-bootstrap --manual-public-ip-logging-ok --server https://acme-v02.api.letsencrypt.org/directory

It produced this output:

I extended the certificate and set up all the challenges then

Waiting for verification…
Resetting dropped connection: acme-v02.api.letsencrypt.org
Cleaning up challenges

IMPORTANT NOTES:

  • Congratulations! Your certificate and chain have been saved at:
    /etc/letsencrypt/live/developerhub.io-0003/fullchain.pem
    Your key file has been saved at:
    /etc/letsencrypt/live/developerhub.io-0003/privkey.pem
    Your cert will expire on 2019-09-19. To obtain a new or tweaked
    version of this certificate in the future, simply run certbot
    again. To non-interactively renew all of your certificates, run
    “certbot renew”

  • If you like Certbot, please consider supporting our work by:

    Donating to ISRG / Let’s Encrypt: https://letsencrypt.org/donate
    Donating to EFF: https://eff.org/donate-le

My web server is (include version): nginx version: nginx/1.10.3 (Ubuntu)

The operating system my web server runs on is (include version): Ubuntu 16.04.4 LTS

My hosting provider, if applicable, is: Amazon Route53

I can login to a root shell on my machine (yes or no, or I don’t know): Yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot): certbot 0.28.0

2

can I see the error with opened advenced box?
I can open your site on chrome 75.0.3770.100 without error. can you update your chrome?

3

Interesting. A few things could happen, and it’s definitely important for us to figure out what went wrong here because this sucks as an experience.

  1. Let’s Encrypt might have somehow issued you a certificate which lacks proof it was logged. This would cause NET:ERR_CERTIFICATE_TRANSPARENCY_REQUIRED since Chrome requires proof. Let’s Encrypt policy says they will obtain proof and fasten it inside your certificate, as far as I know this is mandatory (some other CAs will let you opt out if you understand the consequences)

1a. Let’s Encrypt might somehow not even have logged the certificate. This would be a big problem, unless there’s some policy change I wasn’t aware of (and nor were you apparently) letting you opt out at Let’s Encrypt, and even then it’s a problem if you opted out without understanding what that would do.

  1. The certificate is fine, and has proof (you can post it here – certificates are public documents and it’s safe to show them to anyone, but as a new user it may be difficult to do that, you might be able to post a screenshot? The important bit is labelled an “SCT” there should be two or more of those inside the certificate, it’s your proof it was logged that Chrome is saying it didn’t see) – but for some reason Chrome didn’t recognise this proof. That could be a Chrome bug, so still worth us knowing about

  2. The user’s browser was actually attacked somehow, and shown a bogus certificate, yours is fine, and they were saved by the error from Chrome telling them this certificate is bogus. This is very unlikely but it’s not impossible, if it turns out to be what happened you should conduct an internal investigation to understand who attacked you.

1 Like
4

It seems so

openssl s_client -showcerts -connect app.developerhub.io:443 -ct | grep SCT
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN = *.developerhub.io
verify return:1
SCTs present (2)
SCT validation status: valid
SCT validation status: valid

1 Like
5

Thank you for your replies.

The user has reported half an hour later than the error does not show anymore. Thus, I cannot get more detailed logs or screenshots anymore. May I mention that they did refresh multiple times when the error was happening, and it was persisting.

Would it help to send anything from my log file at /var/log/letsencrypt/letsencrypt.log? From an amateur look, it seems all requests were a success, nothing out of the norm.

How can I help?

It is such a bad experience for a new customer to face security problems.

6

Hi @zed

there was an older topic with the same problem.

If a certificate is very new, Chrome sometimes doesn’t see the CT log -> and reports that error.

7

Sorry, this won't help. The certificate your website is serving is fine and includes the required SCTs (one from Cloudflare's Nimbus log (2019 shard), one from Google's Icarus log).

Unfortunately it will likely be impossible to figure this out without more information about the affected system and being able to reproduce the problem.

I don't think this is the case here. I think what you're referring to is old browsers that don't yet trust newer CT logs whose SCTs are being depended on by the cert. Both of the logs that provided these SCTs should be available in Chrome 74 (afaict).

@zed Is the affected user on a corporate network with some sort of network filtering equipment? Do they use any Antivirus products on their machine that may have resulted in a different certificate being presented to Chrome than the one your webserver is returning?

8

I'd be hesitant to raise this possibility so early in the troubleshooting process. I suspect it will confuse users and should never be the root explanation (it definitely wasn't in this case). While not impossible, it would be a significant process failure.

9

Hrrmf. That's the catch-all. It happens often yes, but as a naive Letsencrypt user I could imagine other possibilities:
stupid idea 1
could the database update at the certificate transparency logs take some time ? AFAIK there are not transactional database a la Postgres, maybe there is a batch processing that only runs at some time interval.
From

Periodically, perhaps once an hour, a log server appends all of its newly acquired certificates to the log

stupid idea 2
could the user system have a time problem ?
SCT validation status: valid
Signed Certificate Timestamp:
Version : v1 (0x0)
Log : Cloudflare Nimbus2019 Log
Log ID : 74:7E:DA:83:31:AD:33:10:91:21:9C:CE:25:4F:42:70:
C2:BF:FD:5E:42:20:08:C6:37:35:79:E6:10:7B:CC:56
Timestamp : Jun 21 11:23:03.569 2019 GMT
so a log has a timestamp, what if for the browser the timestamp is in the future ? it may be considered not valid.

10

Chrome doesn't yet verify that a certificate is actually included in a log. It's only verifying the presence of the SCTs in the Certificate (or delivered through another supported mechanism). The logs incorporating or not incorporating the certificates should have no bearing on the error observed.

As a side note, the SCTs in the certificate being served presently are from Jun 7 (the date of issuance). The maximum merge delay (MMD) of the Nimbus and Icarus logs is 24 hours. If the logs hadn't incorporated these certificates yet they will have broken their MMD and be subject to disqualification. Similar to other lines of discussion this should be a rare event and would indicate significant process failures by the log operators.

@gpatel-fr That's a possibility, good idea!
@zed Do you know if the user's clock was correctly set?

11

Ah indeed, I should have understood that from the time Openssl is taking to verify it. Thanks.

12

Do you know if the user’s clock was correctly set?

I assume so, since it worked fine on his Firefox browser.
The user is investigating his network. I hope he'll come back to me with details. But he is surely using an Antivirus since he's using windows.

1 Like
13

Firefox share now the browser engine with Chromium, not necessarily all the code.

14

Firefox still uses Gecko for browser engine. I think edge change to share broser engine with Chrome

1 Like
15

I appreciate that this is your baby and so it seems to you like we shouldn't start by worrying that the problem lies there, but equally Chrome is a popular web browser and it seems as though we ought to see this reproducibly elsewhere if they've got a general problem with getting this wrong, as an outsider I prefer to list all the possibilities without prejudice.

If there's a general problem where Chrome clients in a normal state (e.g. I'd count a Chome install with a clock that's wrong by 8 minutes as "normal" but maybe not one that still thinks this is May 2018) mistakenly report legit certificates as lacking SCTs then it's valuable for Let's Encrypt to diagnose that, point it out to Google and if they can't get a satisfactory resolution warn users about this problem with Chrome and start talking through counter-measures.

The reality of low rates of true positives in Web PKI defensive stances is that even a fairly modest rate of false positives drowns it out and destroys the good work done, I'm happy to have that argument with Google people but it's much easier for Let's Encrypt to put together the data underpinning it.

As you rightly observe, Chrome doesn't do an inclusion check (and indeed doing so independently without a big cost to privacy is hard) so there isn't actually a particularly good reason it can't trust new SCTs even if it lacks good external network access. If the bug is theirs, they need to get on top of it.

16

Chrome does, or did at some point, for some subset of users behind a CertificateTransparencyLogAuditing feature flag, perform an inclusion check:

Chrome may check that an SCT has been honoured by the CT log that issued it, i.e. that the corresponding certificate is indeed published in that CT log. This is so that a CT log cannot issue an SCT but then never publish the certificate, thereby hiding its existence. Chrome does this by sending a specially-crafted DNS query that requests an inclusion proof from the log. Using DNS allows the user to remain anonymous from the CT log's perspective and enables caching of inclusion proofs. If the log cannot provide a proof, or the proof cannot be verified, then Chrome learns that the log has misbehaved.

1 Like
17

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.