Crt.sh has stopped - int32 problem, new database may start next week

The tool https://crt.sh/ is a certificate log monitor. So it’s visible how many certificates are created with a domain name.

But: There are some problems:

Log monitor stopped, and Upcoming Changes

https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/crtsh/DM8SI-qsE8E/J_ndSkroBgAJ

crt.sh startet 2015-06, before Letsencrypt.

The idea that “2 billion certificates ought to be enough for any PKI” (to misquote Bill Gates) didn’t seem that daft to me back when crt.sh was born (June 2015). Let’s Encrypt had not yet launched, there were only a few million active certificates in the WebPKI (https://blog.cloudflare.com/introducing-universal-ssl/), and certificate lifetimes of >=1yr were the norm. However…

Yesterday, there was an int32-overflow.

Now a new database is startet.

The new database is currently busy importing data and building indexes. I’m hoping it’ll be production-ready and available by early next week (caveat: I am rubbish at predicting ETAs). I’ll update this thread with more details when I have them.

Perhaps next week … the tool may work again.

9 Likes

For the admins out there with MySQL/MariaDB databases, you can monitor your own databases with https://github.com/prometheus/mysqld_exporter and ensure you’re collecting the autoincrement stat.

I’m not sure if the Postgres version of the exporter https://github.com/wrouesnel/postgres_exporter exports autoincrement stats, but the tool is there.

6 Likes

Any more news on when it will be back up?

1 Like

Not really. In the Google group there is an update.

A test version runs under

91.199.212.73 crt.sh

so you can change your hosts file to use that new ip address. My tool “check your website” uses two CT monitors, Certspotter and crt.sh, so Certspotter shows the new certificates.

1 Like

As so far, it was not fixed.

1 Like

As there seems to be no progress or information about estimated recovery of this tool, are there any alternatives for crt.sh? I am aware of https://transparencyreport.google.com/, would you say this is comparable?

1 Like

It’s possible, but there is the same problem: That lists pre- and leaf certificates. So one certificate produces two entries.

There are two other tools - https://tools.letsdebug.net/cert-search from @_az and my own tool https://check-your-website.server-daten.de/ . Both remove duplicates, so one certificate -> one entry.

First uses only crt.sh as source, so:

cert-search is currently inoperable due to ongoing maintenance on the crt.sh:5432 database . Apologies for the inconvenience.

My own tool uses Certspotter and crt.sh. Certspotter lists only active certificates, but the old crt.sh lists all older (max. ~~ 2019-11-*).

1 Like

Thank You, I just checked out your tool. It is very informative and lists a lot of possible problems and checks for best practices. I have two questions about it.

  • How is it possible to display all published certificates for a given domain (and it’s subdomains), like it was possible with crt.sh or the google-tool? I couldn’t achieve this here, I have to explicitly give a specific FQDN, which is then checked, and transparency log for this name is listed.
  • Why is the name automatically prepended with “www” and also checked? This breaks the functionality for me basically.
2 Likes

That’s not possible. And I don’t want to implement it.

That’s one of the most important functions.

I’ve started the tool 2018-10 because of the questions in this forum. One main problem: The first certificate. So all 4 urls (non www + www, http and https) with redirects are checked. If there are three correct redirects and one destination -> Grade B. Two destinations (https + non-www and www) -> Grade C.

A general problem: The webmaster doesn’t see a problem, because he uses one preferred version and the browser has cached the redirect. A user uses the non-preferred version, the certificate has only one domain name (or the non-preferred version has a wrong certificate): The new user has that problem, the webmaster has no problem. Running that tool -> oh, there is a Grade N.

So the first part: Url-Checks, Comments and a small ranking system, later Connections + certificates.

Crt.sh was added 2019-03, Certspotter 2019-05, after a lot of other features.

It’s not the idea to replace a CT-monitor. But checking certificates is helpful to see, if a user has already created certificates or has hitted a limit.

2 Likes

CertSpotter can display all nonexpired certificates for a domain. (In JSON form.)

https://sslmate.com/certspotter/api/

Edit: There are also other CT monitoring websites. Censys likely has a syntax for any kind of search, though they don’t seem to monitor all logs. I don’t know what features Facebook has.

2 Likes

I understand and respect. In this case, your tool is no replacement for crt.sh - because that’s what I used it for. Monitoring what certificates are being published for a certain domain, and providing true certificate transparency in the process.

But how to proceed when there is no www-counterpart of a given site?

2 Likes

Then try to use the new crt.sh. There is a description with an own host entry and the new ip address (didn’t check it). If you have one or two domains, my tool may be enough. With 15 or more domains - it’s painful, too much other things.

That’s not a problem. Read the output of the self check (someone checked the subdomain today) - https://check-your-website.server-daten.de/?q=check-your-website.server-daten.de#ct-logs

The checked domain / subdomain -> used to check the CT-monitors. If no www exists -> no problem.

PS: May be that

works. Check the output of google.com (there are sometimes uses who test google, facebook etc.). There are a lot of certificates with subdomains - meierf.zrh.corp.google.com, tstldap3.ldaps.corp.google.com.

But then the output is big, very big -> and slow.

A specialized solution like Censys may be better.

1 Like

It’s back! :tada:

7 Likes

Now the own code is updated.

One error found:

First, the Sql-Code (used via PostGreSql-ODBC connection) was terrible slow.

Now I’ve found the reason:

The new crt.sh has ipv4 and ipv6, that’s good.

Host T IP-Address is auth. ∑ Queries ∑ Timeout
crt.sh A 91.199.212.73 Bradford/England/United Kingdom (GB) - Sectigo CA Hostname: no-dns-yet.ccanet.co.uk yes 2 0
AAAA 2a0e:ac00:c7:d449::5bc7:d449 Bradford/England/United Kingdom (GB) - Comodo CA Ltd yes
www.crt.sh C crt.sh yes 1 0
A 91.199.212.73 Bradford/England/United Kingdom (GB) - Sectigo CA Hostname: no-dns-yet.ccanet.co.uk yes
AAAA 2a0e:ac00:c7:d449::5bc7:d449 Bradford/England/United Kingdom (GB) - Comodo CA Ltd yes

Ipv6 works via Url-check.

But a connection port 5432 isn’t possible, timeout after 15 seconds, my server prefers ipv6.

Changed, so ipv4 is used -> now it works again.

2 Likes