Creating a webservice for analysis of common problems


#1

Hi,

during the last few weeks following the threads here, I realized that most of the problems people have during authorization are encountered regularly. I would like to propose some kind of a web service, similar to unboundtest.com, where one could enter a list of domain names that will be checked against common misconfiguration:

  • non-resolvable domain names
  • mixed ip address delegations within one set of domain names
  • non-reachable port 80/443
  • differences in IPv4 and IPv6 access

Of course, this list is not complete - please add your comments here. After having identified the most common problems, we could argue about where to host that service and who wants to contribute.

Regards!


Software for large-scale integrations?
#2

That sounds like a great idea!

Some other things it could check…

  • Known issues from https://letsencrypt.status.io/
  • Rate limit status for the domains
  • Redirects missing a trailing slash
  • HTTP or SSH listening on port 443
  • Cloudflare getting in the way

#3

I also like this a lot and was at one time thinking of proposing something similar. A few more items:

  • CAA lookup failures
  • issuance expressly prohibited by CAA
  • subdomains of major cloud providers (issuance prohibited by policy)
  • non-FQDNs
  • known dynamic DNS providers not on the PSL
  • names forbidden by policy for another reason (e.g. the [a-z][a-z]-- regular expression related to IDNs)
  • entered an IDN U-label instead of the A-label form
  • just telling people that they have IPv6 AAAA records at all, if there is no existing TCP listener (because a later-created ephemeral listener might not succeed in listening in IPv6)
  • firewall potentially blocking inbound connections (nmap’s “filtered” state rather than “closed”; existing listener accepts connections but then disconnects immediately)
  • evidence of /.well-known/acme-challenge being routed to a web app rather than to the filesystem?
  • DNS server responds badly to mixed-case queries
  • other CDNs (apart from CloudFlare)

A tricky one because of the high false positive risk:

  • you aren’t running your ACME client on the server pointed to by the A, AAAA, or CNAME records for some requested domain

#4

The trouble with that is that some widely used configurations serve a file from the filesystem if it exists, but route the request to an app if it doesn’t…


#5

Another thing to test for, as a specific sub-case of rate limiting problems:

  • a certificate has been renewed repeatedly with nearly constant, short (< 30 days) time spans between renewals, indicating a cron job incorrectly uses --renew-by-default or --force-renewal

#6

Should have a nice FAQ with help for the different problem types.
Maybe even with tutorials based on the detected setup (IIS/Apache/PHP/OS from Server header, Cloudflare/GoogleDNS/etc from IP)


#7

Also DNSSEC failures, which are often not apparent because the resolvers that many people use disregard those issues.

Great idea, can check this using CT logs.

I’d love to contribute to this project. If anybody is already working on this, please let us know!


#8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.


#9

#10

I’d like to invite people to take a look at letsdebug.net / https://github.com/letsdebug/letsdebug which @eggsampler and I have been working on, implementing the most common issues suggested here.

Feedback about making it more useful would be great.


#11

I just got a false negative that I wanted to point out with

https://letsdebug.net/scammell.co.uk/899

It may be hard to reproduce this because I think the user is likely to fix the problem, but this site had different content in IPv4 and IPv6 and letsdebug didn’t detect the discrepancy. The IPv4 version returned a redirect to HTTPS, where the HTTPS site had an expired certificate serving useful site content. The IPv6 version immediately returned an nginx default page at the top level, or, for any specific file, a 404 error. Any ideas about why letsdebug wouldn’t detect this discrepancy?

(I decided to check this out of curiosity after diagnosing the problem myself with curl, and in the hope of soon making letsdebug a routine part of my investigation of users’ issuance problems.)


#12

Thanks. The discrepancy checker wasn’t looking at anything except for the final (after all redirects) response. It now additionally tracks the initial status code as well as the number of redirects.

The verbosity of the HTTP trace information is increased significantly as well.

https://letsdebug.net/scammell.co.uk/923?debug=y


#13

That’s terrific! Thanks.

I’m going to try to make a resolution to try letsdebug for every failed issuance support request that I deal with for the next week and see how many of them it can resolve. (I have to encourage myself to do this deliberately even if I think I already know the answer, so as to get a better sense of what conditions it can already detect well.)


#14

With crt.sh not deduplicating certs/precerts on their search interface, it’s recently become a bit difficult to figure out domain-based rate limits. @sahsanu’s lectl has a great (terminal) user interface and I wanted to take its ideas and make it available on a web interface (asking people to download scripts to do something seems like a big ask to me!).

I’ve built https://tools.letsdebug.net/cert-search which is essentially a static frontend to crt.sh’s database. Rather than targeting power users (who already understand everything), I tried to maximize understanding of where a person stands with their domains’ rate limits at any moment (so it should understand the PSL, and what rate limits exist, etc).

You can see an example at e.g.: https://tools.letsdebug.net/cert-search?m=domain&q=tusysadmin.com&d=168 and it has a handy link to copy a markdown summary to this forum:

Rate Limit Current Status Domain
20 Certificates per Registered Domain per week OK (15 / 20 this week.) tusysadmin.com
5 Duplicate Certificates per week Limit exceeded. Next issuable at 2018-06-22T17:30:50.000Z cloud.tusysadmin.com, demo1.tusysadmin.com, demo2.tusysadmin.com, demo3.tusysadmin.com, demo4.tusysadmin.com, demo5.tusysadmin.com, estafasfb.com, ing-gt.no-ip.org, tusysadmin.com, www.estafasfb.com, www.tusysadmin.com
5 Duplicate Certificates per week Limit exceeded. Next issuable at 2018-06-22T18:31:46.000Z cloud.tusysadmin.com, demo1.tusysadmin.com, demo2.tusysadmin.com, demo3.tusysadmin.com, estafasfb.com, ing-gt.no-ip.org, tusysadmin.com, www.estafasfb.com, www.tusysadmin.com
5 Duplicate Certificates per week Limit exceeded. Next issuable at 2018-06-22T17:30:23.000Z demo1.tusysadmin.com, demo2.tusysadmin.com, demo3.tusysadmin.com, estafasfb.com, ing-gt.no-ip.org, tusysadmin.com, www.estafasfb.com, www.tusysadmin.com

Summary generated at https://tools.letsdebug.net/cert-search?m=domain&q=tusysadmin.com&d=168 .

Again, any feedback would be great.


#15

Just tried letsdebug for a domain where I’m getting policy forbids issuing for: "xyå.example.com - what was the criteria for valid IDNs? It’s a question that comes up regularly with Swedish web hosts (for example).

I’m assuming that you just can’t use variants of a/o such as åäö but is that defined somewhere? if that’s the case I just need to warn the users and it would be a good addition to letsdebug.


#16

The domains need to be provided to the CA as Punycoded ASCII.


#17

Ah, thanks, I think I know what’s going on . My CSR subject is unicode, oops (validation already completed with punycode versions).


#18

At the moment it automatically applies the IDNA ToASCII algorithm to the submitted domain.

I could make it reject the name and link the user to the proper form of the name, if you think that would be helpful to users. Or maybe add a warning on the subsequent screen.


#19

Thanks @_az I think this was just my bug so you shouldn’t have to update anything. I’d refactored some code and missed out my conversion to ASCII in the CSR, so it’s all good, I just didn’t notice my tests were failing :slight_smile:


#20

@_az, a new suggestion for something else to check: with DNS-01 _acme-challenge.example.com doesn’t exist but _acme-challenge.example.com.example.com does. E.g.

I realize that letsdebug doesn’t reject DNS-01 configurations where the _acme-challenge TXT record hasn’t been created yet, but nonetheless it could generate a useful warning when the corresponding record with a duplicated base domain name does exist!