Blacklist documentation or testing?


#1

I’ve been investigating moving to Let’s Encrypt for a business that hosts thousands of domains. One concern I’m trying to address is that some number of those domains may be unable to get certificates due to the blacklist.
Is there any documentation of the blacklist/whitelist that I can use to determine how many of these domains would fail? Short of attempting to issue certificates for all of the, are there any other mechanisms for testing against the blacklist?


#2

Unfortunately the blacklist is not public, nor is it really possible to make it so because it works with permutations and such.

However, the blacklist is checked at the beginning of the authorization process so you do you do not need to perform the steps necessary for authorization to succeed or actually get a certificate to check against it.

The easiest way I can think of to check would be to run certbot certonly --standalone -d example1.com -d example2.com [...] from an unrelated computer with an inbound firewall and no web server running. Grep the output for blacklist to separate those failures from the expected ones.


#3

Thanks Patches, that seems like a reasonable approach. Hopefully I won’t hit any rate limiter if I try this with a 10,000+ domains :slight_smile:


#4

Actually, this didn’t work so well. After some number of requests, I’m getting this:
Error: urn:acme:error:rateLimited :: There were too many requests of a given type :: Error creating new authz :: too many currently pending authorizations
It looks like it gets far enough into the authorization process to create a “pending authorization”. I’m using the tls challenge, and my client is failing to bind to port 443 (not running as root) intentionally, because all I care about is whether or not I get a “Policy forbids issuing for name” message.
I’m not sure how to move forward with this. Is there some way to cancel the pending authorizations?


#5

Hmm, not sure why your authorizations are getting stuck in the pending state and not failing outright. Do you have a log for one of them?

You can clear pending authorizations, but it’s a manual process. It would probably be easier for you to just register a new account key, since this rate limit is per account key. (That is also rate limited, but you could test 150,000 domains every three hours between the two rate limits.)


#6

Is it possible that the client creates authzs for each of the names that it’s requesting, tries to validate just one of them, gets an ACME error indicating that the challenge failed, and then simply gives up because the certificate couldn’t be obtained, leaving the other authzs in pending state? I think that’s Certbot’s behavior and would tend to explain the phenomenon of authz leakage.

This might, weirdly, be an unintended use case for --allow-subset-of-names in Certbot if this is the underlying reason, because then it might actually fail challenges for each name, avoiding the authz leakage side-effect…!


#7

The script I was running passes only a single domain name in each call. I have tens of thousands to verify, and I couldn’t find a simple way to script it to pass multiple domains to a single call, and be able to verify which one was blacklisted.

For clearing the pending ones: “If you do not have logs containing the relevant authorization URLs, you need to wait for the rate limit to expire.” Unfortunately the certbot seems to keep only the last 1000 log files, and I ran 1437 through before noticing it was failing, so I don’t have the logs for the ones that actually got into the pending state, rather than being rejected.

I can create a new account key, but I’ll keep running into this problem unless I rotate it every few (how many?) domains. Is there some kind of flag I can pass to certbot to tell it to not actually create a pending authorization?

For reference, this is the script I was running to check against the blacklist:

function issue() {
	certbot certonly --standalone --test-cert \
		--config-dir config --work-dir work --logs-dir logs \
		--tls-sni-01-port 443 \
		-d "$1"
}

cat cns | while read cn; do
	echo "Testing $cn"
	if ( issue "$cn" 2>&1 | grep -q "Policy forbids issuing for name" ); then
		echo "Result: $cn Blacklisted"
	else
		echo "Result: $cn OK"
	fi
done

#8

The relevant rate limits are “300 Pending Authorizations” and “500 Accounts per IP Address per 3 hours”. (There’s also the “Failed Validation limit of 5 failures per account, per hostname, per hour” that should be a non-issue.)

Note that --test-cert uses the “staging” Let’s Encrypt infrastructure that does not issue real certificates and has higher rate limits, so your mileage may vary.

You need to create an authorization for the blacklist to be checked. The oddity is that some of them are not failing when they should be.


#9

Presumably the authz are never submitted for validation because certbot fails to bind to port 443 once it gets to the challenge stage and exits. It would be weird for certbot to try to submit the challenge despite knowing it would fail. I think it’s okay for certbot to behave that way under normal circumstances, given that the pending authz rate limit is relatively high.

For this particular use-case, I’d look into using a low-level ACME library rather than a full client. Rather than submitting challenges for validation (which would take a while for each domain), you could simply disable the corresponding authorization right after you’ve checked whether the domain is blacklisted. These libraries exist for most popular programming languages, though you’d have to check if they implement disabling authorizations. An example for go would be lego’s ACME library. Aside from the registration boilerplate code from the README, you’d probably need getChallenges and disableAuthz.


#10

@pfg, do you think this is a protocol or Boulder issue because the client is told early that the authorization for that name will fail and hence doesn’t submit a challenge for it, yet perhaps Boulder should then destroy the authz itself?

I was thinking that the specific method of testing on a machine that is not actually entitled to these certificates (which I think is what @lewisd is doing, if I understood correctly) causes the client to give up after the first failed challenge, even though challenges for the other authzs haven’t been attempted (and therefore they are still pending).


#11

I don’t think the domain being blacklisted is related to this at all. From my reading of boulder’s code, it doesn’t persist authorizations for blacklisted domains - that is, the control flow is roughly if (domainNotBlacklisted() && someOtherChecks()) { insertAuthz() }.

The pending authorizations (presumably) exist because certbot exits once it realizes it’s unable to bind to port 443, and that would happen before it submits the challenge (or disables it, if that’s supported).


#12

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.