Validation server is blacklisted

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: wikiworld.litts.net

I ran this command:
certbot-auto renew --quiet --no-self-upgrade …

letsencrypt’s validation server, 66.133.109.36 is on the sbl-xbl.spamhaus.org blacklist. Thus, it can not connect to my webserver (for .well-known.acme-challenge./…) My webserver rejects all black-listed sources.

The listing comes from the CBL details are at
https://www.abuseat.org/lookup.cgi?ip=66.133.109.36

The short form, according to the listing is:

This IP address is infected with, or is NATting for a machine infected with the “nymaim” malicious botnet.

“nymaim” is also known as “Gamarue”.

Once the issue is resolved, the admin for this server needs to visit the url above, and https://www.spamhaus.org/query/ip/66.133.109.36 to get it removed from the black list.

Thanks.

2

See this advice:

Since Spamhaus chooses to continually re-list the validation servers, it's up to you to either stop using the blacklists, whitelist the URL, or choose a validation method (e.g. DNS-01) that avoids them.

Even if they requested to be unlisted (again), what's the point? The ACME protocol by definition requires an "open redirector" to be run by the CA. I could setup a 302 redirect from my domain to http://{spamhaus-honeypot}/{exploit_url} and get them re-listed within hours.

The problem requires a different solution.

2 Likes
3

Well, it’s kinda hard to whitelist the validator IPs when LE states emphatically and often that they won’t provide a list because they aren’t stable. (It’s even harder because the apache module that I use doesn’t have a whitelist feature - but that’s a SMOP.)

Maybe I’ve just been lucky - I’ve been using le and this config for almost 2 years, and this is the first time I’ve run into this.

The blacklists do stop many, many harrassing actors at minimal cost to my servers.

I don’t use dns-01 because that would require placing the DNS update credentials on an external-facing system.

If I could ask what the validation servers are “now” (say, a stable DNS RRSET for ‘validation-servers.letsencrypt.org’ containing all the A & AAAA), I could add them to a (to be implemented) whitelist before (and remove them after) certbot runs. Providing this would not seem to reduce LE’s flexibility to move/add/change their validation servers.

Since that doesn’t seem to be possible, I guess I’ll have to come up with an even uglier hack for this circumstance.

4

For most software, whitelisting the .well-known/acme-challenge/ path (for all IPs) is probably the simplest, most effective workaround at the moment.

A less simple, but maybe more elegant, option could be to delegate just _acme-challenge. DNS records to another nameserver (like acme-dns), and then give your ACME client privileges to push dynamic updates to only that special nameserver. That might help mitigate the risk you’re concerned about with dns-01.

2 Likes
5

I took the path of least resistance, and modified the apache module to “whitelist” /.well-known/acme-challenge/*

It’s not as bad as it could be since the acme-challenge directory only exists while certbot is active.

Creating a zone cut & delegating _acme-challenge.example.net. to a dedicated nameserver is a pretty costly approach - the maintenance effort for a nameserver is non-trivial - and this one would be used for a few seconds every 2 months. Technically, one would be required to have at least TWO, geographically dispersed servers for the zone. Perhaps if one has a LARGE number of domains/certificate renewals it might be worth the effort.

I’ll also note that dynamically updating DNS (for dns-01) can be a challenge if you use external nameserver services. The larger suppliers provide APIs, but rarely support DNS UPDATE+TSIG, instead offering some proprietary mechanism.If you’re lucky, something like JSON over http. The less enlightened ones provide a webform interface, nowadays with a Captcha, making automation painfult to impossible. They rarely let you specify a zone cut / different credentials for subdomains. I do use dns-01 for some systems - and have had to write interfaces to some of the proprietary mechanisms where I don’t use my own servers. Not a lot of fun. Which is why the .well-known path is sometimes the better choice.

It would be helpful if you would publish the list of validation servers in DNS as I suggested above, since this would allow whitelisting by IP for all kinds of security screens (firewalls, webservers, intrusion alarms, etc).

As I noted, publishing a list in DNS doesn’t prevent you from adding, deleting, or reconfiguring the validation servers. Just add the record for a new server a TTL or so before activating it, and the world will let it work. And don’t release the IP of a discontinued server (as if anyone gives up IPv4 addresses these days!) until a TTL has passed. Even if you don’t precisely sync the DNS records with the servers, in the worst case a client will block renewal – but a subsequent retry will work.

There is no practical limit to the number of address records that can be associated with a hostname - if you need 100 servers, just add the records. (Though an anycast address might be a better choice…) And if the zone is DNSSEC signed, the appearance of an IP address in (say) validation-servers.letsencrypt.org would be a pretty strong guarantee that LE is behind the probe.

Depending on whether your ISP(s) delegate reverse (PTR) records to you or make it easy to update them, setting them up might be more painful for you. While this would make life simplest for clients (look up the inbound IP, match the name & check signature), it’s not difficult to create the reverse mappings for this special case at the clients. It’s not uncommon for hosts to be missing PTR records, nor for ISPs not to sign their reverse zones, so coping is a solved problem. Just having the forward records would be a step forward.

In any case, for now I have a workaround for my situation.

Thanks for your consideration.

6

There is no such requirement, and given the limited use of such a server, I'd question whether it's even particularly advisable.

...and 50+ of those proprietary mechanisms are supported out-of-the-box by acme.sh, to name just one ACME client. I don't know which (if any) of them support setting different credentials by subdomain (Cloudflare, which I use, doesn't), though.

7

Just for the record, and to avoid confusing others: the requirement is quite real and has existed since the dawn of the DNS. Starting with RFC1034:

A given zone will be available from several name servers to insure its availability in spite of host or communication link failure. By administrative fiat, we require every zone to be available on at least two servers, and many zones have more redundancy than that.

See also Technical requirements for authoritative name servers

Minimum number of name servers There must be at least two NS records listed in a delegation, and the hosts must not resolve to the same IP address. The name servers must be in at least two topologically separate networks. A network is defined as an origin autonomous system in the BGP routing table.

And RFC2182:

The DNS specification and domain name registration rules require at least two servers for every zone. That is, usually, the primary and one secondary. … with the others not geographically or topologically close to the site, or each other.

It is true that no technical constraint would prevent a zone from being served by a single authoritative nameserver. Or by two that are not geographically dispersed. And some people, especially small operators, flaunt the rule. However, the parent zone operator would be within his rights to delete a non-conforming zone. As would an auditor be justified in flagging it.

The point is moot, for me at least. This approach is too expensive for the problem at hand.

I don’t know which (if any) of them support setting different credentials by subdomain

Neither do I. And changing service providers to solve this issue would, again, be a very expensive (in terms of time & side effects) solution to this problem.

Again, a moot point - I run my own servers for most of the domains that I administer. For the others, I have solutions - it's nice to know that there are more out-of-the-box solutions today than there were when I had to roll my own.

I still believe that publishing the validation servers' addresses in the DNS would solve a number of problems.

But as I have worked-around my immediate issue, I'm on to other things.

8

I sit corrected, and thanks for the pointers. It appears it's a requirement that's honored as much in the breach as in the observance, but that's a separate issue.

If that requirement is ignored, acme-dns (already linked up-thread) is quite simple to install and administer, and would completely avoid the whitelist requirement. As is nearly often the case, there's more than one way to skin the cat.

9

I don't think that's what @JamesLE meant in his message (despite the wording). You don't need an actual DNS delegation for the sub-zone. You just need to add a CNAME for the record that points to a record in another zone that you're comfortable giving the ACME client credentials to update.

So _acme-challenge.example.net points to _acme-challenge.less-critical.net which is where the TXT record is actually created. Then LE follows the CNAME and checks the TXT record at the alternate location and you're good to go. Many ACME clients support this notion of alias'ing the validation record right out of the box.

If you don't want to hassle hosting something like acme-dns yourself, there are plenty of free DNS providers out there that have support in various ACME clients.

2 Likes
10

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.