The DNS resolution issues were a bit ironic – Let’s Encrypt’s secondary validation servers are currently run in AWS. (Not in GovCloud, of course.) Even if your network was inaccessible, you’d think they’d be able to successfully resolve your DNS records using your DNS server in AWS.
The GovCloud IP is in Great Britain [not global/anycast]?
Not sure if that plays any part in the DNS disruption.
Fortunately, my organization’s security staff are still awake, and found that some of the above IP addresses had been blocked for probing port 80. I do not know all the details yet, but they temporarily whitelisted the IP addresses in question, and I was able to successfully renew the cert.
I am trying to get more information on exactly what triggered the block at the border, which I hope to have tomorrow.
…and I will probably attempt at least a dry run of a DNS validation on one of our hosts so that I will know how to do it if this crops up again. (Perhaps that will expose more details of the DNS failures that were reported this afternoon.)
Security never sleeps!
Sweet; congrats!
I'm also interested, please copy me with whatever they find - thanks.
From what I’ve been told, it sounds like our organization’s IDS saw queries on port 80, from IPs whose PTRs were not to letsencrypt.org names, to enough internal hosts at our site, that it thought the IPs were an attack and triggered an automated block. They have said they will think about how to make the IDS less sensitive to this sort of blocking in the future.
I think it would be useful if the debug logs included which IPs failed to reach the server for validation, but it sounds like LE may be reluctant to do that.
We have another certificate which was expiring soon, and I was able to successfully renew it earlier today. So it seems like the issue is resolved, at least short term. At least I know what to look for if it does recur, and my security group is more aware of the details.
1 Like
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.