One of my production websites is on an Ubuntu virtual server where I am the root admin and primary developer. However, I do not administer the domain record, DNS, or firewall.
This server was installed January 2023 and ran smoothly since. I recently received an email about the TLS certificate expiring in less than 30 days. I logged in to check this today and found an expiration date of 2024-04-30, which is next week.
I saw the pinned announcement here
I confirmed there is a secondary validation timeout in the logs. But I'm not seeing a solution that would allow me to continue using certbot.
I will get in touch with other admins and ISP contacts to see what they can do, otherwise this looks like the end of the road for Let's Encrypt and certbot.
We have discussed various problems with the geoblocking before and got nowhere. So from a b2b perspective, no I don't have enough time.
I'm optimistic that I can figure out the DNS-01 requirements and forward them to another admin. That takes time. Testing takes time. Renewal should be quick if it works. Fingers crossed.
You could CNAME the "_acme-challenge" entry(entries) to some other name that you can handle in a more automated fashion.
Right now [in short-term], you only need them to help renew the current cert(s) once.
Then you will have 90 days to figure things out [for the long-term plan].
I don't even know as I'd call it a "hack", it's just the normal way of delegating a name to another party. You may also want to investigate tools like acme-dns, where you delegate to a special-purpose DNS server you've set up that's just for answering the challenge.
Sticking with HTTP-01 is often simpler for most cases. Even if your network uses geoblocking for port 443, you may be able to convince them to not geoblock on port 80 (since all it normally does is redirect to https and can generally be left open, or maybe convince them to at least not geoblock on /.well-known/acme-challenge paths.
You may have a more fundamental problem, though, if you have responsibility for a website but don't actually have any administrative control over the infrastructure for it, then it's not really your website.
Also if you (HTTP) Specify the User-Agent string as
"Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
could help further restrict the window as well, I don't think it would add any real security.
This particular issue (geoblocking results in failed validation) is affecting thousands of users who have merrily geoblocked in the past and thought it would never be a problem, but I think Lets Encrypt are standing firm on the issue. I would doubt anything is going to change, so your solutions include:
remove geoblocking
target geoblocking to specific countries
allow /.well-known-acme/challenge/ http requests from any country (possible if using a web application firewall).
switch to DNS validation
DNS validation requires automated DNS updates, so your domain DNS needs an API thats supported by whatever ACME tool you use (cerbot). It needs to add/update/remove the _acme-challenge record for the domain and any subdomains (e.g. _acme-challenge.www.yourdomain.com). As an alternative already mentioned you can CNAME a permanent record to another zone, e.g. one you control, in order to delegate challenge response handling to a DNS zone that has a working API).
Some people use acme-dns as mentioned, either self hosting or using the unsupported hosted version that the acme-dns project provides. You can also optionally use Certify DNS (which is a product my company provides) which is a paid, supported product that does much the same as acme-dns but is managed, supported and cloud hosted.
Really the immediate workaround is to be much more selective about geoblocking, typically geoblocking is a reaction to concerns about traffic from specific countries.
Well, you're not wrong. It also means the decision to stop using Let's Encrypt is moot and its replacement will be decided by the folks who run the hardware. Mine is not the only server affected by the firewall and I've sensed some desire to switch to premium certs. In the meantime, I will make sure the firewall adjustments and DNS delegation are being considered.
Paid certs aren't necessarily manual; many CAs have automated systems (some even including ACME) and can allow for easier centralized management of an organization's certificates and getting them all installed where they need to go, plus centralized alerting and tech support and such.
But yes, paid certs need to follow the same rules for validating domain control as a free CA, and the industry is moving to validating from multiple perspectives in general. So I wouldn't suggest switching to a paid CA just based on this. If you want to use a different CA as a temporary workaround, there are plenty of other free ones.
Yeah, probably in general certificates should be considered part of an organization's infrastructure, and I would expect them to be managed by the same team that handles the firewalls and DNS and other key infrastructure.
FYI, the max lifetimes of premium/paid certificates are increasingly getting shorter. It was dropped from 2 years to 1 year a few years ago, and is expected to drop again. Eventually the options will likely be 90 day or 10 day certificates. In other words, the switch will be temporary and you'll likely have this same problem in a few years.
I suggest @miqrogroove reads the acme-dns docs. They best explain all the setup required for advanced delegation techniques – even if you don't use acme-dns.
With the acme-dns technique, you set up a FQDN to act as a nameserver, then CNAME records from your primary DNS onto that nameserver. Certbot then automates manipulating those records - which are isolated away from the primary/normal dns system.
You don't need to self-host or use acme-dns for this type of setup, you can use a commercial/third-party dns server like Cloudflare or Amazon.
Please note that Certbot is just an ACME client and is not causing any rate limit or geoblocking or whatever limitation imposed by the ACME server (i.e., the CA). Certbot can be configured to work with any ACME server out there. (See the list mentioned earlier.)
It's probably good to know that in time when the CA/Browser Forum ballot SC-067: Require Multi-Perspective Issuance Corroboration (Version 1) passes, all publicly trusted CAs need to do some kind of multi-perspective validation to keep issuing publicly trusted certificates. While these perspectives might or might not be more or less problematic with geoblocking, it's good to realise that.
Yup, I've tried it myself once with a NS RR for the _acme-challenge label pointing to the Cloudflare nameservers for that domain and using the certbot-dns-cloudflare plugin. Works quite nicely. With a NS instead of CNAME you also don't need a separate zone/domain for the ACME stuff.