I am bound to an external DNS provider not supporting the DNS-01 challenge, but I wonder if it would be possible to support a "postgresql challange"? Instead of modifying a public DNS directory it could modify a public postgresql database.
While I suppose in principle such a challenge type could be added, I doubt that the effort of standardizing, testing, and security-reviewing it would be worth it for a pretty unusual use case. (You have a postgres database accessible to all, with tables read-only to most users but only writable to you?). And even if somebody were to start working on all that, it'd be quite some time before it were usable.
You're probably much better off delegating the _acme-challenge. DNS name from your non-API-supported DNS system to a system like acme-dns, or having a web server on the same system just for fulfilling an HTTP-01 (or TLS-ALPN-01 I suppose) challenge. Certbot, for instance, has a "standalone" mode for this sort of thing, where it spins up a mini-webserver just for doing the challenge. You just need to ensure that the firewall is open on port 80 for the server, even though most of the time nothing would be running on that port. (If you want to get really fancy, you might be able to script updating firewall rules to allow it only during renewal times with certbot's --pre-hook & --post-hook; I know that some people prefer doing it that way.)
The HTTP-01 challenge is not an option, because I need support for wildcard certificates. And as written before, my DNS provider doesn't support letsencrypt.
The postgresql instance would be a dedicated database, running just for the challenge. It would be read-only for everybody, except for certbot on the internal network changing a specific entry in the database, similar to DNS-01.
Even if there would be a new challenge type of reading a database, it probably wouldn't be allowed for wildcard certificates. Being able to run software on a hostname doesn't necessarily mean that you "own" all subdomains of that name. Whereas being able to edit DNS pretty much does.
You need to either not use wildcard certificates, or you need to be able to automatically update your DNS.
The DNS-01 challenge proves you control the DNS records for the domain. Your suggested postgresql challenge would only prove that you have write access to a given database – it does not suggest or imply any control or ownership of the domain.
This does not prove, imply or suggest that you have control of the domain you want a certificate for. Anyone can create such a database.
The HTTP-01 challenge works, because it proves control of port 80 -- which requires advanced privileges -- on a given domain. PostgreSQL does not run on a privileged port, and there is no expectation or guarantee that only privileged users could control a specific database. There is also no standard way to map a domain to a PostgreSQL database.
This is the best advice in this thread. The acme-dns solution (which I second/third/whatever) uses the DNS delegation technique.
Certbot likely supports a large number too, either natively or with the lexicon dns module. If your DNS provider offers an API key, then it is almost-definitely supported by either package.
Note that it's Let's Encrypt putting this limitation on the http-01 challenge. It's perfectly allowed to issue a cert with a wildcard hostname which was validated through the http-01 challenge if you'd ask the CA/B Forum Base. See section 18.104.22.168.6 of the current (1.7.4) Baseline Requirements.
Therefore, if a whole new challenge would be invented (which would also require approval of the CA/B Forum by the way), it isn't impossible that this new challenge would also allow wildcard hostnames.
You can also CNAME _acme-challenge.yourdomain.com in DNS and point it to a TXT record in a zone hosted on an API enabled DNS provider (such as AWS Route 53 or Cloudflare), then when answering the DNS challenge you would set the TXT record there.
Let's Encrypt will follow the CNAME redirection, which is also how the acme-dns approach works (pointing your CNAME to an API controlled DNS servicer, then updating the challenge response there).
I have found the _acme-challenge.yourdomain.com and the acme-dns project on github. I agree that this the way to follow.
But I think there is a misunderstanding: I did not mean to suggest to get rid of DNS in the challenge itself. A postgresql challenge would still rely upon a SVC record in DNS to access the dedicated postgresql database (proving that I am in control of this DNS zone). The point is that this would be a static record, similar to the static CNAME entry for _acme-challenge.
Yeah so if you mean dynamically serving your TXT response based on the contents of a database (and performing your DNS challenge by updating your database so that your TXT record updates) then that's basically what acme-dns does, except the update is via an http api (which in turn updates the database). I do with same using Google Firestore as the backing store.
Yeah, that's unlikely to work out, database communications aren't standardised for a start.
You could maybe (?) have an svr record point to a standardised challenge response API and also have the CAA record idea that says which acme accounts are allowed to request certs for each domain. That way one API service could handle many challenge responses for many domains.
Yeah, I'm not saying I think it's practical, and I'm not even sure it's what OP was proposing. But I think it would adequately demonstrate domain control--if the delegated challenge is good enough, I'd think this would be too, by the same logic.
But implementation would be a problem--after all, if we'd support Postgres, why not MySQL? MongoDB? MSSQL? Etc.? And is it really needed?