I have LE certs set up on several domains and they are set to auto-renew via an HTTP ACME challenge. This recently stopped working though as it appears an overzealous application firewall somewhere in our organization is blocking HTTP GETs including the /.well-known/acme-challenge/ string on port 80:
My proposal: the ACME protocol allows for a http->https redirect of course. Would you consider automatically trying an http->https substitution if the original ACME challenge fails in this way (perhaps only if HSTS is set up)?
Of course I would prefer if our organization did not interfere with HTTP traffic in this fashion, but we are large enough that the IT folks I've spoken to aren't even aware of this firewall or how it might be reconfigured!
The challenge using port 443 is called tls-alpn-01. That's the challenge that will try port 443 the first time. The http-01 challenge will always start on port 80 and can only change protocols (and thus ports) using redirects.
Changing the http-01 challenge to retry on an entire protocol (and thus port) is a major change and I'm afraid has a very slim change of ever being reality, even if LE would think it's a good idea.
An alternative due to blocking firewalls could always be the dns-01 challenge, but that has its own drawbacks.
Also, while I sympathise with your troubles (I'm sure a large company with ignorant/incompetent IT department(s) is very frustrating), I don't think that a firewall vendor wreaking havoc on the ACME protocol by setting the default to DENY is a good reason to even propose to change the http-01 challenge.
Just to add some clarity to the above comment by @Osiris: there once was a challenge TLS-SNI-01 that functioned just lot like http-01. It was removed several years ago because of a security vulnerability. Due to implementation details in most (all?) major web servers and shared hosting providers, it is impossible to authoritatively prove control of a domain with a secret file – like the http-01 and tls-sni-01 – rely on. The tls-alpn-01 challenge introduced a new protocol designed to avoid the vulnerability in tls-sni-01, but most web servers do not support this new protocol yet.
@9peppe, they could still implement it alongside HTTP within the same software application—I think Caddy can natively do the TLS-ALPN challenge in addition to being a web server.
IMHO: based on their earlier decisions, mission statement, and statements of employees - if there were a way to securely offer HTTPS validation outside of tls-alpn-01 that is available to a wider audience, LetsEncrypt would definitely pursue it.
There are several OpenResty modules that bring this into the Nginx ecosystem.
Under "vanilla" nginx, you can proxy only the ALPN requests to something like dehydrated.
Yes, they can. It's a bit out of scope for a webserver.
The one advantage of tls-alpn-01 is avoid interfering with http routing or the webroot. So you can host websites for arbitrary clients, and nothing they can do will break validation.
It makes sense for a big provider. But for a single user with complete control over machine and website... not really.