I am wondering if there is a project to allow doing HTTP-01 and TLS-ALPN-01 challenges on a port other than the 80 (http) and 443 (https) defaults?
There are issues with using theses default ports in some cases (shared hosting, port already is use by a running web server, …).
So something like a new challenge, similar to HTTP-01 and TLS-ALPN-01 but using a well-known IANA-assigned port (say "2263", "ACME" on a phone :P) the ACME servers would contact to check challenges over HTTP or HTTPS (instead of 80/443) could be a good alternative (I guess).
Is it something viable? Was it experimented in the past and abandoned?
Quite intentionally so. Imagine being a shared hosting customer, and all of a sudden, one of the other customers on the server spins up a webserver on port 45000 and obtain a certificate for your domain. Oops.
This approach goes back in some sense to the well-known ports concept (where remote systems could use the port number to determine whether a process was being run by a machine's administrator, or by a regular user, in order to decide whether that process could be considered to "speak for" the machine). But of course the authorized port concept has taken the concept further and been stricter about it, while still having the same basic goal.
I guess shared hosting provider could block this well-known port to disallow any use of ACME client with this kind of challenge because it's not compatible with it's service.
It was my thought exactly: we have ports for each specific usages (Whois, DHCP, WoL, time, SNMP, BGMP, …), why not for ACME/certificate usages?
Yes, but the DNS challenge has it's own issues too, like: latency, compatibility, etc.
It's too much of a risk to leave such a thing to the (shared) hosting providers.
I can't see into the future, but I don't see the CA/Browser Forum changing this Baseline Requirement changing anytime soon. And as said before, Let's Encrypt has to abide to these requirements.
Specifically, it would be hard to introduce now because it would be hard to get all of the shared hosting providers to be aware of what it is and what it means. Let's Encrypt has already disabled or replaced two different existing or proposed proof-of-control challenge methods because of shared hosting providers' empirical behavior (that created a possibility that one customer could pass a challenge for another customer's domain).
And since false positive certificate issuance is considered drastically worse than false negative certificate denial, the motivation to get hosting providers to change their behaviors is kind of lacking (both for Let's Encrypt and for the CA/B Forum, I expect).
Since I'm doing a consulting project related to an attempt to update old Internet standards in a non-fully-backwards-compatible way, I can feel the sadness in the fact that it's challenging to get all of the Internet sites to pay attention to present-day conversations about new proposed standards and best practices! It's really annoying and unfortunate sometimes. Although I think in the certificate issuance case, Let's Encrypt's current options ultimately work decently for a huge portion of the web sites that want certificates.
There are few other threads this has come up on, but that thread covers almost every concept.
I am 100% behind the idea of an RFC defining a dedicated port for an "acme-challenge", as that would handle the problems you suggested. This would require the port to be reserved/privileged, so only server administrators could obtain the certificate.
The big issue is that the "well-known" reserved/privileged ports are not consistent across operating systems and operating system versions. There are a handful of random operating system versions where ports <1024 do not require admin privileges for binding. There are very few ports under 1024 which are commonly privileged across most systems.
For reasons mentioned above, the best solution here is to use a server with the ACME client integrated natively into it so that port contention is not a problem.
