Cannot negotiate ALPN protocol "acme-tls/1" for tls-alpn-01 challenge

[ddevault]

Greetings. We are developing a client called tlstunnel which is designed to register certificates for incoming TLS connections on-demand, then proxy the connections to non-TLS services elsewhere. This is not designed to be a web server, and the http-01 challenge is not an option for us.

We have had success with the tls-alpn-01 challenge before, but this particular deployment is causing us esoteric problems. We are trying to register a certificate for pages.sr.ht and receiving the error in the topic subject, but little else:

https://paste.sr.ht/~sircmpwn/1efc1123dee6c8bf55fb4cdbbe938986a05351cf

DNS is set up correctly and port 443 faces the internet directly, not through any kind of firewall or reverse proxy. I've confirmed that ALPN is working in general via openssl s_client.

The config file we're using is the following:

https://paste.sr.ht/~sircmpwn/7c4f348dbf96600a09609f575cc49dffae3460a3

In case anyone wants to try to reproduce this issue. We're using the master branch of tlstunnel and running it as root on Alpine Linux w/musl libc and Go 1.15.7 on amd64.

[ddevault]

My colleague found the issue:

https://git.sr.ht/~emersion/tlstunnel/commit/f0bd8e92148660cc325bce986cdc2f21aa6d96fb

It's pretty difficult to debug ACME client issues due to LE's restrictions, caching, and rate limits, even on the staging server.

[rmbolger]

Pebble might be a better choice to test against in a case like this. It's self-hosted and has knobs to disable things like caching and no rate limits.