No, that section deals with non-SNI clients. Modern browsers are all SNI capable. See the section just below that for how SNI works. And, I meant there are no docs that deal explicitly with the inheritance rule for missing ssl_certificate lines. I am not surprised as that should never be in normal use but comes up in debugging or clever new setups.
Even with non-SNI that doc shows which server block is selected - which would be the default one (or first if no explicit default specified). That would still be a functional server block and the reason for your protocol violation remains a mystery.
You have a somewhat unusual setup (your own binary, nchan) and an unusual failure. I am glad you found a work-around. If it is a bug in a later nginx handling self-signed certs it would be a serious regression as using a self-signed cert has many valid uses. (I am on 1.20)
UPDATE: I saw your sample config but it only had the includes not the contents of those. A better debugging method is to use nginx -T to see everything. But, I don't think it's worth pursuing your unusual symptom now you have something working.