I have some subdomains for non-public access.
I.e. they are visible from outside, but protected by ip-rules and/or login/password.
Can I make rules for allowing the challenges to be seen, either based on the filename/directory of the challenge or by looking at the source ip (no, they will change someday) or the user-agent?
Out of the options you listed this is probably the best bet. The challenge will always be in the directory /.well-known/acme-challenge/. You should be able to carve out an access exception for that path in your webserver config.