Mixing validation types would only be possible if there was already a valid authz for certain hostnames with challenge type X, but certbot uses challenge type Y for the remaining hostnames without a valid authz. This was actually a bug where certbot would error out when it retrieved a valid authz with a challenge type it wasn't using itself.
In any case, it is not possible, as far as I know (and I'm pretty certain) to have a renewal configuration file with more than one authenticator. That is the issue here: the manual authenticator. Challenge type isn't really the question here: certbot sees a renewal file with the manual authenticator without any manual-auth-hook set. Challenge type isn't the issue here and --allow-subset-of-names doesn't fix anything.