Intermittent SERVFAILs against Google Cloud DNS is not a good sign.
On the other hand, we don't have any other user reports. Usually there are, for widespread problems.
If Google is maybe rate-limiting the secondary VAs, it might help to actually create a CAA record for each of your 10 subdomains, to prevent extra DNS queries from the CAA algorithm. I don't know if Let's Encrypt caches negative responses so the difference might be negligible.
OK so I added a CAA record for the top level domain (I had not explicitly created one before, and it was never a problem). I have now had to fully successful sets of challenges in a row. Unless I post back in this thread saying otherwise, let's assume that fixed it.