You should get 4 different HTTP requests per challenge URL, as described in ACME v1/v2: Validating challenges from multiple network vantage points.
In your case, you should see a total of (5 requests * 4 vantage points =) 20 requests.
If the error leads with "During secondary validation", then the likely cause is that the cloud-based vantage points were unable to connect to your server. These (currently) come from AWS. Worth double checking that you're not blocking any AWS ranges.
If the problem is intermittent, I can point to some previous instances where unhelpful "anti-DoS" home router features have caused these kinds of symptoms. The requests tend to arrive all at once, which may trigger firewalls that do throttling.