This problem applies to many domains we tried today - one above it's just an exmaple.
Not only we've tried renewing but also issuing new certificates. We've also tried both with --preferred-challenges=http and --preferred-challenges=dns. With or without manual authentication it's the same outcome.
uacme: version 1.7 starting on Tue, 12 Jul 2022 14:51:06 +0200
uacme: fetching directory at https://acme-staging-v02.api.letsencrypt.org/directory
uacme: retrieving account at https://acme-staging-v02.api.letsencrypt.org/acme/new-acct
uacme: account location: https://acme-staging-v02.api.letsencrypt.org/acme/acct/60526044
uacme: creating new order at https://acme-staging-v02.api.letsencrypt.org/acme/new-order
uacme: order location: https://acme-staging-v02.api.letsencrypt.org/acme/order/60526044/3154128824
uacme: retrieving authorization at https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/2984752594
uacme: starting challenge at https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/2984752594/RS54gw
uacme: polling challenge status at https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/2984752594/RS54gw
uacme: polling challenge status at https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/2984752594/RS54gw
uacme: polling challenge status at https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/2984752594/RS54gw
uacme: polling challenge status at https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/2984752594/RS54gw
uacme: polling challenge status at https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/2984752594/RS54gw
uacme: polling challenge status at https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/2984752594/RS54gw
uacme: polling challenge status at https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/2984752594/RS54gw
uacme: polling challenge status at https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/2984752594/RS54gw
uacme: polling challenge status at https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/2984752594/RS54gw
uacme: polling challenge status at https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/2984752594/RS54gw
uacme: polling challenge status at https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/2984752594/RS54gw
uacme: polling challenge status at https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/2984752594/RS54gw
uacme: polling challenge status at https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/2984752594/RS54gw
uacme: polling challenge status at https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/2984752594/RS54gw
uacme: polling challenge status at https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/2984752594/RS54gw
uacme: polling challenge status at https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/2984752594/RS54gw
uacme: polling challenge status at https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/2984752594/RS54gw
uacme: challenge https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/2984752594/RS54gw failed with status invalid
uacme: the server reported the following error:
{
"type": "urn:ietf:params:acme:error:serverInternal",
"detail": "During secondary validation: Remote PerformValidation RPC failed",
"status": 500
}
A software upgrade changed how our networking configuration was interpreted in AWS when a new instance starts, where we run parts of our "secondary validation" infrastructure. Autoscaling up and down based on load automatically replaces instances on demand. Overnight, all the instances ended up getting replaced as load scaled up and down, but the new instances weren't able to reach the internet. We only alert on our staging instance during business hours, so staging was down for a few hours until we fixed it in the morning.