webprofusion wrote the ah-ha comment:
When your client creates the order it's give[n] a list of challenges (by the CA) to attempt for each DNS identifier you want to prove control of. Your client then picks one (per identifier), prepares to complete the challenge, then tells the CA you're ready for them to check. If your chosen challenge fails to validate then so does the rest of your order.
The following is a simplified response to the WASD wuCME community, by way of explanation and bug fix, provided here to close the issue. The fix has been trialled on three independent systems and is confirmed. Thanks to the LE community for its inputs.
---------------
Now the "nor really come to grips with the internals", has returned like the Ghost of Christmas Past.
The wuCME version log begins
01-JUN-2020 MGD v1.0.0, initial release
08-AUG-2019 MGD initial development
so just over four years in production. Four years, and hundreds of Let's Encrypt certificates issued across multiple systems (with a few enhancements and tweaks along the way) ... basically all due to serendipity.
Within a handful of days some fifty months on, multiple reports emerged of strange challenge failures. In particular
challenge=tls-alpn-01 ident=www.******.au token=bxf_******aPg8 key_auth=ZYPv******EMxBc
and also the likes of
challenge=dns-01 ident=www.******.au token=oMj3Vk******A3VU key_auth=cayvE2 ******QbAuY
but challenges that were successful as well
challenge=http-01 ident=www.******.au token=bxf******jbaPg8 key_auth=bxf_y******d9_G-L8
I had coded wuCME's use of the core uacme.c on the presumption that unless you specified an alternate, the challenge defaulted to the simplest, "http-01". Nope! Incorrect! Not even close! wuCME defaults to the first offered by the Let's Encrypt servers. LE currently offers three.
Just happened that for the first four years of wuCME production LE must have offered "http-01" first, which wuCME supported. Recently LE's algorithm obviously changed from a fixed order of challenges, to a varying order. wuCME's simple grab the first and run as often as not was suddenly broken (probably with a large enough sample, 2 in 3 likely broken :-)
This bug had remained latent for all those years.
The fix, an embarrassing
if (strcmp (type, "http-01"))
{
/* discard unsupported challenge */
free(key_auth);
continue;
}
And to rub salt into the wound, the uacme.c default behaviour (right there in adjacent code) is to similarly step through the challenges allowing the user to accept or decline each provided by LE.
msg(0, "type 'y' followed by a newline to accept challenge"
", anything else to skip");
if (scanf(" %c", &c) != 1 || tolower(c) != 'y') {
free(key_auth);
continue;
}