Boulder is the same server that LetsEncrypt runs.
Pebble is a lightweight test server that LetsEncrypt developed. Wherever possible it makes a different implementation decision than Boulder, to ensure ACME clients do not hardcore behavior to only LetsEncrypt.
A good client might run test systems against both. Pebble is lightweight enough it can be used for unit tests.
IIRC, the changes that caused this hiccup were basically configuration changes and unrelated to code. The errors described in the passage above are all due to clients not being correctly programmed to the RFC’s specification and ignoring both the RFC and the payloads that LetsEncrypt serves.
In this case, the default chain was swapped and clients failed to correctly process the payloads.
The last step of the RFC is to download the certificate. A first certificate using the default chain is directly provided, and the location header will contain value(s) for certificates with alternate chains. (The certs are actually the same, but that is not guaranteed). A proper client will process the certificate download and extract the correct chains. Robust clients will grab and process the alternate chains.
Improper clients will not process the download and make an assumption on what the chain is, expecting it to be a hard coded value. Improper integrations might use a proper client, but ignore the (correct) chain from the client and make their own assumption on what the chain is.
many of the errors that people experienced are due to an improper client or integration, which ignored the payload from LetsEncrypt and assumed another chain. errors also likely happened due to the client’s operating system (or trust store) not having the root of the default chain.
While simulating these changes on staging or a test server can be helpful, they’re not required for testing. All that client needs to do is follow the rfc and analyze the downloaded certificate to determine the chain and act appropriately. (Note: pebble uses a new chain on every invocation which reinforces the need for clients to analyze the payload and make no assumptions about the chain).
IMHO, a lot of these issues could be avoided or at least more easily detected if the ACME server advertised which chains are currently configured as default and alternate before trying to renew instead of requiring analysis after renewal.