The HTTP-01 challenge mechanics are defined as part of the ACME protocol. If you want the really nitty gritty details you’d be interested in reading ACME draft-07, Section 8.3. That describes the overall HTTP-01 challenge.
The contents of the file itself are an ACME key authorization which Section 8.1 describes. Key authorizations are used with the other challenge types as well.
The condensed explanation is that Certbot creates an ACME account & keypair for you when you run it initially and register an account. The public key of that account keypair gets registered with the ACME server (Let’s Encrypt in this case).
When it comes time to prove ownership of a domain Certbot tells the server it wants to be authorized for the domain. The server sends back a pending authorization that has a HTTP-01 challenge as one option for completing the authorization. That HTTP-01 challenge comes with a random token and Certbot takes the token and uses your ACME account private key and the token to creates a key authorization string as described in Section 8.1. The key authorization string is then put in a file identified by the random token and then placed in
.well-known/acme-challenge/ for the ACME server to check.
When the ACME server sends a HTTP request to
yourdomain/.well-known/acme-challenge/TOKEN it can look at the key authorization string from inside that file and validate it with the token and the public key known for your account. If it checks out then your ACME account is considered authorized to issue certificates for that domain.
Hope that helps explain somewhat!