ah, brilliant, I can test with that.
thanks!
1 Like
It's almost always something silly and mind-wracking. 
Don't be too hard on yourself. I went through similar when I first wrote CertSage.
1 Like
Well, exactly. Don't worry, I wasn't!
thanks again for your help.
1 Like
Happy hunting! 
1 Like
Hummmm.... I am sure I am wrong and I will go through and check everything again when I am less tired... but I think I found a bug somewhere in the stack.
The encoding of the values in the new-account should be base64url.
Here's some go code that proves my problem:
type newaccount struct {
Protected string `json:"protected"`
Payload string `json:"payload"`
Signature string `json:"signature"`
}
var acct newaccount
body_bytes := []byte(body)
err = json.Unmarshal(body_bytes, &acct)
if err != nil {
fmt.Println("unmarshall new account error!", err.Error())
panic(err)
}
payload_bytes, err := base64.URLEncoding.DecodeString(acct.Payload)
if err != nil {
fmt.Println("error decoding:", err.Error())
}
payload_bytes, err = base64.RawURLEncoding.DecodeString(acct.Payload)
if err != nil {
fmt.Println("error decoding:", err.Error())
}
payload := string(payload_bytes)
fmt.Println(payload)
My data above in this thread should be good enough if you really wanted to run it (this is what AI's should be able to do but in my experience it's still a massive trial).
What I get is this:
error decoding: illegal base64 data at input byte 92
{"contact": ["mailto:nic@REDACTED"], "termsOfServiceAgreed": true}
that is the first decode, with Base64Url decoding, fails and the second, which Go calls Base64RawUrl encoding, succeeds.
This is all down to pad characters, which aren't normally present in base64url encoding implementations (python is fine with my base64url strings, so is node).
RFC4648 specifies base64 and base64url but it does say that padding MUST include appropriate pad characters at the end of encoded data unless the specification referring to this document explicitly states otherwise
So I am not sure it's an error per se, because the base64 spec is clear that the absence of specification means there should be padding and as far as I recall the Acme specs do NOT specify.
This is a horrible mess though. The absence of something being specification is a lot less precise than it could be and as far as I am aware implementations do base64url encoding routinely without padding.
Anyway, being able to test against Pebble helped uncover this bug so thanks.
I'd be interested in any views about this padding issue.
RFC8555 clearly specifies that it doesn't use trailing = characters anywhere:
Trailing '=' characters MUST be stripped. Encoded values that include trailing '=' characters MUST be rejected as improperly encoded.
From RFC8555 Section 6.1 HTTPS Requests
This is also not something special to ACME: The entire JWS spec RFC7515 does it this way (so you will find this everywhere JWS is used):
Base64 encoding using the URL- and filename-safe character set defined in Section 5 of RFC 4648 [RFC4648], with all trailing '=' characters omitted (as permitted by Section 3.2) and without the inclusion of any line breaks, whitespace, or other additional characters. Note that the base64url encoding of the empty octet sequence is the empty string. (See Appendix C for notes on implementing base64url encoding without padding.)
3 Likes
Thanks, maybe I am just exhausted then... but isn't RawUrlEncoding wrong?
Ah... I'lll just go look at that Appendix and check I've got everything right.
I'm sure that will be it.
Hence why I said... 
// *** ENCODE BASE64 ***
function encodeBase64($string)
{
return strtr(rtrim(base64_encode($string), "="), "+/", "-_");
}
The rtrim(x, "=") is easily (and often) overlooked. I agree though that the specs are a morass to navigate though at times.
2 Likes
And as I said to you... I am pretty sure that is NOT my issue, look above, none of my base64url strings have '=' in them.
Which is why I am really surprised to find that go's JOSE module uses URLEncoding and NOT what Go calls RawURLEncoding, which looks to be the correct choice.
That can't be right though, because it works, it simply wouldn't unless this was a recent change.
My comment was more supporting @Nummer378's finding. As for the difference between the functions, I haven't looked into that myself. I just know what's worked for me for over five years.
2 Likes
Your code is correct.
But your code does what my code does.
I strip off trailing '=' in base64 -> base64url conversion.
But go does not do that unless you specify RawURLEncoding which Go's JOSE module appears to use and which, in turn, Pebble uses.
Hence my confusion. You're saying yours works. But mine doesn't, for really clear reasons.
Sorry, Go's JOSE module appears NOT to use.
Go's JOSE module appears to be expecting base64url values WITH padding.
But maybe I am just tired.
My initial instinct on strategy (for me) would be to try to run the data of concern through CertSage's functionality in an online PHP sandbox and compare the result with the output you're seeing from your code. If it were something goofy in the lower conversions or such, it might become evident. It could also scope potential gigo down. Process of elimination. The extra/different data you have and language differences might pose some challenges though. PHP is also not everyone's cup of tea either. I am very curious what you find though for future reference.
1 Like
we use different keys... you're using RSA and I'm using EC... not that it's affecting this but it does reduce the ability to do like for like testing.
I also noticed that dehydrated is using RSA keys for this.
I can't think that has anything to do with it.
The different data for the differing key types and how that's handled in terms of ordering and maybe concatenation or such could be involved. There could be multiple factors that are presented under a single error catch. I've been meaning to look into EC account keys. I've implemented EC cert keys, but those aren't as involved in terms of the ACME process.
3 Likes
it could be the keys... but this base64 problem is super weird.
it doesn't help that I am not a go expert... but I'll get there.
The lego ACME Client is written in go. We often recommend this client. It is open-source so you could review what that does: GitHub - go-acme/lego: Let's Encrypt/ACME client and library written in Go
Or, you could just use it as is 
Not sure why you are working on your own client but it is a fair amount of work. Proper error handling, ARI, profiles, quirks of different CA, different challenges, ...
2 Likes
Ok, my new theory is that Go JOSE must be trying to decode the base64 EC values wrongly.
Because as far as I can see the JOSE implementation uses the RawURLEncoding but I am getting an error from the URLEncoding...
The way it works (as you all know) is that you first base64url decode the protected and the payload... but then, to verify the signature you'd need to import the public key.
An EC public key includes further base64url encoded values... Perhaps they should have padding, who knows. We'll see!
I don't need to look at the Go client, but thank you. Now I have pebble I can simply reimlement all of Go JOSE's parsing until I hit the problem.
If I could use go better I could probably just use a debugger.
not sure why you are working on your own client
A spec that cannot be reimplemented is no spec at all.
Besides, I don't like ANY of the existing clients. As you say, it's a fair amount of work so one doesn't undertake it lightly and neither did I.
Myself, and many others, have implemented the full RFC8555 protocol - including ECDSA and ED25519 keys, so it definitely works. The Boulder implementation is correct.
ECDSA keys are a bit more finicky in the sense that if you're doing the JWS yourself, the signature formats vary a bit from crypto to crypto implementation which can be troublesome. But the base64 things are the same everywhere. The rule of thumb here is that when working with JOSE objects, you always use the url-safe alphabet without padding (any half-decent base64 library should have an option for this). All three objects in a flat JSON Web signature (header, payload and signature) are base64url-no-padding encoded (ACME does not use the unprotected header).
If this is what you're sending to the ACME server, it's definetly wrong. The signature seems to contain a JWS in compact serialization (you can see the dots and the duplicated header there). This is definetly not correct and likely something the parser wil trip over. The signature in a JWS is just the base64url-no-pad encoding of the signature bytes*, nothing more. To compute the signature you do need an encoding somewhat similar to the JWS compact serialization (with fixed ordering), which is probably where things went south.
*The JWS specs wants ECDSA signatures to be in PKCS#11 format, which isn't used by all crypto libraries. This is what I meant earlier.
7 Likes