Sudden increase in http validation errors

Hi,

This could be a bug in our (new) software but we’re seeing a sudden increase in users reporting failed http validations and I wondered if anyone had seen anything similar recently.

It manifests as (for example):
The key authorization file from the server did not match this challenge [KMOuDJTWHpoH_D0JrmFwQEKRCHgdDIms-TqCd8xDsW8.bWNHHDwqcLiiiRgAPwI20OY11ww-tGvPKnPoaLWRdNE] != [KMOuDJTWHpoH_D0JrmFwQEKRCHgdDIms-TqCd8xDsW8.OZiapo_qbetlNpVoS-6pJ7Bq9qD4hRDab2O6dOxzBds]

I would have blamed our recent updates to our client but someone is also reporting it with an old version.

Hi @webprofusion

the content of the file is the token, a dot and a hash value of the public key.

[KMOuDJTWHpoH_D0JrmFwQEKRCHgdDIms-TqCd8xDsW8.bWNHHDwqcLiiiRgAPwI20OY11ww-tGvPKnPoaLWRdNE]

[KMOuDJTWHpoH_D0JrmFwQEKRCHgdDIms-TqCd8xDsW8.OZiapo_qbetlNpVoS-6pJ7Bq9qD4hRDab2O6dOxzBds]

The two tokens are ok, the hash values are different.

So it looks like your client uses the wrong public key to compute that value.

Thanks, investigating this now. The weird this is the reported failures using the old version which has been working for about a year. That could be a coincidence perhaps.

Perhaps now there are two accounts and two keys. So the old version had a bug when more then one key exists.

Using only one key -> no bug.
Using two or more keys -> "oh, there is a bug"

I want to let you know that I have the same issue.
I’m going to do some local testing/debugging if I can find a cause for it.

Exactly the same, the first part is equal, the part after the dot is different.

[yj-yGqOXodxcTFwA9QtDaoGM1kwC2Nq74EnW8R4jcVI.WNMbB0A2kFt2-myXnACBO7AJKblBSYOLD8jw-mY9Ffg]
[yj-yGqOXodxcTFwA9QtDaoGM1kwC2Nq74EnW8R4jcVI.ovuqspxvIS8bK0R-SMLaP8pZqgJFuaaxahLX8zOAP7A]

It's a bug of your client-program.

That creates the wrong content. The length of your content looks correct, so perhaps the wrong key is used to create the string.

Interesting! What kind of programming environment is your code? My app is .net using the certes library. In my testing I can’t reproduce the problem but other users are seeing it. They will be using a new account key specific to the acme v2 api but they will be re-using their old registered email address for the new account.

Hi Christopher,

We build a console-application which can be called to fetch a new certificate or to renew an existing certificate. The program is build with C#, version 4.5.2 (webserver does not support a higher version) and the Certes Nuget package version 2.3.0 is used for the Acme.

Unfortunately the collegue who build the application is not present, so I do not know the exact workflow of the program (had to debug step-by-step). In this case it we try to get a new certificate for a domain. The account is created succesfully, all steps seem to be ok. Seems that problems always occur at fridays when I have a lot to do :wink: Because other issues have more priority I stop debugging and will check monday if I can find the cause somewhere in our code.

Why do you create a new account?

You can use one single account with a lot of domains / subdomains and a lot of renews.

I don't think the library "Certes" (I don't use it) is buggy. Looks that the function of the library is called with the wrong key.

Hi @webprofusion

Sounds pretty strange! I concur with most of the folks in this thread that suggest it sounds like a client side problem computing the HTTP-01 key authorization for the challenge response.

From the server side we haven't made any changes to Boulder/HTTP-01/the prod environment that I would expect to cause an uptick in this kind of problem. I also haven't seen any reports besides this thread.

Let me know if there's something you think I could help with from this side!

1 Like

@webprofusion

I just wanted to let you know that it works with us again.
What did I do?

On friday we used the AcmeClient.CreateAccount(…) to create an account, the private key we retrieved did not seem to work. So today I cleaned the “createdate”-field in our database, which is a trigger in our code to call AcmeClient.CreateAccount(…) again. The private key which we fetched is now used by trying to register 2 domains and a miracle: it works.

So the problem sees to be with the private key which was returned from Let’s Encrypt. Maybe you could do the same?

1 Like

@lordofcode thanks for that. I just don’t know that Let’s Encrypt would be returning invalid account info as that would affect other clients - it seems more likely that there’s something in the fact that we’re both using certes on .net, and it may only affect certain users in certain situations. I’m still debating the likely causes in our code base, we do ‘upgrade’ from an previous account key to a new one on install, as we are storing the artifacts in a different way between versions. I thought perhaps we were weren’t cleanly changing account key within the current certes AcmeContext (and perhaps the acmeContext needs to be recreated after the account is created).

I'm probably crossing wires here from not understanding Certes/.net but there shouldn't ever be a point at which Let's Encrypt or an ACME server gives you a private key. That would be bad security! :slight_smile:

It sounds like @webprofusion's theory is that the public key associated with the account doesn't match the private key being used on the client side because of some kind of state confusion.

@cpu to follow up, as @lordofcode found, creating a new account fixed the issue for those affected. I haven’t had many further reports however one user reported DNS challenges were failing and creating a new account also seemed to fix that. We did have a bug/feature where registering a new contact didn’t use a new private key (and kept the same account key). Switching to a new account private key results in a new accountid when registering a contact and that fixes the issue. I don’t know if it was a transient thing yet.

Hmm. I'm not sure I follow: It would be expected to keep the same account key after changing account contact information. You mean registering a new account didn't use a new private key? In that case the system should prevent you from creating the second account because the key is already in use from the first.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.