OVH Posh-Acme suddenly stops with query out of time despite "good" time

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: dbridgemailer.com

I ran this command:

Set-PAServer LE_PROD
$acct=get-paaccount
$pArgs = @{
OVHAppKey = 'xxxxxxxxxxxxxx'
OVHAppSecret = (ConvertTo-SecureString "xxxxxxx" -AsPlainText -Force)
OVHConsumerKey = (ConvertTo-SecureString "xxxxxxx" -AsPlainText -Force)
OVHRegion = 'ovh-ca'
}
$CertNames='dbridgemailer.com','www.dbridgemailer.com'
New-PACertificate $CertNames -AcceptTOS -Contact xxxxxxxxxxxx -Plugin OVH -PluginArgs $pArgs -Verbose -Force -ErrorAction Stop

It produced this output:

VERBOSE: Updating directory info from https://acme-staging-v02.api.letsencrypt.org/directory
VERBOSE: Using ACME Server https://acme-staging-v02.api.letsencrypt.org/directory
VERBOSE: Using account XXXXXXXXXX
VERBOSE: Order name not specified, using 'dbridgemailer.com'
VERBOSE: Creating a new order 'dbridgemailer.com' for dbridgemailer.com, www.dbridgemailer.com
WARNING: Fewer Plugin values than names in the order. Using OVH for the rest.
VERBOSE: Publishing challenge for Domain dbridgemailer.com with Token XXXXXXXXX using Plugin OVH and DnsAlias ''.
Submit-ChallengeValidation : {"message":"Query out of time","httpCode":"400 Bad Request","errorCode":"QUERY_TIME_OUT"}
At C:\Program Files\WindowsPowerShell\Modules\Posh-ACME\4.23.1\Public\New-PACertificate.ps1:253 char:9

  •     Submit-ChallengeValidation

  •     ~~~~~~~~~~~~~~~~~~~~~~~~~~
    • CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Submit-ChallengeValidation], WebException
    • FullyQualifiedErrorId : WebCmdletWebResponseException,Submit-ChallengeValidation

My web server is (include version):
Windows Server 2016

The operating system my web server runs on is (include version):
Windows Server 2016

My hosting provider, if applicable, is:
OVH

I can login to a root shell on my machine (yes or no, or I don't know):
Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
Posh-Acme

This worked perfectly fine initially, now it fails, and yes the time is set correctly (EST timezone)

1 Like
2

Hi @Only1Atreyu,

Tagging the author of the ACME client @rmbolger

2 Likes
3

Thanks

3 Likes
4

This is a response from the OVH API. It makes it seem like there was some sort of timeout on the server side. But I'm not sure why they'd return that as a Bad Request error.

The Bad Request implies that it's the client's fault. And I found at least one other person talking about this error saying they fixed it by fixing the local system time. But if you say you're local system time and timezone are correct, I'm not sure what else the problem would be particularly if it was previously working.

Maybe just a temporary issue on the OVH side?

If you want to test just the plugin separately from the cert order process, try running this to create a dummy record:

Publish-Challenge dbridgemailer.com (Get-PAAccount) faketoken OVH $pArgs -Verbose
6 Likes
5

All good now - Turns out the API was just not responding.

Thanks eveyrone,

6 Likes
6

This is broken again when ran this morning. Still Query Times Out.

    + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand
Invoke-RestMethod : {"class":"Client::BadRequest","message":"Query out of time","httpCode":"400 Bad Request","errorCode":"QUERY_TIME_OUT"}
At C:\Program Files\WindowsPowerShell\Modules\Posh-ACME\4.24.0\Plugins\OVH.ps1:412 char:17
+     $response = Invoke-RestMethod @restArgs @script:UseBasic
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Either I'm not doing something right or the OVH API is unreliable.

Absolutely frustrating.

7

Best just to contact OVH support and tell them there is a problem with their API timing out, it's possible they don't know.

p.s. sorry about Artax, still getting over that one.

3 Likes
8

I called OVH this morning and they confirmed that the API has changed.

More to follow...

1 Like
9

This is the reply from OVH this morning:

Hello xxxxxxxxxxx,
Sorry for the delay occasioned and thank you for your patience.

before continuing, please note that there was indeed multiple updates and/or changes that happened throughout the last months. This includes the implementation of the new API and changes within our infrastructure. Although, the new API is the main change that happened.

With that said, we would recommend you to adjust the call to use the new API as it's faster and more reliable, although you should be able to keep using the old one. For more information, we would recommend you to take a look at the following links:

.............. I will see where in the LE OVH script I can try "console-old".....

Atreyu

10

@rmbolger will see this already but to properly log an issue for Posh-ACME please see Issues · rmbolger/Posh-ACME · GitHub

Regarding API changes, if they do something that breaks callers then that's their fault and they should fix it, as a customer pushing back to them is the best course of action. They can happily have different API versions, but they shouldn't break what's there without a formal sunsetting process, otherwise nobody can really take their API seriously (cough GoDaddy cough).

[As for the original "query out of time" error, that's clearly a timeout of their internal process so still very much an internal OVH problem and you would expect it to be transient.]

4 Likes
11

Hello @Only1Atreyu ,

OVHcloud API administrators here!
I just come across this thread as I was searching for other resources.

Unfortunately, the answer you got from our support team is wrong, I'm sorry.
The only change between Ovh Api and OVHcloud API is the UI displaying the API routes. Behind the scene, nothing has change on our API.

{"message":"Query out of time","httpCode":"400 Bad Request","errorCode":"QUERY_TIME_OUT"}

This message is a real Client Error. It is returned by the API engine when the signature generated by the wrapper is too far from the current timestamp.

You said that your server is at the right timezone, but have you sync your server with a NTP server to make sure you are at the "right" time ?
If yes, there might be something wrong, either on the wrapper or on our side, I would be happy to assist, I will subscribe to this thread from now on.

Also, could you share with me the support ticket number, I would like to clarify internally the communication around this "new API" ... :sweat_smile:

Thanks,
Romain

7 Likes
12

The server is synced with the right time.

The support ticket at OVH is CS10146765.

Atreyu

2 Likes
13

I haven't found any requests that worked in the last 30 days.
Any idea when you order/renew a certificate correctly for the last time ?

I'm using the information (customer account) you gave in your support ticket, if you are using another account to perform Let's Encrypt challenges, feel free to indicate it, it will help for my research.

Romain

3 Likes
14

It worked August 10th.

I'm using the same OVH account.

1 Like
15

@ovhcloud-devplatform as this is going to be a common source of problems for your API users could you perhaps rephrase the error message to be something clearer (and the QUERY_TIME_OUT error code is also the ambiguous but obviously difficult to change now), e.g. "Timestamp header is not current time UTC (diff -237s, signature tolerance is +/- 30s)"

4 Likes
16

I can't find any calls from your account. I searched from August 5 to August 12.
I also look for your user-agent or your source IP address.

Regarding calls with "Query out of time", I found some calls on September 6 night and yesterday that I would identify coming from your account (it include some DNS zone of your customer account).
On those calls:

  • API Signature timestamp is drifting, with a difference of 3 minutes regarding our NTP. I'm not a Powershell expert, but either there is an issue in Posh-ACME, either your server is lagging 3 minutes away.

By any chance, are your servers not drifting away time ? If not, we should discuss with the Post-ACME maintainer as I don't understand what Get-DateTimeOffsetNow is doing, and where it is defined.

Romain

3 Likes
17

Yes I tried on the 6th .... and just tried now, 9:20am EST.

The server is synced via NTP on a Server 2022 domain controller on Vmware Exsi 6.0

There is no clock drift I can see?

The NTP servers are
ca.pool.ntp.org and 0.us.pool.ntp.org

1 Like
18

That's just a wrapper function for the .NET DateTimeOffset.Now property which makes mocking in PowerShell tests easier.

4 Likes
19

On my side, with our logs:

signature_timestamp
    1726147223
call_start_timestamp
    1726147387
timestamp
    2024-09-12 15:23:07.290692400 +02:00

Your server sent timestamp 9:20 EST while it was 9:23 EST on our server.
You might want to check with https://time.is/ which indicate what drift you have with a reliable time source.

4 Likes
20

I showed 30 seconds drift.

C'mon... can you please just fix this for a little more leniency on the drift than 3 minutes.

I cannot be the only OVH customer with this problem.

Atreyu