OVH Posh-Acme suddenly stops with query out of time despite "good" time

Sounds like your VM may actually be taking its' time from the host.
Check the host time drift as well.

3 Likes

The EXSI host has the NTP service running with the two time servers mentioned.

There appears nothing wrong with the time on the server or the domain controller, for example:

C:\Users\xxxxxxxxxxxxxxxxxxxxxx>w32tm /query /computer:xxxxx /configuration
[Configuration]

EventLogFlags: 2 (Local)
AnnounceFlags: 10 (Local)
TimeJumpAuditOffset: 28800 (Local)
MinPollInterval: 6 (Local)
MaxPollInterval: 10 (Local)
MaxNegPhaseCorrection: 172800 (Local)
MaxPosPhaseCorrection: 172800 (Local)
MaxAllowedPhaseOffset: 300 (Local)

FrequencyCorrectRate: 4 (Local)
PollAdjustFactor: 5 (Local)
LargePhaseOffset: 50000000 (Local)
SpikeWatchPeriod: 900 (Local)
LocalClockDispersion: 10 (Local)
HoldPeriod: 5 (Local)
PhaseCorrectRate: 7 (Local)
UpdateInterval: 100 (Local)

[TimeProviders]

NtpClient (Local)
DllName: C:\WINDOWS\system32\w32time.dll (Local)
Enabled: 1 (Local)
InputProvider: 1 (Local)
AllowNonstandardModeCombinations: 1 (Local)
ResolvePeerBackoffMinutes: 15 (Local)
ResolvePeerBackoffMaxTimes: 7 (Local)
CompatibilityFlags: 2147483648 (Local)
EventLogFlags: 1 (Local)
LargeSampleSkew: 3 (Local)
SpecialPollInterval: 1024 (Local)
Type: NTP (Local)
NtpServer: ca.pool.ntp.org 0.us.pool.ntp.org (Local)

NtpServer (Local)
DllName: C:\WINDOWS\system32\w32time.dll (Local)
Enabled: 1 (Local)
InputProvider: 0 (Local)
AllowNonstandardModeCombinations: 1 (Local)

VMICTimeProvider (Local)
DllName: C:\WINDOWS\System32\vmictimeprovider.dll (Local)
Enabled: 1 (Local)
InputProvider: 1 (Local)

And yet, you've found it to be 30 seconds off.

4 Likes

Implementations in Python/Go/Java/Node/C# are computing the drift between your server time and the real time, and is using this drift to send correct timestamp and then signature.

It seems, from my understanding of the Powershell, that the implementation in PS from PoshACME is not computing this drift. Thus, it can leads to errors if your server is not sync with time.

2 Likes

How can monitor in real time, the time between my system and what OVH expects?

You can use the endpoint https://ca.api.ovh.com/1.0/auth/time that returns the current time, from OVH point of view. It is use in the other wrappers to compute the time drift as I was referring just above.

3 Likes

Okay. Let me see what I can do

I feel it's worth mentioning that NTP should have no problems synchronizing your clock to sub-second accuracy. If you are consistently off by 30 seconds (much less 3 minutes) then something is seriously wrong with your NTP setup.

6 Likes

Which is why I asked for a basic way to compare time for my testing, now that it has been answered I already mentioned above I was looking into it.

How do I know the NTP servers I'm using, mentioned earlier, are the same ones OVH is using? How do I know for a fact that OVH's time is accurate with NTP and not just a guessing game between my system and OVH?

From the moment I looked at the time on the server and viewed the link shared to answer someone's question about it, it appeared to be 30 seconds.

I'm not looking to get schooled in NTP-101.... I'm just curious why this API appears to be so strict, within minutes if not sooner.

Atreyu

Note: I have no knowledge of OVHs systems or APIs, so I cannot state their reasoning.

That said, the usual reason why APIs reject signed timestamped requests is replay protection: If an attacker gets their hand on a signed request, they can only use it for a limited time before it expires, limiting attack surface.

6 Likes

Many protocols require accurate time synchronization, in order to help prevent replay attacks.

I know offhand that the AWS API requires signatures to be within 5 minutes. I think Azure, GCP, etc. are similar, but can't find documentation saying so quite so easily. Kerberos (used by Windows Active Directory) needs all the computers to be within a few minutes as well.

It's a pretty common thing, and systems with incorrect time generally will have many problems interacting with other systems.

8 Likes

I'd have to agree here.
To me, anything less than 1 hour off [+/-] should be "good enough".

3 Likes

Thank you for this explanation.... will run tests and circle back here

3 Likes

Like Peter said, to prevent replay attacks in security protocols a very common timestamp acceptability interval is a low number of minutes.

In a perfect world all NTP servers should be synchronizing to a common, universal time source. You can verify this is the case by using a locally trusted time source (any modern cell phone should automatically have the same accuracy) and compare that to the local time on the computer in question.

I was curious and checked the time on the OVH servers using the above mentioned API endpoint; it matches the time on my local computer (which is also synchronized via NTP) to the exact second, so I am confident that the OVH servers have the correct time.

I did take a look at the NTP configuration you posted, but I am unfamiliar with NTP configuration on Windows. The information you showed looked more to me regarding how NTP is configured, not the current status. On the systems I am involved with it is possible to ask the system if it can talk to the NTP servers and what is the current time offset is from those servers. I do see you are configured using one of the NTP pool servers which is usually a safe bet, but that does mean you are trusting some random NTP server on the Internet, and sometimes those can go away or be misconfigured.

5 Likes

What shows?:
w32tm /resync

4 Likes

As an anecdote, I have seen VMWare ESX (specifically) regularly time drift in the past. The root cause there was the electricity supply (which was an industrial generator) had a slightly different Hz and messed with the host time keeping. ESX is a linux host so there is specific config there for the VM hosting. The fix (within windows vms) is to NTP sync very frequently: W32time settings fail when starting Windows Time Service - Windows Server | Microsoft Learn

4 Likes

It is not strict below the minute.
But, from the access logs I shared with you, you were 3 minutes drifting from universal time, which is greater than what our system accept. But, to be honest, 3 minutes drift is way to big to be just some NTPs servers not perfectly sync. A few seconds would be acceptable, a few minutes not.

Romain

5 Likes

@Only1Atreyu You can see the status of the peers of your NTPd with the sudo ntpq -p command. It will show you, among other things, the offset (in milliseconds) between you and the peers.

1 Like

By default, I don't think Windows speaks "sudo" nor "ntpq".

4 Likes

Uch, Windows :scream: Forgot that. I'll run away fast then, nevermind me, carry along now, good luck :slight_smile:

5 Likes