And of course the COVID-19 situation here in the UK and elsewhere will delay our in field fixes too if we have to go down that route.
2 Likes
Gotcha. That is still... a lot on our end and would still affect thousands of certs that we could potentially not reach the subscribers. I still think the best route would be to try and figure out a workaround with the community. I'll keep and eye on this thread over the holiday and ping @josh as needed too. We are all really rooting for you and did dig deep to see if we could per chance help and not affect other subscribers. It's a really hard place to be.
A few quick questions (as I am not highly technical, as stated before) - but does this hardware need a cert upon startup and that's why you can't communicate with the hardware because you cannot get a cert as you hardcoded the intermediate? And how many devices does this affect?
Would there be any way (like controlling TIME) to use a certificate that has already been issued for that device, if the cert for that device is not yet expired, or are all the certificates on the unreachable devices already expired?
If possible I want to point a few community members to this thread if I think they can help.
3 Likes
A few quick questions (as I am not highly technical, as stated before) - but does this hardware need a cert upon startup and that's why you can't communicate with the hardware because you cannot get a cert as you hardcoded the intermediate? And how many devices does this affect
It has the chain in the modem store and the session starts by requesting a TLS tcp connection to the server by dns lookup. Then it will complete the handshake and exchange and start a normal comma session. The failure happens as you say because the intermediate doesnāt match.
There are about 4000 devices across UK/Europe and Australia
The devices donāt have client certs but the server cert that expired is used by all of them to verify the server identity and also uses SNI also.
Hth
Mike
1 Like
Where do those devices get their TIME from?
Where do they get their DNS from?
Which other certs do they trust?
1 Like
They get TIME under normal ops from a time server we control in a remote data centre but we are checking if for the TLS check it is fact the modem firmware that gets the time from the cellular network and not yet in our code.
DNS is from Route53 in AWS.
We believe the modem has no built in certs they trust, only the chain we have set which has the X3 intermediate in there.
Mike
1 Like
This would be a good point to control the situation.
Do you control the DNS server they get their DNS from?
Do they use only that DNS to resolve the TIME server(s) IP(s)?
1 Like
We can manage the DNS entries but not the server as itās in Amazon infrastructure.
Thanks
Mike
1 Like
Does it have any port/service open to client?
I hope this wasn't rude maybe look up for CVE in that stack?
1 Like
Which DNS entries do the clients use for TIME?
Do all your (problematic) clients use these same entries?
Do any other (non-problematic) clients also use this same entry for their TIME?
[this will go a lot quicker if when I ask multiple questions, I get more than just one answer]
2 Likes
Just to be clear about what @rg305 is suggesting, and what I think you're talking about here too: If you adjust your time server to tell your devices the date is, say, December 20, then those devices would trust your expired certificate. That would let you get in communication with them long enough that they could download a firmware update from you.
6 Likes
Internet time-setting client software might vary in terms of whether or not it's willing to go backwards by a significant amount (because it's considered a potential sign of an attempt to try to get them to trust an expired certificate... which, of course, is the explicit goal here!), but this depends on which specific software the devices are using to get the current time from the network, because not all software will have this restriction.
2 Likes
Just rip out the CMOS batteries. I've found that causes temporal amnesia instantly and without fail. 
1 Like
You might find it funny (I do), but I run similar types of devices without their batteries.
Given: It should always work; even if the battery were to die.
Naturally this always needs to pass the lab testing before going production.
But so far, all have passed that test.
If it can work without the battery, then it can survive any power failure (it will always work).
And, yes, I only use NTP servers that I can fully control.
[although I have never had to turn back time (knock on wood), I would be able to do so]
2 Likes
No only the port we use for our comms to our backend server and the time server.
1 Like
All client devices use the same entry when itās our time server which is load balanced. The problem is that we know that the modem firmware is responsible for the TLS stack and thatās not our code. We have been in contact with them to establish just when, how and where they get TIME from too.
2 Likes
Iām not sure this is the same situation . Our devices only run from battery and if we were to pull power we would be at the device in the field and can repair the problem directly then.
2 Likes
Understood. That does complicate things - good to know.
Do you have remote SSH access (or the likes) to the devices?
1 Like
No we donāt only through the normal comms can we interact and that is the blocker as the first part is getting a secured socket
We also donāt think we can hack the time as it also subject to this successful session being established before we can inject any code to be downloaded.
We have to plan now to visit all devices in the field at the earliest opportunity at huge logistical and financial cost.
3 Likes
1 Like
They have no other built in certs that they trust as far as we know - we are checking with them on that as another avenue
2 Likes