Again, a worthy idea but I can find neither software by name or process in the server running that should not be there. Not saying it is impossible, but it is unlikely.
Another clue. Points back to an auto-renew timed task of some kind some where.
Most of the recent time stamps were 17:00 UTC exactly apart from the Dec27 which was the unusual fix for the expired cert (Log time: 20:56:35 UTC). One was 18:00 UTC but perhaps daylight savings related?
I also realized 56 divided by 7 is an equal 8. Looks like someone setup a timed task that only renewed a cert during a weekday. I added day of week to my earlier post of dates and all were Mon or Tue except the most recent was Thu
5 Likes
If it is an automatic process we cannot account for the anomalies.
If it is not an automatic process we cannot account for the regularity.
That speaks to me of an automatic system that is not perfect and still needs human intervention at some point. In asking around, I found out the company was still adjourned for the Christmas holiday for the 27th with nobody in the office. That strongly points to something outside the company. Nobody around and no restart means SOMEONE kickstarted something.
No matter what that something was, it provably wasn't internal to the company.
Good find. The timestamp for that was about 21:00 UTC which I believe is about noon your time. So, if not in your office then could it have been someone working from home? This only shows when the cert was issued not necessarily when it was applied to your server.
How do you explain someone outside getting a cert that your IIS server is using? The cert must be accessible to it. Does anyone have such permissions into your server?
5 Likes
Hi,
I'm the developer of Certify The Web (for Windows). If the acme client is running on the same server then it's statistically most likely that your client is either:
- win-acme (a command line tool look for a C:\ProgramData\win-acme data folder)
- Certify The Web (GUI, it will just show up in all programs, or you'll see the files under C:\Program Files\CertifyTheWeb, it also as a Certify.Service background task running under Services)
- Posh-ACME (powershell)
- Certbot (which would typically have C:\Certbot folder)
There are several other tools it could be (le64.exe etc), but these are the most likely ones.
Other possibility do include remotely acquired certs pushed to the server, this would most likely be either:
- deploying using Centralized Certificate Store(CCS), where certs are copied from somewhere else as PFX files to a share and IIS loads them from there - such bindings would have "Use Centralized Certificate Store" checked in the IIS bindings editor. This is more common in server farms where multiple servers might need the same cert.
- using powershell to install the cert to the certificate store and update the IIS Binding (there would likely be windows event log entries at the time of the last renewal).
9 Likes
On the topic of not being able to find the certs in the certificate store, if they are stored they would be found via Manage Computer Certificates (certlm.msc) under Personal > Certificates or Web Hosting > Certificates
If instead they are using centralized certificate store (a file share) then they won't be in the local machine store at all, they'll either be on a local share or a share on another machine (see IIS > Centralized Certificates> Edit feature settings..)
7 Likes
@webprofusion is wise. Lots of good info there. For Posh-ACME, there wouldn't be an add/remove programs entry because PowerShell modules don't typically have installers. But if it was installed system-wide rather than in a particular user's local profile, it would be one of these two folders depending on the PowerShell version they were using:
%ProgramFiles%\WindowsPowerShell\Modules
%ProgramFiles%\PowerShell\Modules
If it was installed in a user's profile it would be in either of these folders:
%USERPROFILE%\Documents\WindowsPowerShell\Modules
%USERPROFILE%\Documents\PowerShell\Modules
But if they were using Posh-ACME in an automated fashion, I would've expected there to be something referencing it in the Task Scheduler.
This is ultimately what I'd do. Use the time remaining before the current cert expires to setup a new automated certificate config with whatever client you like. Then the expiration pressure is off and you can go back to cleaning out skeletons from closets. Being able to let existing cert expire should also at least tell you whether there's still an automated process out there other than your own.
7 Likes
I'm just chiming in to clarify some things that people already shared above...
There is no real interaction or account registration with LetsEncrypt.
The ACME API service generally works like this:
-
You run a client - manually or automatic - that will create it's own AccountKey and register it with the API. An email address is required, but the only things that will be sent to that email are automated "Your certificate is about to expire" emails.
-
The client will obtain/renew certificates by performing challenges over HTTP, ALPN or DNS. DNS is the only method that can generate a wildcard certificate. DNS challenges can be delegated to other systems; HTTP challenges can be redirected.
The ACME clients do not need to be run on the webserver. They can be run on any machine. It is common for DNS challenges to be run on an "office machine", and the certificates are then loaded manually (or scripted) onto the relevant machines. It is less common, but not uncommon, for people to manually place HTTP authorization files on the webserver.
That being said, if you do not find evidence of an ACME client on the machines in question - they probably were run on other machines.
I would personally do the following in your situation:
- [Already Done] Look for evidence of an ACME Client
- Audit login credentials, installed ssh keys, etc. Remove anything that shouldn't be there. If there is an automated task on some random office machine, that would prevent it from affecting the server.
- Install a new LetsEncrypt client on some machine, either the production one or an office machine, get a new certificate, and set up / document an automated renewal policy.
If you can risk a few moments of downtime, I would do what @rmbolger suggested and see what happens if you let the certificate get very close to expiry - or even a few seconds past the date - just to see if some automated system SOMEWHERE picks this up.
You can also use crt.sh to monitor your domain and see if some other service has obtained a certificate. That won't tell you where the service is, but it will tell you that some service somewhere is indeed configured and running.
5 Likes
Again, great idea but there was no one working remotely.
The only way I think this could be happening from outside is if one has been granted authority to manage our DNS, which one company certainly has been granted that authority. That is what I cannot understand - if they are managing the sub-domain as far as SSL, how could they NOT be running the show at the top level? I checked once again, and they are still absolutely adamant there is no way they are renewing our SSL certificate at the top level.
So far as permissions go, it's me, myself and I for all practical purposes. I just don't see how anyone else could be doing this remotely or otherwise as when I took over I changed all possible admin passwords.
1 Like
Thank you very much for the benefit of your wisdom.
Sadly, the only remote possibility would be a powershell script, but if it is there I can't find it. Searching for the usual suspects (.ps1 files) comes up with nothing. I will try combing the logs, but I have to confess I don't hold out high hopes for it being successful.
1 Like
As mentioned before, I did a search of every certificate in certlm just to ensure I didn't miss anything. There isn't a single certificate that - even taking into account a few days slop factor - remotely matches up to the time frame needed. They are either long expired, or cast into the far off future.
I am at the point of giving up, as it would seem - except for searching the logs for a crumb that may not exist - we have done a very thorough job of eliminating the possibilities. Were I to be paranoid I would change the server passwords again, but the rational side of me says - C'mon! Who would take the time to crack a 32-digit alphanumeric password for THIS level of hijinks? It just doesn't make sense.
How did you receive that notice?
Who did you receive that notice from? (Phishing possibly?, spam)
What did the notice say?
2 Likes
Thank you for that, @rmbolger ! Another tree missed for the forest. Seriously, another brilliant idea.
Sadly, still another dead end. Scoped out everything in those folders and they all track back to bone stock server installs. I though maybe we really had it this time, but my hopes, dreams and aspirations once again collided on the cold, jagged rocks of reality.
I do think it nearly time to go the new client route as I think we have more than beat this dead horse into glue.
1 Like
I do think that has been well documented by the staggeringly brilliant folk that have tried to help thus far. We may quibble a bit on the details, but I am convinced this is a semi-automated system that is going wonky from time-to-time, and needs kicked in the pants by a really-real hooman bean to get it working correctly once again. That hooman bean is not in our company, and it is not an automated service running on the server wherein our self-hosted website originates.
The absolutely maddening part of it all seems to be the magical nature by which this process was set in motion in the first place. Apparently, nobody did it and the system started completely on its own. There isn't one byte of evidence internally and the most likely suspects shout their total innocence to the heavens. It continues despite the change of all admin and admin-adjacent passwords to 32-digit alphanumeric passwords that should only be known by me.
Logically speaking, every bit of this is impossible - yet here we are.
I should think I will obtain permission from the CEO to just ignore the Godzilla in the room and just go our own way and probably with a service we pay for just so we can have some auditing/tracking ability in there somewhere.
I agree you are near the point of giving up and just starting fresh with a new client
Two things ... Did you check your WordPress config thoroughly? There are plug-ins for that which can do certs. That doesn't explain why you can't find the cert on your server though. Which leaves me to ask ...
Are you sure you are looking at the right server? Your DNS points to IP 206.174.43.52. That IP should be the one with the cert. You could use routing from there to an internal IP but then you'd have to check that routing. Whatever terminates the HTTPS connection must have that cert available to it (repeat: must).
You could double-check that by placing a file in the web root folder and checking that you see it with https://adaktu.net/mytestfile.html (for example)
4 Likes
You're not going to do much better with a pay service. They typically won't be able to share any info on the account or contact info, as that can be a security breach.
5 Likes
It was from a security plug-in newly installed to our WordPress.
It is a very long, very tedious journey how that came to be but suffice it to say our SSL cert install is... ummmm... not up to snuff. The plugin was downloaded to correct the errors, of which I noticed there are still DNS errors that need cleaned up, but that's another story.
It quite simply said your SSL Certificate is coming up for renewal on xx date. That's it. So it was nothing external if that is what you are fishing for at this time.
1 Like
I would like to think I did, given that no such plugin was installed before I installed it.
I will chase down the rest you mentioned, but again - not holding high hopes at this point in time.
2 Likes
Oh, the bitter irony!
So just to see what would happen I used the plugin (Really Simple SSL) to see if I could just go ahead and generate a new SSL certificate and be done with all this, at least for three months. Guess what it told me?
The generated certificate may need to be installed manually.
Oh, I am just dying inside. It also pointed to some DNS issues, which I am following up on, but apparently we are back to the impossible. Someone or something somewhere in the universe is semi-automatically updating that SSL Certificate, but I cannot. Taking it to the logical conclusion I even checked the cron jobs and there is nothing that fires even close the 56 day schedule.
Some days you just have to laugh.
1 Like
To rule out automated DNS-01 authentication:
I would lockdown the DNS zone [no API updates].
OR
Move the zone to another DSP.
2 Likes