A Saturday Morning Ring! Ring!
Me: Hello.
Caller: Hello? Dan? This is Terry.
Me: Morning, Terry. What’s up?
Caller: Man … I am so f&^%*$, right now. My server is like empty. All my files are gone.
Me: OK, Terry. I’ve got you on both the cloud and the local backup solution. Hang on a second. Let me check the service logs to make sure the backup completed correctly.
Terry: Dan, please, just get over here. Help me, please.
Me: Relax, Terry. We got this. The service logs show that the local USB backup and the cloud backup both completed without errors like they were supposed to. So we have your files. We can get them back on the server, today.
Terry: Oh, thank God! You are a lifesaver, my friend!
Me: But, Terry, shouldn’t we find out what happened and how those were removed? So we don’t have to go through this, again?
Terry: Yeah, sure, of course. How did it happen?
Me: The entire folder structure is gone, not just the files. It looks as though someone just deleted your files. Let me work a little magic here … OK. The folder was last modified about 3:15, this morning. And now, let’s see who was logged in at 3, this morning … It was you, Terry. Your account was active, on the server.
Terry: Dan, I was NOT on the server, this morning. At 3, I was in bed, sound asleep.
Me: Then that means that someone who knows your credentials logged into your network.
Terry: Clint! It HAD to be Clint! When we fired him, he swore he would get even with me.
Me: And this is EXACTLY why I recommended a full security audit of your environment when you hired me for just your backups. I don’t know Clint, but this certainly has me concerned about your entire network.
Terry: OK. Do it. And I want logs of who did it, when and from where.
Me: OK, Terry. I will get you all I can in that regard.
Terry: I want to hang the SOB who did this!
Me: Who did what, Terry? Who erased your files or who did not follow a good protocol after terminating a known vengeful sysadmin?
Terry: I got it, Dan. I’m ultimately responsible. You have been telling me for months I need to get my security tightened up. And I have not been listening.
Me: OK. Here is what I’m going to do for you, then. 1. I’m going to sweep the server for software that should not be on it. Then I’m going to scour the logs for who did what, when and from where. Once I have that, I will start the restore of your data and move on to looking for the logs on your router and such.
Terry: Thanks, Dan. Please get me back up and running as soon as possible.
Me: Glad to, Terry. But I need you to do something for me, while I work on this for you.
Terry: Anything, Dan.
Me: Go home. Relax. Put this in my hands, let me resolve this for you. I will keep you updated every couple of hours on the progress of the restore and what I find.
Terry: You know I can’t jus…
Me: Terry, It will go A LOT smoother and faster if you just let me do this and not sit on my shoulders watching everything I do.
Terry: But, I need to …
Me: Right now, you NEED to do nothing but get out of my way and let me do my work to get you back up and running as quickly as possible. With all of these files missing, Terry, there is very little you CAN do, today, other than send a few update e-mails and such. So, do that from your laptop, at home and let me do what I need to do to get your business back online AND your network locked down, correctly.
Terry: I need to …
Me: Terry, three weeks ago you were telling me how you needed to spend more time with your wife. How you needed to be more spontaneous. Remember? Today is a perfect day to do that. Go surprise your wife. I will text you with updates and call if I need anything. I have this, my friend. And I will do everything that is needed to get you back up and ready for business.
Terry: Are you sure you can do this?
Me: Yes, I am certain.
Terry: Do you need into the office?
Me: I should not need to go in. But if I do, I will text you for the door lock code and the alarm code. Let’s not worry about that, until we need it.
Terry: OK, then. I’m trusting you to not let me down, Dan.
Me: OK, Terry. I got you covered.
All went well and I called Terry a handful of hours later to let him know.
Ring! Ring!
Terry: Hello?
Me: Hi, Terry. This is Dan of Indy’s I.T. Department. I’ve got some good news and some not so good news.
Terry: I KNEW I should have gone into the office to supervise you!
Me: Terry I said ‘not so good’. I did not say I had bad news. The good news is that I have all of your data restored. The not so good news is a bit longer to explain. First, someone installed a keystroke logger on the server. Second, the router was set up for VPN connectivity. Whoever connected to your network did so through a VPN tunnel and then used your credentials to delete your files. The machine that was used to connect to your VPN was masked through a service specifically used for hiding behind. They were connected for a long time, so I don’t know if they copied any of your data or not.
Terry: So … what does that really mean?
Me: It means that I am unable to PROVE who did this. All we have, really, are details of what happened, and when, but not by whom. So, there is no real legal action you can take against Clint. IF this was Clint.
Terry: Do you know Clint?
Me: I do not believe so.
Terry: OK. That’s what I get for hiring a kid. I’m sorry, Dan. I should have listened to you when you first helped me with backups. Have you gotten rid of the rest of his tricks?
Me: I believe I have gotten them off of the server. I do not know about the workstations, though. My management tool is not on them, so I can not access them. BUT, I configured the server to background install that on each workstation, on Monday, when your staff comes into work. I need you to go through and make sure every machine is rebooted and someone in the domain logs onto each machine. Then, on Monday evening, I will scan each machine and check them for spyware and the like.
Terry: Thanks for doing all this. But how much is this going to cost me?
Me: Terry, I don’t know, yet, because I’m not done. But I can assure you It will cost less than your business is worth and more than a cup of coffee.
Terry: That is not funny, Dan.
Me: Terry, this morning you called me with a big problem. I have fixed that problem and others you were unaware of. Together, because you listened to me previously about backups, we have saved your business. Saved your livelihood. And, I can guarantee you this: I will not invoice you as much as this work should rate out at. Because I’m going to give you a quote, on Tuesday, for my full managed services AND some hardware and software we need to put in place. And you are going to sign both. We will work out a schedule for the new hardware and software so we don’t break your cash flow and still get the new assets in place to protect you. One of those will be a new router so we can give you better security. The Netgear router you have is fine, but it is old and no longer supported by Netgear. Also, you have at least three computers running Windows 7. We need to upgrade them to Windows 10.
Terry: This is going to be expensive, isn’t it?
Me: As compared to what, Terry? How many years have you built up your business? How many people do you have working for you that would be in a different situation, were it not for how you take care of them as more than an employee? Not just any employer will give an employee a $2500 gift to be part of a down payment on a house. You do good things, Terry. And I will not abuse you or your trust.
Terry: OK, Dan. Let’s talk more, on Monday, OK? I want to sleep on this.
Dan: OK, sounds good to me.
Forward to Monday Morning
Ring! Ring!
Me: I.T. Department, this is Daniel.
Caller: Hey, Dan! This is Terry.
Me: Good morning, Terry. How can I help you?
Terry: I just want to ask, what did you do to our network? It is screaming fast, now!
Me: I just cleaned a bunch of garbage off the server. Services you did not need but are installed by default from Microsoft. Also, I changed up the DNS order in the DHCP service. Which is a complicated sounding way of saying the server told each of the workstations to look for it on the local network first, then the public internet, if not found.
Terry: OK. I will take your word for it because that makes no sense to me. Also, I’ve got every workstation rebooted and logged in. Everyone is commenting on how much faster files are opening and even print jobs are going through faster.
Me: Great. I’m pleased to hear that productivity will be up.
Terry: Yeah, I guess it will be, now, won’t it? This is great, Dan. Be sure to get me those quotes, tomorrow. I will sign them, right away. If I can afford them.
Me: Terry, You will be able to afford them.
Terry: OK, Thanks, Dan. See you tomorrow.