Wednesday, November 27, 2013

The Long and Dirty Story of Me and Perforce (with a cameo from our friends Backup and VMware)

This will read somewhat as a comedy. Considering how much time I have spent on a task that was meant to be trivial (or at least should have been trivial)...well, guess I am laughing too, but it's more of those shaking-my-head-in-dismay laughs.

I was tasked with making an image of our Perforce server. Okay, I've done this before. I've used Mondo Rescue in my previous job to do just this very thing, so I'm not anticipating trouble. First thing's first. I log in to get the lay of the land because this box isn't/wasn't under the jurisdiction of Ops, and I've never been on it before. I check it out and find that it's running RHEL 4. Oh dear. Our other boxes are CentOS and at least 5, so this machine has not seen love in a long time. Out of curiosity I start poking around, checking out the specs, wanting to know what kind of hardware we're dealing with. Dmidecode tells me it's a virtual machine. Not only that, but VMware. We're running XenServer now, but no worries. We still have a couple of vSphere Clients installed on our limited supply of Windows boxes. Looks like we have one VMware host in the data center, so let's hit it.

Except...the perforce machine isn't on that host. Oh dear again. That means we have an undocumented VMware server somewhere. To the Googles!

First try was looking to see if there's any way to ascertain a hosts's name/IP from the guest itself. No such luck. Apparently this functionality is locked down for security reasons. Next stop: this fellow wrote a handy little VMware scanner tool that pings your network for live hosts and then makes a call to VMware's API to test if the other end of the ping is indeed a VMware server. Brilliant. Found the little sucker easily with this tool. IP in and I went to log in...and found that none of the standard passwords worked. Montage of me trying every single password we have in our password file until I stumble across the one that works. Very efficient, I know.

Finally I'm able to connect to the host. At this point I figured I could simply clone the vm in question and call it a day. I could even use the cloned vm to work on converting it to XenServer. Here is where I run into one of the many limitations of ESXi vs the paid version of VMware. You can't clone a running vm in ESXi. You can't make a clone period. You can export to OVF, but you have to power the guest down. When you're talking about a perforce server in heavy use by your development team, not to mention a host that hasn't been powered down in who knows how long, people get nervous, with good reason. Suddenly the scope increases. Before you can power down the guest and export to OVF, you have to take a backup and verify that it works so that you can recover the data if things go south.

Did you notice that I said "take a backup"? One thing I've noticed over the years is that you will rarely come across a corporate Windows environment that doesn't have some kind of backup going on, be it Backup Exec or even the built-in backup utility that Windows servers have. Linux environments? For some reason backups take a backseat and more often than not or left to a handful of scripts scattered here and there that get written and cronned, never to be heard from again. Such was the case here. There was in fact a cronjob that was supposed to be backing up the perforce checkpoint, journal, and versioned files, but that hadn't reliably ran since December.