Tuesday, November 16, 2010

Backup And Recovery

Let me start off by saying that I hate backup and recovery. It's always been one of my least favorite parts of IT. Let's face it: it's not sexy. Plus, it's hard to get decision-makers to take it seriously and put forth the big bucks to do it properly until something goes wrong and suddenly they can't get their decade-old email back or someone's really really really really important spreadsheet is hosed. Backups also require constant attention and tweaking (what good's a backup if it doesn't reflect the latest dataset?) and backup technology seems needlessly complicated. I almost lost my mind when I encountered my first tape library. Partition what now? LT-who? And let's not even start with the differential vs. incremental vs. daily vs. normal grandfather/father/son/holy ghost holy cow!

Did I mention that I don't like backups?

My new gig however was begging for a new backup strategy. To my surprise there was no autoloader, no tape at all. And no vaulting. Backups were being done via a number of scripts and scheduled tasks to a USB-attached hard drive. Everything was reliant on this one little Seagate. No duplication or anything. Eek. The girl who hates backs up has to make a plan.

So, in keeping with my newfound resolve to not simply do what I've been doing just because it's what I've been doing, I went back to the drawing board for backups. I did know that I was not going to suggest the tape backup route. Tape is not reliable. It gets old, I've seen more than a few tape-based backups with those annoyingly cryptic Symantec errors that don't make you feel too confident about your ability to recover data at all times. Can we really trust "Completed with exceptions"?

I went looking for options for vaulting. Truth be told my previous company had been making some large moves in that direction, and I was all for it. I'm evaluating Venyu and i365 to see who'll give me the most robust, reliable solution for my servers. The weird thing for me is that the majority of our servers are Linux. I know what needs to get backed up to restore Windows…or at least I feel comfortable saying I do. C:\. System State. Data. Voila. Linux, things are a little more spread out. I can back up all of /, but then I'm backing up a lot of unnecessary stuff like /proc and /tmp and a number of directories within /var that don't need to come along for the ride. The benefit is that I make sure I don't miss random configs that are floating around the system that I didn't know about. Like say, that /usr/share/tomcat6/webapps directory. Yeah, that'd be good to have in a backup.

So I'm feeling my way around that and I think, given the small real estate that those Linux files take up (backing up / sans some of those other directories still only yielded a 2GB backup), that I'll go the "better safe than sorry" route. Admins are so very cavalier when it comes to Linux boxes as well that it makes me doubt myself. I don't think you'll ever hear a Windows Admin say, "Eh, just back up a couple of directories from C, and the data. Everything else we can put back together pretty quickly." With Linux though no one seems to be very concerned about the thoroughness of the backup selection.

The other thing I'm testing out is imaging the servers. I've been testing doing live imaging of the Linux servers (a nice feature not available with Windows). I've been following the tips at http://www.backupcentral.com/wiki/index.php/Linux_&_Windows_Bare_Metal_Recovery and using dd to create images. The book definitely makes it seem easier than it is. Maybe I just have had a bad string of luck with this as well. I'll go into details about my trials and tribulations in a later post. Now I must get me to some network reading. You can never revisit networking skills too often imo. 

No comments:

Post a Comment