Friday, November 18, 2011

MySQLDump Data Integrity

Your backup is only as good as your backup. That's a silly way to say it, but of course it's true. You can have a backup running all year long, reporting no errors, but if you've never checked it and never made sure that your data is actually recoverable, i.e that your backup is actually a good backup, you are bound to be screwed.

I had a situation once where I was working with Symantec Backup Exec with a client who had GBs and GBs of data (everyone's home drives was mapped to the server with no quotas or limitations, and there were file shares scattered everywhere) using a Dell 8-slot drive and LTO4 tapes. I've rarely gotten better than 1:1 compression with tape, so that 400GB tape pretty much only stored about 400GB. We used them in slots, 2 per day, and it was a struggle to get the backups to complete successfully because they just had too much data and weren't willing to cut much out. We made some changes to their backups (I was working with one of our support center engineers who was definitely better than I was with Backup Exec and tape loaders and the whole partitioning scheme) and they finally started working. 

I went back to do a test restore about a month later as part of our standard environment checks, and found some weird discrepancies between how much data was supposedly being backed up and how much space was actually available in the library. Long story short: the overwrite protection period on the tapes had been changed to a week or less, and so the jobs were completing because they were overwriting data on other tapes...for the same week! They didn't have any complete backups; they had bits and pieces of data. Yikes!

Having kept that experience in my heart I set up my own schedule at my current company to verify the integrity of our backups. I am mostly concerned about the MySQL backups because that's customer data. People rarely turn to backups for recovery unless it's a system outage kind of scenario. I've rarely had someone ask me to go through backups to find a deleted Word doc (although it has happened). So, I set out to find ways to test my MySQL dumps to make sure they were actually completing correctly.

I was surprised to find that there are no built-in utilities in MySQL to test the integrity of the dumped files. The only solutions I've found are pretty surface. You can restore the database and run queries against it to verify that the data is there. This of course isn't a fail-proof method, especially if you have many different databases. You can also do a select count on your tables and also run mysqlcheck.Again, if you have multiple databases this can become unwieldy. Sounds like a script that iterates through the databases and runs a range of tests would be handy. Anyone else have methods for checking the integrity of multiple databases? 

Thursday, November 17, 2011

SSH with Private/Public Key Pairs

I started experimenting with Amazon EC2 recently, and was finally forced to use the private/public key authentication method with SSH. Like most folks, I have always used the standard way of logging in to SSH with a username/password pair. EC2 doesn't give you that option, so I had to figure out how to configure Putty to do this. 

I tried it on my own first because it seemed like it should be simple enough. I perused the category options in the left pane and determined that SSH>Auth must be where to indicate the private key I was using. I set this up using the key I downloaded from Amazon and was not able to connect. Turns out I missed a pretty simple step. 

According to I was missing the part where I convert the key Amazon gave me to one that Putty can actually understand. I downloaded and ran PuttyGen, which allowed me to load the key and then save it in a Putty format. That made everything else work as expected. 

Now I have to work on figuring out what version of Linux Amazon has set me up with in my Micro Instance. It looks like a RHEL-variant based on the fact that the version info is in /etc/system-release and not /etc/lsb-release as I'm used to, but cat'ing that file shows "Amazon Linux AMI release 2011.09". I also tested my package installers and it's definitely using yum and not apt-get. I suppose technically that's all the info I need to know, but I'm a curious one. I want to know how Amazon's changed it so that it's "theirs". 

Wednesday, November 16, 2011

Using Snort

One of the goals for me in my current position was to meet a number of requirements to be able to get insurance for our data center installation. The insurance form had a ton of questions, such as "Is your wireless secured? What is it using?" and "Do you have a Written Information Security Policy? How is it shared with users?". Obviously you want to be able to answer yes to most of the questions, otherwise chances are you won't get approved for the insurance, so I've been working my way down this list and trying to change our infrastructure and/or practices to be able to answer yes, within reason. This has led to investigating options for an IDS/IPS system.

I've never worked at a place that had such a system. Even for the largest company that I've worked for we didn't have anything like that, and only started tossing the idea around when PCI compliance became a mandatory goal.

As always I started my search off with Google, and solutions varied from expensive hardware solutions to open-source software solutions.

Thursday, November 10, 2011

Connecting Smartphones to Wireless Network

The wireless network has always been a challenging wild wild west of connectivity. It isn't generally difficult to secure or control, but it does tend to give people a sense of entitlement that they would otherwise lack. How many people walk into an office, whip out a Cat5 cable, and plug into an open data jack? Not many, unless there's a conference room situation with an obvious switch or something, clearly for that purpose. No one expects automatic physical access to your network, but wireless is different. Everyone seems to expect that they are owed access to your wireless network. They need to get online, they need to check their email, whatever it is. It's pretty common to have someone come into your office space and start asking how to get on the wireless. And, if you work in a typical office, chances are there are several people who could tell you what the wireless password/key is. Heck, some of them likely have it written on a Post-It note and stuck to their monitor or something. Visitors can get on your wireless network without ever having to actually talk to you, the steward of that network. Frustrating, isn't it?

So what do you do? You could set up MAC filtering, that old 2nd-level standby pal of WEP encryption that everyone realized wasn't really all that secure, but who wants that kind of administrative hassle? Adding and removing MAC addresses every time someone wants to get online...yuck. I've tried simply not giving people the wireless key. If they don't know it, they can't share it, right? You need to get on the wireless? No problem; bring me your device and I'll set that up for you. It also gives me an opportunity to check out your machine and verify that you have active and up-to-date AV installed and nothing obviously funky going on I see that Torrent shortcut on your desktop, buddy). The downside to that is that it does mean that every time that visitor comes to your office they have access to your network, at least until you change your password. We all change our password quarterly, right? Right?

Wednesday, November 9, 2011

Do I Really Need to Document Everything? Yes!!!

Here's a story to illustrate one of the many reasons documentation is so important (and why one of the first things I do at a new job is review it all and update it if possible):

A user came up to me and said, "Are we having problems with the Exchange server?" (Note: if you're a user, it's actually more expedient to just jump right in and state what your problem is rather than approaching with a hypothesis, because the next question I have to ask is, "Why? What's happening?" You can start off with what's happening and save us a smidge of time.) I wound up at his desk and sure enough Outlook showed as disconnected. A check of the event logs showed specifically that Outlook was not able to validate the security certificate for the site. He was set up to use RPC/HTTPS, a remnant of my predecessor (although I would have set him up that way too since he's using a laptop, but even desktops are configured that way). I checked the certificate information through IE and guess what? It expired today.

Oh crap.

Monday, November 7, 2011

Heavy Rain Review

I take a long time to play games. I think I pretty much have to be the slowest player in the history of the world. Also, I only just recently got a PS3, so now I have access to PS3 exclusives. Yay! Because there's a nice little stable of games I've been jonesing to play.

Heavy Rain was, in a word, awesome! I've never played a game in one day, but man, I could not put the controller down.

Friday, November 4, 2011

Ebay World Profile and Privacy

It's admittedly been a very long time since I've been on Ebay, so a lot of the stuff I'm about to mention may be old news to a lot of people. I was in the market for a PS3 recently and had cause to start using it again. A lot has changed. I frankly find the "new" interface confusing and not at all intuitive.The thing I ran across that bothered me the most though was this "My World" feature.

I Google myself from time to time (including one of my online personas ever since one of my colleagues ran across that information and decided to go on a personal fact-finding mission to see what sorts of things I get into online), and upon Googling one of my usernames found a link to a My World page for me on Ebay that I didn't know existed. It listed my username and my feedback, both given and received. I was able to see this information despite not being logged in to Ebay or Google.