Friday, March 8, 2013

CentOS Installation on XenServer

So this was interesting.

At my old job we used Ubuntu Server, so when I went to play around with XenServer vs. VMWare server to test if either was a good for our environment I wound up choosing VMWare ESXi because it natively supported the version of Ubuntu Server we were standardizing on, where I recall needing to do some levels of muckery to get Ubuntu to work. My playbook therefore consisted of ESXi and Ubuntu.

At my current company we use CentOS 6 primarily, and XenServer is our virtualization platform. We have two Xen hosts set up already for testing and some production use, but I decided to deploy my own because I was looking to install the Cloudera CDH4 stack for testing and it requires at least 4GB of RAM to do. I didn't want to overload our already pretty densely populated virtual environment. I figured, "How hard can it be?" I've installed XenServer before, I've installed CentOS before. Let the good times roll.

The good times did not roll. In our actual Xen environment there are a number of templates set up already that we use when we want to deploy a vm. Here I had to start from scratch, so I downloaded the 6.3 iso and went to do the install. It crapped out on me numerous times. Each time it would start with a prompt to re-initialize the disks for some reason, and once I did that it error'd out with a message about the bootloader and that it was unable to access a required file in the repository. This happened at least twice before I decided I needed to start Googling. I didn't find any answers or solutions.

Finally I decided to just look up instructions for installing CentOS in XenServer. I came across the blog sysadmin.circularvale.com and followed their instructions for doing this. Instead of installing from disk I did a net install from a mirror provided. That actually worked just fine, but I also needed to jump through a couple of hoops to get the graphical installer instead of the text-based installer. I had to install a VNC client, startup the VNC server on the guest (which was actually a prompt during the initial installation), and connect to my vm from my client. Then I could proceed with the normal installation that I'm used to.

I am definitely making a template out of this once I'm done.

Monday, March 4, 2013

Dropped Frames on Linux Server

A few weeks ago we got an alert from Logicmonitor, our monitoring host, regarding dropped TCP frames on one of our servers. The alert looked like this:

The host Server-X is experiencing an unusual number of failed TCP connections, probably incoming connections...This could be caused by incorrect application backlog parameters, or by incorrect OS TCP listen queue settings.

What?

This server is part of a general pool of boxes that sit behind a load balancer. It was the only box on which we were receiving such errors, and it didn't appear to actually be affecting network performance. A quick check on the server using ifconfig showed that there were 3 dropped frames at the time of this alert. We acked it and moved on, deciding it was a one-off. Unfortunately the alert has continued to happen on and off since then. It was put on the back burner because we had higher priority issues to deal with, but eventually I had to come back to it and see what the deal was.