Thursday, November 1, 2012

Aerospike (formerly Citrusleaf)

I was asked to deploy 3 development servers to run Aerospike, formerly Citrusleaf. I was, as always, psyched to have an opportunity to learn something new. I got that and then some. Mysteries upon discoveries upon more mysteries. I could have simply sneaker-netted the install since we're only talking 3 servers here, but the idea was to get it as automated as possible so that when it comes time to roll this into production and we're spinning up 30+ servers, we're not standing in front of KVMs and crash carts all day.

So, where do I begin?

Let's start with the hardware. We use Kickstart and PXE boot to provision new servers. This was already set up by our senior admin so the only thing I needed to do to get the base image installed was edit an existing config to match the layout for the servers I was to provision, and add the new image to the boot menu in Cobbler. That was simple enough but the first install I did failed at the step where the drives were partitioned. The servers have 8 SSDs, and I only wanted to install using the first 2 (I'll explain why later). Our Kickstart config creates 2 RAID 1 pairs using a portion of each disk for the OS and on top of that we do LVM.

I did a fairly simple edit where I specified sda and sdb as the disks for the OS RAID, and left the remaining disks alone. The relevant part of the Kickstart file looks like this:

bootloader --location=mbr --driveorder=sda,sdb,sdc,sdd,sde,sdf,sdg,sdh --append=" rhgb crashkernel=auto quiet"
part raid.0011 --size=500 --asprimary --ondrive=sdc
part raid.0012 --size=1 --grow --ondrive=sdc
part raid.0021 --size=500 --asprimary --ondrive=sdd
part raid.0022 --size=1 --grow --ondrive=sdd

This is actually the edited file because what I hadn't realized was that sda and sdb being the OS disks is the standard I'm used to mainly because my installs have been manual, allowing me to select the disks I want to use. Or, I've worked in systems that only had two disks so there was no issue. In a system with 8 disks however, if you're trying to specify the disk to use for the install and not use all of them, you have to be aware of the fact that the system may initialize disks in a different order than alphabetical. This was something that the senior admin had seen with another set of servers that also used SSDs, and he advised me to change the config as above, specifying sdc and sdd as the first two drives. It worked, though I am still not entirely sure why.