Friday, May 6, 2011

Replicating MySQL

We're currently running with one MySQL server. It's a pretty big deal to fix that and get some kind of replication going. I'd initially investigated using MySQL's built-in master-slave replication setup. This worked fine and I had no issues setting it up at all, but it wasn't the desired solution due to fact that failover didn't occur immediately and automatically. It still required the intervention of an administrator and more importantly, it required downtime. One the developers in my office had been working with using MySQL-MMM as a solution and passed that info on to me.

MySQL-MMM is essentially a set of scripts that almost virtualizes MySQL replication. It uses a set a virtual IPs for read and write queries, and moves those IPs around your "cluster" as needed. There's even an extension for Nagios that allows you to monitor the status of your replication.

There's decent documentation out there on it, not the least of which is on their website, There are a few things lacking in the details. For example, it's not always clear whether the instructions are asking you to do something only on the database hosts or if it needs to happen on the monitoring server as well. The version I'm using, version 2.2, is also still new enough that the amount of documentation outside of the website and user experiences reported are low. The FAQ has 3 questions in it, none of which were common things that I'd run into in my setup. The manual also has a number of sections that are still awaiting actual information (noted with TODO tags). However, I did find some other useful information at Packt Pub, a Google Groups post that hipped me to the fact that if I didn't need to split my reads and writes, I didn't have to, and a video presentation on the technology. Quite frankly I didn't make it all the way through the video even though I wanted to; the content seemed like it was worthwhile but the delivery was shaky and awkward and quite distracting. I'll be giving it another go though.

Everything is installed now, and we're off to testing!

Monday, May 2, 2011

Updating PHP From Source

So, as much as I wanted to be able to simply throw the Apache segfault problem on the development team and let them handle it, alas 'twas not meant to be. After reviewing the code and attempting to reproduce whatever issue it was supposed to be causing, it was determined that the issue was more likely related to the version of PHP and not the code. I mean, the code is still funky, but ultimately it isn't what's making Apache segfault; it's the PHP side of things. My challenge then was to upgrade PHP.

Ubuntu, like most distros that offer packaged versions of software, doesn't offer the latest and greatest versions. PHP 5.3.2 is the latest in the Ubuntu repositories, but 5.3.6 is available from The only way to get to that version is to install from source. Now, this whole scenario leaves me queasy and unhappy for three main reasons:
  • Installing from source means I can't update PHP along with everything else. It's all new, all the time. 
  • We don't actually know what the problem with the PHP version is, i.e. what about it is causing segfaults, therefore there's no way to know if updating it will actually solve the problem.
  • If an update to PHP itself is indeed what caused the issues in the first place, who knows what updating it again might break? 
But, it's the best lead we have so far so might as well get to it. Luckily I also still have this image of the server to work with so I can do my best to test everything in dev prior to unleashing this on our client base.

First thing I had to do was figure out how exactly I could "upgrade" PHP from source when it was installed from package. First thing I did was go to I have to say I found this site to be a little difficult to work with. It assumed a lot of prior knowledge. For example, the two things I really wanted to know were the minimum version requirements for Apache (since it's up to 2.2.17 and we're using 2.2.14, same reasons as above), and what modules were compiled in by default. Just like Apache takes some things for granted if you don't pass it any arguments at compile time, I was certain PHP must do the same, but I could find nothing online to tell me what the defaults were.

The beauty of a development environment is that eventually you just say to hell with it and bite the bullet in a way that you would never (or should never) do on a production server. I downloaded the compressed file (another long story on that one), unpacked it to /opt/src, and ran ./configure. I used a hodgepodge of advice and instructions from various sites:,, and In the end the install worked pretty well without too much trouble save for a couple of dependencies that I had to resolve. However, it had not loaded the new Apache module, and that was my fault entirely. I'd left out the --with-apxs switch because apxs wasn't installed. Apache was installed from repository and this wasn't included. I thought I would be able to get away without it, but I really did need it. I installed the apache2-prefork-dev package without fuss though and there it was in /usr/bin/apache2.

I ran ./configure again and everything completed. I restarted Apache and got an error:

* Restarting web server apache2 * We failed to correctly shutdown apache, so we're now killing all running apache processes. This is almost certainly suboptimal, so please make sure your system is working as you'd expect now!
... waiting apache2: Syntax error on line 215 of /etc/apache2/apache2.conf: Syntax error on line 1 of /etc/apache2/mods-enabled/php5.load: Cannot load /usr/lib/apache2/modules/ into server: /usr/lib/apache2/modules/ undefined symbol: OnUpdateLong

Googling gave me the answer to that. Turns out you can't just run ./configure again; you have to either remove the entire directory or use make distclean if you're recompiling this way. Thanks to this link for that save.

Now I'm off to investigate the myriad modules available to be compiled in and determine for once and for all what it is I need. Then we test in-house for a day or so, and then we move it on to the production server.