Tuesday, April 26, 2011

Chasing the Elusive Segmentation Fault, aka The Rabbit Hole, Part 2

The Analysis

I managed to get a crash file from a child Apache process. I informed one of my co-workers, a developer who participates in the IT side of things (although he claims he doesn't like it), and asked his advice about how to use it. The actual dump data was in hex64, so not readable to me. I assumed he knew of some "development" tool that could be used to read it. I was prepared to use gdb given the standard core file that I had been expecting, but wasn't sure what to do with this. He took apport one step further and found apport-unpack, which takes the file apport creates and separates it out into proper dump files and associated information. With that I was able to run gdb against a dump file and get on the road to identifying the problem.

Right away running gdb /usr/bin/apache2 against the core file showed that the issue was with libapache2-mod-php5. Not shocking, but at least now there was proof. The next step was to figure out exactly what was happening. bt didn't give us much more than that. The hunt was on again.

I went back to my old pal at http://www.omh.cc/blog/2008/mar/6/fixing-apache-segmentation-faults-caused-php/ and skipped the whole section about putting Apache into single process mode and jumped right into his steps for using gdb to debug PHP. I created a file, gdbinit, and ran the command

(gdb) dump_bt executor_globals.current_execute_data

This essentially did nothing. It complained that dump_bt was an undefined command. I searched on this, and eventually found my way to a gdb guide. Apparently gdb reads gdbinit from your home directory first and then from the working directory (if it's different). That didn't seem to be working, but you can also point it to a file using the source command, which is what I did. That worked beautifully.

Well, almost.

Now gdb is reading gdbinit properly, but it's telling me "Attempt to extract a component of a value that is not a structure".

Have I mentioned that I'm getting a little weary of all the Googling?

I come across this guy's really cool blog (I am now a fan) and try a few of his suggestions, but I'm getting nowhere. At last long I have a stroke of inspiration. Maybe the issue is that I am missing debug symbols. I'd already had a go-round with debug symbols as they relate to Apache. The instructions I'd found in /usr/share/doc/apache2.2-common/README.backtrace had indicated that I needed to install apache2-dbg, but I found this package to be completely unavailable via package management and couldn't locate it anywhere online for manual installation. I decided to try my luck with the PHP5-dbg package, and it was in fact available and downloadable. Unfortunately, it didn't work. They weren't loaded according to php -i, and it didn't remove the error message I was getting in gdb. I found my way to this article and installed the libapache2-php5-dbgsym package according to the instructions there. I did this on my virtual server, not the live server.

I took the captured dump from the live server, transferred it to the virtual server, run gdb and successfully found the exact lines in the PHP code that caused the seg faults. Now it's up to the developers to figure out why the code is trying to access inappropriate areas of memory.


  1. Hi
    I am also wokring on apache segmentaion fault caused by php.But new to this.
    I stopped over here.

    (gdb) dump_bt executor_globals.current_execute_data
    Undefined command: "dump_bt". Try "help".

    How did u fixed this undefined command dump_bt.

    Thanks in advance

  2. Hi DC-

    I believe you have to download the appropriate gdbint file from the http://www.omh.cc/blog/2008/mar/6/fixing-apache-segmentation-faults-caused-php/ (or you can use the source code for PHP as well), and then you actually have to point BT to it after launching it by typing "source /path/to/the/file".

  3. I've tried everything so far and pinned down the problem to a function in a file that worked fine on a previous server.

    I can't see anything wrong with the function but the backtrace insists it's the problem.

    Any idea?

    if( ! function_exists('__autoload')) {
    function __autoload ($class) {
    // Array of folders to look in (ordered by preference)
    $folders = array(

    foreach($folders as $folder) {
    if(file_exists($folder . '/' . $class . '.class.php')) {
    include_once $folder . '/' . $class . '.class.php';

  4. rsmarsha, I'm, afraid I don't know a ton about PHP itself. However, if you're telling me that the code worked fine on an old server and is segfaulting on this new one, my first thought is to check for an updated version of the PHP module. Unless everything on your new server is the same version as your old server, I would look into that as the problem. HTH.