HipHop PHP in Chroot

HipHop PHP to C++
If you are unaware of HipHop PHP it is a program written by Facebook developers to convert PHP into optimized C++ code and compile the result into a single binary which provides a basic and fast web server with your code emebedded:
https://github.com/facebook/hiphop-php/wiki/

You can read about building it and how it works.  My goal once I got it compiled and working was to make it secure by running it in a chroot environment, but it still needed access to the network and other data access like HBase, HDFS and MySQL.

I was able to succesfully get HipHop PHP binary (HTTP Server) running in chroot environment.  There wasn't much different then any other chroot.  You can do a quick search on Google to figure out the basics of using a chroot.

The biggest hassle in this job is the HipHop binary since it is dynamically linked to so many libraries.  The command you want to run is 'ldd hiphopbinary'.  This will create a dump of all the libraries this executable needs in order to run.

Once you have a list, you need to copy all these binaries into your new chroot environment and they have to be in the same directory structure for HipHop to find them.  I used the output of ldd and a quick Perl script to create a copy script.

Create the Directory Structure
I created a root direct as /opt/hiphop/chroot, then created the following directories:
/usr/lib64 - this is a 64bit machine so some libraries go here
/usr/lib64/mysql - some mysql specific libraries
/usr/lib64/php - some php libraries
/usr/lib64/php/modules
/dev - special files directory...read the following paragraph
/var/www/html - some static content was needed so it is put here
/var/run - the .pid file needs a home
/var/log - logs go here
/lib64 - other libraries are searched in this path
/tmp - junk folder
/bin - only /bin/sh is here
/opt/hiphop/local/lib - when you build HipHop it requires separate compiled libraries, which I put here
/etc - OS config files

Making special files
Depending on how thorough you want to be or some other special circumstances, your program may need access to special files such as /dev/null, /dev/urandom, etc.  In order to create these you need to use the mknod binary.  For example, to create /dev/urandom, go to your chroot ennvironment
cd /opt/hiphop/dev
mknod urandom c 1 9

If you need to create other special files, I found this random page for a shell script for a RAM filesystem that creates the special files:
http://linuxfirmwarekit.googlecode.com/svn/trunk/initramfs/dev.sh

Get the web server running !!!
You can start the HipHop server using the following basic command:
sbin/httpd -u httpd -m daemon -p 8080 2>/var/log/http_error.log >/var/log/access.log

Change sbin/httpd to wherever you put the binary.  Once I was sure that was working properly I put it into a startup script called sbin/start_httpd and replace sbin/httpd with /sbin/httpd.  This was done because it will be executed under the chroot context and / will actually be referring to the chroot environment.

I then took the regular Apach http startup script and slightly modified it.  I have included the relevant portion below for the start.  The final command is:
$CHROOTBIN --userspec=99:99 $CHROOTHOME $startcmd

This translates to:
/usr/sbin/chroot --userspec=99:99 /opt/hiphop/chroot /sbin/start_httpd

/usr/sbin/chroot - means execute the chroot system binary
--userspec=99:99 - this is the uid of the user I want the process to run under (obvious not root)
/opt/hiphop/chroot - the chroot binary which make this the "home" directory for the next command it is about to execute
/sbin/start_httpd - this is the startup script that actually executes the HipHop binary.  The file actually sits (on the Linux filesystem) /opt/hiphop/chroot/sbin/start_httpd, but we are chrooted so it only knows about its new home.

init.d/chroot_httpd
...
prog=chroot_httpd_api
pidfile=${PIDFILE-/var/run/httpd/chroot_httpd_api.pid}
lockfile=${LOCKFILE-/var/lock/subsys/chroot_httpd_api}
CHROOTBIN=/usr/sbin/chroot
CHROOTHOME=/opt/hiphop/chroot
startcmd=/sbin/start_httpd
RETVAL=0

start() {
        echo -n $"Starting $prog: "
        $CHROOTBIN --userspec=48:48 $CHROOTHOME $startcmd
        RETVAL=$?
        echo
        [ $RETVAL = 0 ] && touch ${lockfile}
        return $RETVAL
}
...

You can ignore the rest if you like, but I did a find on my chroot environment to show what the final product looks like with directory structure and files.  I took all the shared library files (.so) out of the list because there were about 60 of them.  Make sure you edit your etc/passwd, etc/shadow, etc/hosts and other files to remove any sensitive data.  The hosts file may need some entries for hostnames such as your MySQL database.
/opt/hiphop/chroot/
/opt/hiphop/chroot/usr
/opt/hiphop/chroot/usr/lib64
/opt/hiphop/chroot/usr/lib64/..many .so libraries here
/opt/hiphop/chroot/usr/lib64/mysql
/opt/hiphop/chroot/usr/lib64/mysql/libmysqlclient_r.so.16
/opt/hiphop/chroot/usr/lib64/php
/opt/hiphop/chroot/usr/lib64/php/modules
/opt/hiphop/chroot/usr/lib64/php/modules/apc.so
/opt/hiphop/chroot/dev
/opt/hiphop/chroot/dev/urandom
/opt/hiphop/chroot/dev/null
/opt/hiphop/chroot/dev/random
/opt/hiphop/chroot/var
/opt/hiphop/chroot/var/www
/opt/hiphop/chroot/var/www/html
/opt/hiphop/chroot/var/www/html/favicon.ico
/opt/hiphop/chroot/var/run
/opt/hiphop/chroot/var/log
/opt/hiphop/chroot/var/log/access_log
/opt/hiphop/chroot/var/log/access.log
/opt/hiphop/chroot/var/log/admin_log
/opt/hiphop/chroot/var/log/http_error.log
/opt/hiphop/chroot/var/log/error_log
/opt/hiphop/chroot/sbin
/opt/hiphop/chroot/sbin/httpd
/opt/hiphop/chroot/sbin/start_httpd
/opt/hiphop/chroot/lib64
/opt/hiphop/chroot/lib64/...many more .so libraries here
/opt/hiphop/chroot/tmp
/opt/hiphop/chroot/www.pid
/opt/hiphop/chroot/bin
/opt/hiphop/chroot/bin/sh
/opt/hiphop/chroot/opt
/opt/hiphop/chroot/opt/hiphop
/opt/hiphop/chroot/opt/hiphop/local
/opt/hiphop/chroot/opt/hiphop/local/lib < this was the local folder with libraries built for hiphop
/opt/hiphop/chroot/etc
/opt/hiphop/chroot/etc/run.conf
/opt/hiphop/chroot/etc/nsswitch.conf
/opt/hiphop/chroot/etc/log.conf
/opt/hiphop/chroot/etc/shadow
/opt/hiphop/chroot/etc/hosts
/opt/hiphop/chroot/etc/passwd
/opt/hiphop/chroot/etc/resolv.conf
/opt/hiphop/chroot/etc/httpd.conf
/opt/hiphop/chroot/etc/group