typedef int (*funcptr)();

An engineers technical notebook

Apache mod_fastcgi and PHP with PHP-FPM

Recently I did a server migration from an older server to a newer server and in an attempt to help stability I wanted to see if there was a better way to do PHP FastCGI. In my research I came across running PHP using the FastCGI server that spins up a PHP on a TCP/IP port and allows the web server to connect to it. However this doesn't help with spawning or keeping track of instances or error recovery.

This is where PHP-FPM comes in handy. It does all of the hard work for us, it spawns the processes and has a bunch of really awesome features that help run PHP as various different users as required with different PHP ini files and memory limits. PHP-FPM's defaults, at least from a ports install, are extremely sane and I don't really suggest changing them. After setting up PHP FPM I had to set up Apache.

mod_fcgid doesn't allow for remote connections, and as such I was unable to use it for what I needed it for. mod_fastcgi provides the FastCGIExternalServer configuration key, which is exactly what I needed.

"The FastCgiExternalServer directive defines filename as an external FastCGI application. If filename does not begin with a slash (/) then it is assumed to be relative to the ServerRoot. The filename does not have to exist in the local filesystem. URIs that Apache resolves to this filename will be handled by this external FastCGI application."

What this documentation does not state is that the path up to the last part of it has to exist in the local file system. So in my first couple of attempts I pointed it at /usr/local/www/fastcgi/php5.fcgi without having an actual fastcgi directory located in /usr/local/www/. There are a lot of examples that require creating a new FastCGIExternalServer for each and every VirtualHost this is unacceptable to me, the reason they require it is because they set the FastCGIExternalServer path to the folder where they are going to be serving files from.

In the end I found that after creating the /usr/local/www/fastcgi directory (and reading the mod_fastcgi source code) that all it does is make Apache believe a file exists in a certain directory, much like Alias, except Alias allows full paths to be aliased, not so here.

The AddType and Action add custom types and what the action should be when such a type is encountered. In this case the action is to redirect the request to /php5.fcgi which will handle the rest of the request. This does not require another FastCGI section per VirtualHost as each PHP request will just get shuttled to the php handler.

Do take note that I have specifically disallowed Apache to serve anything from the /usr/local/www/fastcgi/ folder, except for a single file php5.fcgi which is our FastCGIExternalServer file.

<IfModule mod_fastcgi.c>
    Alias /php5.fcgi /usr/local/www/fastcgi/php5.fcgi
    FastCGIExternalServer /usr/local/www/fastcgi/php5.fcgi -flush -host 127.0.0.1:9000
    AddType application/x-httpd-fastphp5 .php
    Action application/x-httpd-fastphp5 /php5.fcgi

    <Directory "/usr/local/www/fastcgi/">
        Order deny,allow
        Deny from all
        <Files "php5.fcgi">
            Order allow,deny
            Allow from all
        </Files>
    </Directory>
</IfModule>

Note that even-though in my last post concerning mod_fastcgi I as moving away from it, I am now doing the opposite, instead of moving from mod_fastcgi to mod_fcgid I'm back to mod_fastcgi, only because mod_fcgid doesn't offer the same functionality.

So far this has provided far more stability, along with PHP-FPM doing all of the process management I can now use a single PHP instance that is running on a single port for the various web servers I am testing. At the moment I have both Lighttpd and Apache using the same PHP-FPM instance. It is faster, less memory is wasted and PHP-FPM is much better at process management than mod_fastcgi or mod_fcgid.

Magic Properties on Google AppEngine

Update to my previous Google AppEngine auto-updating properties post.

In my last post I was talking about:

[...] however there is still an issue, for now the MagicProperty is only updated the first time the property it affects is set

I fixed that issue, it has a local cache that is added to the model, so if the value does change, the cache would no longer be valid and the value is recreated, otherwise the already computed value is returned. There was however another issue I had to deal with, and that was the fact that the cache was not being written to the datastore, this meant that upon getting the data back from the datastore we recomputed it anyway. Now the programmer has to do some extra work, they have to add a cache unindexed property to the model, and pass it into the MagicProperty to be used as the caching variable. Within the save() they have to prime the cache by setting a temporary variable equal to the to be computed variable just in case the cache variables have not been primed, this has not yet been documented other than in my SVN history.

We still don't allow setting of the magic property, however upon returning from the datastore it calls make_value_from_datastore, which returns a _MagicDatastore, as long as we set with an instance of _MagicDatastore it will allow it to go through, so we can get the value from the datastore without having to recompute the value.

Read the code for an example on how to use this.

Code is added below, feel free to use it as you wish under the license that is attached.

###
 # Copyright (c) 2010 Bert JW Regeer;
 #
 # Permission to use, copy, modify, and distribute this software for any
 # purpose with or without fee is hereby granted, provided that the above
 # copyright notice and this permission notice appear in all copies.
 #
 # THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
 # WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
 # MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
 # ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
 # WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
 # ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
 # OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
 #
###

import logging
import hashlib
from google.appengine.ext import db


def MagicProperty(prop, magic_func=None, cache_prop=None, pass_instance=False, *args, **kw):
    if magic_func:
        # No pants required.
        return _MagicProperty(prop, magic_func, cache_prop, pass_instance, *args, **kw)
    else:
        # We are putting some pants on the function.
        def pants(magic_func):
            return _MagicProperty(prop, magic_func, cache_prop, pass_instance, *args, **kw)
        return pants

class _MagicDatastore():
    """This is an internal class for _MagicProperty."""

    def __init__(self, val):
        self.value = val

    def retval(self):
        return self.value

class _MagicProperty(db.Property):
    """MagicProperty which will modify output based on a function that it is given.

    This has several ways in which it may be called:

    In this example we create the model, no caching is done, so any data that is returned from the datastore
    will be erased the first time that the MagicProperty is accessed and recomputed.

        class MagicTest(db.Model):
            title = db.StringProperty(required=True)
            chars = utils.MagicProperty(title, len, required=True)

        mytest = MagicTest(title="It was for the good of the school!")

        >>> print mytest.title
        It was for the good of the school!
        >>> print mytest.chars
        34

        mytest = MagicTest.all().get()

        >>> print mytest.title
        Hello
        >>> print mytest.chars      
        5   # Do note, this is recalculated the first time it is called, as long as title does not change it won't be recomputed.

    In this example we create the model, and we also create a caching property so that even when we get values back 
    from the datastore we use the cached computed value rather than running the function again. Do note that this requires overriding put()
    to prime the cache as there is currently no way to specify that certain properties should be "saved" before others.

        class MagicTesting(db.Model):
            title = db.StringProperty(required=True)
            cache = db.UnindexedProperty()
            chars = utils.MagicProperty(title, len, cache_prop=cache, required=True)

            def put(self, *args, **kw):
                prime_cache = self.chars
                super(MagicTesting).put(*args, **kw)

        mytesting = MagicTesting(title="Get the pocket knife out of my boot.")

        >>> print mytesting.title
        Get the pocket knife out of my boot.
        >>> print mytesting.chars
        36

        mytesting = MagicTest.all().get()

        >>> print mytesting.title
        How are you?
        >>> print mytesting.chars   
        12  # This is not recomputed so long as the title has not changed, however it uses more datastore space to store a hash.


    Inspired by: 
    http://appengine-cookbook.appspot.com/recipe/custom-model-properties-are-cute
    http://code.google.com/appengine/articles/extending_models.html
    http://googleappengine.blogspot.com/2009/07/writing-custom-property-classes.html

    """
    def __init__(self, prop, magic_func, cache_prop, pass_instance, *args, **kw):
        """
        Extra parameters you can give this initializer.

            prop        = Property to be acted upon
            magic_func  = The function to be called when the property is accessed
            cache_prop  = The property that can hold our cache, I suggest it is an db.UnindexedProperty() since it just stores sha1 hashes
        """
        super(_MagicProperty, self).__init__(*args, **kw)
        self.magic_func = magic_func
        self.magic_prop = prop
        self.magic_cache = cache_prop
        self.magic_pass = pass_instance

    def get_cache_val(self, model_instance, class_instance):
        if self.magic_cache is not None:
            return self.magic_cache.__get__(model_instance, class_instance)
        return getattr(model_instance, self.attr_name() + "orig", None)

    def set_cache_val(self, model_instance, val):
        val = hashlib.sha1(val).hexdigest()

        if self.magic_cache is not None:
            self.magic_cache.__set__(model_instance, val)
        setattr(model_instance, self.attr_name() + "orig", val)

    def attr_name(self):
        # In google.appengine.ex.db there is an explicit warning not to use this method, so we test for it first.
        if self._attr_name:
            return self._attr_name()
        else:
            return "_" + self.name

    def __get__(self, model_instance, class_instance):
        if model_instance is None:
            return self

        cur = self.magic_prop.__get__(model_instance, class_instance)
        cur = cur.encode('utf-8')       
        last = self.get_cache_val(model_instance, class_instance)
        if last == hashlib.sha1(cur).hexdigest():
            logging.info("Cache hit: %s" % (cur))
            return getattr(model_instance, self.attr_name(), None)

        logging.info("Cache miss: %s" % (cur))

        magic_done = u""
        if self.magic_pass:
            magic_done = self.magic_func(model_instance, cur)
        else:
            magic_done = self.magic_func(cur)

        # Set the attribute in the model
        setattr(model_instance, self.attr_name(), magic_done)
        self.set_cache_val(model_instance, cur)

        return magic_done


    def __set__(self, model_instance, value):
        if isinstance(value, _MagicDatastore):
            setattr(model_instance, self.attr_name(), value.retval())
        else:
            raise db.DerivedPropertyError("MagicProperty is magic. Magic may not be modified.")

    def make_value_from_datastore(self, value):
        return _MagicDatastore(value)

Auto-updating Properties in Google AppEngine Models

EDIT: I have an updated version of this code available in a newer post, please go to Magic Properties on Google AppEngine for the updated information. For the generic information that has not changed, and the idea behind the code please continue below.

In Google AppEngine a model may contain many different properties, from this model you can generate a form using djangoforms. This is all good and well until you have a required property on a model that needs to have part of it be calculated based upon something the user types in. For example, a blog entry generally has a "slug" which is generated from the title, now in Google AppEngine's models, if a property is required it has to be set at object creation time.

Djangoforms has a form.save(committed=False) that allows one to get back an unsaved Model, however since slug is not in the form (since it is auto generated) it cannot be set at model creation time. Even adding a hidden field to the form with the name of the property is not going to satisfy the model since a value of None is considered to be empty, which means it does not meet the required=True status.

After trying a multitude of different options I ran across an article about extending models that specifically talked about adding custom properties that were saved into the DB, and could be searched on, this got me digging and I found a blog post on the Google AppEngine blogspot about the exact same thing, using yet again a different example. The one that got me on the right track was the following post by Rodrigo Moraes on his custom model properties.

I came up with the following fairly quickly after also taking a look at googleappengine.ext.db within the Google AppEngine framework, however there is still an issue, for now the MagicProperty is only updated the first time the property it affects is set. I am not yet sure how to work around that, I will have to do some more digging, maybe there is a way to get notified if another property gets modified or updated.

import logging
from google.appengine.ext import db


def MagicProperty(prop, magic_func=None, *args, **kw):
    if magic_func:
        # No pants required.
        return _MagicProperty(prop, magic_func, *args, **kw)
    else:
        # We are putting some pants on the function.
        def pants(magic_func):
            return _MagicProperty(prop, magic_func, *args, **kw)
        return pants


class _MagicProperty(db.Property):
    """MagicProperty which will modify output based on a function that it is given.

    Inspired by: 
    http://appengine-cookbook.appspot.com/recipe/custom-model-properties-are-cute
    http://code.google.com/appengine/articles/extending_models.html
    http://googleappengine.blogspot.com/2009/07/writing-custom-property-classes.html

    """
    def __init__(self, prop, magic_func, *args, **kw):
        super(_MagicProperty, self).__init__(*args, **kw)
        self.magic_func = magic_func
        self.magic_prop = prop

    def attr_name(self):
        # In google.appengine.ex.db there is an explicit warning not to use this method, so we test for it first.
        if self._attr_name:
            return self._attr_name()
        else:
            return "_" + self.name

    def __get__(self, model_instance, class_instance):
        if model_instance is None:
            return self

        # TODO: If the property that this one is functioning on is changed, we need a way to know that so that we recalculate

        # Original __get__ in google.appengine.ext.db has getattr to retrieve the value from the model instance
        magic_done = getattr(model_instance, self.attr_name(), None)
        if magic_done is None:
            magic_done = self.magic_func(self.magic_prop.__get__(model_instance, class_instance))

            # Set the attribute in the model
            setattr(model_instance, self._attr_name(), magic_done)
        return magic_done


    def __set__(self, *args):
        raise db.DerivedPropertyError("MagicProperty is magic. Magic may not be modified.")</pre>

The approach that the DateTimeProperty takes is one that I can't use because my class is dependant upon an external property, not whether or not a value was set when the object was instantiated.

Google AppEngine is posing some rather interesting challenges. Although, for slugs it may not be entirely bad that it can only be set once, as that means title changes on the article won't cause the slug to change possibly breaking old links. This definitely requires more research and more thinking.

Property class on AppEngine does not support a tuple for choices

Google AppEngine does not allow one to set the choices field of a model property to a tuple; this workaround will have have a form give the user the choices with the user friendly items but have the backend store the non-user friendly version. 

# Set up our non-user friendly and user friendly choices
REDIRECT_CHOICES = (
    (301, "301 - Permanent Redirect"),
    (302, "302 - Temporary Redirect"),
    (403, "403 - Access Denied"),
    (404, "404 - File Not Found"),
    (500, "500 - Internal Server Error")
)

# Sample model that has the required Property with the choices set to the non-user friendly values
class Shorturl(db.Model):
    uripath = db.StringProperty(verbose_name="Path", required=True)
    httpcode = db.IntegerProperty(verbose_name="HTTP code", required=True, default=301, choices=[x[0] for x in REDIRECT_CHOICES])
    location = db.StringProperty(verbose_name="Location")

# This is our djangoform that has the choices set as required so that the user gets the user friendly version and not just simply
# the numbers.
class ShorturlForm(djangoforms.ModelForm):
    httpcode = forms.IntegerField(widget=forms.Select(choices=REDIRECT_CHOICES), required=True, label='HTTP code')

    class Meta:
        model = Shorturl

Given the above tuple it will set the choices in the model for Google AppEngine and pass it to the field on the Django form allowing user friendly output. This will allow the user to pick the right redirect choice without having to know that various different redirect codes or have to look them up.

Upgrading an OpenSolaris ipkg zone

Recently I was working on my OpenSolaris machine (file server, ZFS rocks, will write more on that later), and I have one non-global zone, one that use for web development, which is aptly called dev-web.

So the following shows up when I run zoneadm:

xistence@Keyhole.network.lan:~# zoneadm list -iv
  ID NAME             STATUS     PATH                           BRAND    IP    
   0 global           running    /                              native   shared
   4 dev-web          running    /storage/zones/dev-web         ipkg     shared

The thing is, I had upgraded the global zone with the latest update available for the version (snv_101b):

pkg image-update -v

This had not upgraded my one none-global zone. And running pkg image-update from within the zone itself is not possible, because you can't do an upgrade on a "live" system, mainly because the image-update wants to create a new bootable environment, something that was already created when I upgraded the global zone. So what we have to do is mount the non-global zone to /mnt and tell pkg with -R where to find it and upgrade it anyway!

First we are going to halt the current zone, since I am mean and nothing important is running on the test bed system, I just used:

zoneadm -z dev-web halt

However, the better way is to off course use the shutdown command:

zlogin dev-web shutdown

and then for good measure a halt!

Next up, looking at what we are going to be mounting.

zfs list
NAME                               USED  AVAIL  REFER  MOUNTPOINT
rpool                             11.9G  42.7G    75K  /rpool
rpool/ROOT                        2.93G  42.7G    18K  legacy
rpool/ROOT/opensolaris            7.34M  42.7G  2.74G  /
rpool/ROOT/opensolaris-1          2.92G  42.7G  2.74G  /
rpool/dump                        1.87G  42.7G  1.87G  -
rpool/export                      5.27G  42.7G    19K  /export
rpool/export/home                 5.27G  42.7G    21K  /export/home
rpool/export/home/guest           5.24G  14.8G  5.24G  /export/home/guest
rpool/export/home/xistence        37.0M  42.7G  37.0M  /export/home/xistence
rpool/swap                        1.87G  43.0G  1.51G  -
storage                            493G  3.08T  35.1K  /storage
storage/media                      398G  3.08T   398G  /storage/media
storage/virtualbox                2.10G  3.08T  2.10G  /storage/virtualbox
storage/xistence                  91.7G  3.08T  91.7G  /storage/xistence
storage/zones                      957M  99.1G  30.4K  /storage/zones
storage/zones/dev-web              957M  19.1G  32.0K  /storage/zones/dev-web
storage/zones/dev-web/ROOT         957M  19.1G  28.8K  legacy
storage/zones/dev-web/ROOT/zbe    1.60M  19.1G   936M  legacy
storage/zones/dev-web/ROOT/zbe-1   955M  19.1G   935M  legacy

When pkg image-update was run on the global zone it created a new bootable environment named opensolaris-1, the cool thing is, that beadm at the same time will also create a new bootable environment for your zones. That way you can upgrade your zones afterwards, and if stuff does not work, you can revert the ENTIRE machine back to the previous state (ZFS is cool like that), thereby also making sure that your zones are reverted so that there are no incompatibilities.

So what we are looking for in this case is a zbe-1, this is the new root for the zone that we need to update, so we now need to mount it.

mount -F zfs storage/zones/dev-web/ROOT/zbe-1 /mnt

Note, that there is no / in front of storage, this is because we are specifying a pool name, since there is no "real" path that is defined as /storage/zones/dev-web/ROOT/zbe-1. Now that it is mounted, we are able to pass the -R flag to pkg, to get it to update our zone:

xistence@Keyhole.network.lan:~# pkg -R /mnt image-update -v
Creating Plan / Before evaluation:     
UNEVALUATED:
+pkg:/entire@0.5.11,5.11-0.101:20081204T010954Z

After evaluation:
pkg:/entire@0.5.11,5.11-0.101:20081119T235706Z -> pkg:/entire@0.5.11,5.11-0.101:20081204T010954Z
Actuators:

None
PHASE                                        ACTIONS
Update Phase                                     1/1 
PHASE                                          ITEMS
Reading Existing Index                           9/9 
Indexing Packages                                1/1

---------------------------------------------------------------------------
NOTE: Please review release notes posted at:
   http://opensolaris.org/os/project/indiana/resources/relnotes/200811/x86/
---------------------------------------------------------------------------

Voilá, and the deed is done. The last command is to off course unmount the zone, that we can then issue a zoneadm boot command to start it back up:

umount /mnt
zoneadm -z dev-web boot

And then on the zone after we log into it (over SSH in my case):

xistence@webdev.network.lan:~# pkg list -u
No installed packages have available updates

Which is exactly what we wanted! Your zone is now upgraded with the latest version available from the global zone.