Most browsers are quite liberal in what they accept, except for Safari/Konqueror. For some reason it doesn’t like it if a JavaScript file ends in .gz, it ignores the content-encoding and attempts to read the compressed data as JavaScript. The key to make it work is to create a new extension, for example .jgz and set the Content-Encoding for this extension to gzip AND to set the mime type of .js to text/javascript, not text/plain or application/x-javascript.

To recap, call the file script.js.jgz and make sure your webserver delivers it with the following options

Content-Encoding: gzip
Content-Type: text/javascript

Just include the file as usual

<script type="text/javascript" src="script.js.jgz"></script>

This has been tested with the following browsers

• Firefox 3.0/3.5
• Opera 10
• Internet Explorer 7
• Konqueror (webkit based, behaves like Safari)
• Google Chrome 3
• Epiphany

Apache

Put the following in a .htaccess file

AddType text/javascript .js
AddEncoding gzip .jsz

thttpd

Add jsz gzip to mime_encodings.txt and change the type for js in mime_types.txt to js text/javascript.

Update 10 Sep: The patches have been committed upstreams.
Update 9 Sep: working vboxnetadp and patchset sent upstreams.
Update: Re-written to use netgraph, now works on FreeBSD 7 and 8

Patches can be found at

http://www.shapeshifter.se/pub/patches/virtualbox/vbox-freebsd-netif-20090908.patch
http://www.shapeshifter.se/pub/patches/virtualbox/vbox-freebsd-vboxnetadp-20090908.patch
http://www.shapeshifter.se/pub/patches/virtualbox/vbox-freebsd-vboxnetflt-20090908.patch
http://www.shapeshifter.se/pub/patches/virtualbox/Config.kmk-20090908.patch
http://www.shapeshifter.se/pub/patches/virtualbox/src-VBox-HostDrivers-Makefile.kmk-20090908.patch

Additional patches to test with the 3.0.51.r22226 version in FreeBSD ports.

http://www.shapeshifter.se/pub/patches/virtualbox/Config.kmk-r22226-20090908.patch
http://www.shapeshifter.se/pub/patches/virtualbox/virtualbox-port-20090907.patch
http://www.shapeshifter.se/pub/patches/virtualbox/ConsoleImpl2.cpp.patch

Re-install VirtualBox through ports using the following commands

Apply/compile with
cd /usr/ports/emulators/virtualbox
make clean patch
mkdir work/virtualbox-3.0.51r22226/src/VBox/HostDrivers/VBoxNetFlt/freebsd
mkdir work/virtualbox-3.0.51r22226/src/VBox/HostDrivers/VBoxNetAdp/freebsd
patch < virtualbox-port-20090907.patch
patch -d work/virtualbox-3.0.51r22226/Config.kmk-r22226-20090908.patch
patch -d work/virtualbox-3.0.51r22226/ConsoleImpl2.cpp.patch
patch -d work/virtualbox-3.0.51r22226/src-VBox-HostDrivers-Makefile.kmk-20090908.patch
patch -d work/virtualbox-3.0.51r22226/vbox-freebsd-netif-20090908.patch
patch -d work/virtualbox-3.0.51r22226/vbox-freebsd-vboxnetadp-20090908.patch
patch -d work/virtualbox-3.0.51r22226/vbox-freebsd-vboxnetflt-20090908.patch
make install

In VirtualBox network settings, under "Bridged Adapter" you should now see your available network interfaces. Select the one connected to your network and boot your virtual machine. It should now be connected to your local network as any other machine.

Host only adapter mode can be used to create a virtual network with multiple guests, it creates a special vboxnetX adapter on the host. You'll have to do normal routing between this interface to get outside connectivity.

In addition to vboxdrv.ko you'll have to load vboxnetflt.ko and vboxnetadp.ko too.

kldload /boot/modules/vboxnetflt.ko
kldload /boot/modules/vboxnetadp.ko

1-wire: is a serial bus from Dallas Semiconductor/Maxim that only requires 1 data line, there are a number of cheap sensors and other devices for this bus. The strength of this bus is not its speed but that it supports large ranges (up to 300 meters).
Also, each 1-wire device has a permanent unique 64-bit serial number.

IPv6: Insanely large address space. It’s common to use a 64-bit netmask for site networks so that EUI-64 based addresses can be used for auto configuration. This leaves 64-bit for the node address – do you see where this is going now…

Yes..I’ve built a device that assigned each 1-wire device it’s connected to its own IPv6 address. Why? you ask, mostly because I can.

Hardware

As mentioned above, the device is based on an AVR ATmega644. It has 64KB of flash memory for program code and 4KB of RAM. It’s running on its built-in oscillator at 8MHz. The ENC28J60 Ethernet chip is connected to the AVR using SPI. The rest of the hardware is mostly for power distribution and management.

The PCB was manufactured by BatchPCB, cheap service but a bit slow turn-around time.

Unfortunately I screwed up the SPI connection but I managed to fix that with some green wires (or black wires in this case). You’ll note them in the picture above.
I also intended to run the AVR at 5V and the ethernet chip at 3.3V. This is what the quad AND-gate in the upper right
corner was for, but since I screwed up the SPI routing it’s disconnected and the whole circuit is running at 3.3V.
The ENC28J60 can only run at 3.3V, the AVR has a range from 2.8-5V and 1-wire should be ran at 5V but works at 3.3V. Hence the need for TTL voltage translation.

As for the 1-wire devices I had implemented a bus master in software that generated the require waveforms. It worked great up to about 10-15 meters. Any cable length greater than that refused to work.
This was a bit unexpected and without an oscilloscope it was more or less impossible to figure out where and how the signals got mangled. So I simply got a DS2480 1-wire line driver that generates the required signals in hardware with more precise timing.

Add-on board with a 1-wire master

This required an add-on board and because I didn’t want to wait for a new PCB I used a 2.54mm prototype board. With the DS2480 only available in SOIC8 packages it required some “creative” soldering .
The DS2480 required 5V, thus It had to get its own power supply and also required level translation on the UART line between this device and the AVR. I choose an approach using MOSFETs and a few resistors for this (the TO92 packages in the picture above). This turned out to work really good and I think I’m going to use this for the SPI level translation in the next revision of the board.
The wire leaving the board on the left side leads to the 1-wire sensor devices.

The add-on board is extremely ugly. But hey, it works.

Future improvements for the next revision

• Use of external crystal at 16MHz instead of internal 8MHz clock.
• Use MOSFETs for 3.3-5 V translation. Need to test it at 16MHz before manufacturing a PCB though.
• Obviously fix all PCB errors
• All SMD parts (resistors and voltage regulators) to shrink PCB size even more.
• Better power distribution. I was a bit too conservative with the decoupling capacitors resulting in some weird power problems (fixable with some caps)
• Create a real add-on board

I’ll publish the PCB CAD files when the next revision is complete.

Software

The only small IPv6 stack I know of is the uIPv6 stack in the Contiki operating system created by Adam Dunkel et al. This is unfortunately only available together with Contiki and not as a stand alone package as the originally uIP (IPv4) stack.

Contiki is a great operating system, but when you only have 4KB of RAM it becomes a bit heavy weight. So I broke out the uIPv6 stack from Contiki and made it run stand alone and ported in to AVR. I also ported the web server application from Contiki and made it run on AVR. As I wanted to use multiple IPv6 addresses I also had to add support for IP aliases to the uIPv6 stack.

Since the uIPv6 was integrated with Contiki it used the Contiki process model which it self is based on “proto-threads” (another thing invented by Adam Dunkel). I felt that this didn’t fit so I turned all processes into a polling mode instead. So one has to call a set of polling functions from the main application loop or from timers.

The other major parts of the code are drivers for ENC28J60, DS2480 and DS1820.

Software

1-wire devices

30 second polling interval with auto-discovery of new devices.
Each device is assigned its own IPv6 address, requires a /64 network to be available.

Webserver

Integrated web server makes it possible to visit each address. An XML file with the latest sensor reading is returned. An “age timestamp” is also provided which makes it possible to determine how old the reading is.

Currently, with 5 1-wire devices connected it uses about 3KB of RAM.

• Source code
• uIPv6 port

In-action

I only have temperature sensors connected at the moment. If you happen to have an IPv6 capable connection you can access the sensors through a web browser.

2001:16d8:ffe5:002:2894:eaf6:100:0c7
2001:16d8:ffe5:002:28c1:b4f6:100:035
2001:16d8:ffe5:002:2809:aef6:100:0ca
2001:16d8:ffe5:002:28c5:a5f6:100:058
2001:16d8:ffe5:002:2813:caf6:100:050

(If you don’t have IPv6 you should get it, or you can view graphs based on the sensor values at lindberg.tl instead).

Carefully crack the case open using a knife. You can be quite rough at the PS/2 end of the casing as this part will be
removed anyway. Don’t destroy the USB side of the casing as this part will be reused for our adapter casing later on.

Once opened you should basically have a USB-connector connected to a PS/2 connector covered in hot glue/plastic wrapped in a thin-foil like material (for shielding). Remove the wrapping and cut away the PS/2 connector with a knife.

Now, carefully remove the remaining plastic. Don’t worry about the soldered wires but make sure that you don’t damage the pins on the USB connector. You should end up with something like this.

Do the same with another USB-PS/2 adapter so that you basically end up with two USB connectors.
Bring out your soldering iron and de-solder the remaining wires from the connectors.

Place the connectors back-to-back and turn one of them 180 degrees so that it becomes “up-side-down”. If you don’t do this you’ll end up connecting the pins in the wrong way.
To make the pins touch each other you’ll have to bend them roughly 30-45 degrees. Solder the pins together.

Cut out a piece of wire and solder it to the casing off each connector. You can use the existing solder points used for the shielding/ground or create a new. Some flux will help a lot here.

That’s it for the connector. Next up is to creating a new casing using the existing casings. Unfortunately I forgot to take pictures of this process.
Take on pair of the two cases, take one of the two pieces and place the USB-USB adapter inside it . Make a mark on the casing approximately in the middle of where the soldered pins meet.
Cut at this point and do the same for the same half of the other case. Just do a rough cut and file the pieces until the adapter fits. Once the bottom fits, do the same for the top part using the bottom pieces as a template.

Place it in the new casing and pour some hot glue over it to make it steady. Glue the top on and it’s complete.

USB-USB female adapter in action.

Will it be as good as a commercial adapter? probably not. Will it effect data transmission? probably (depending on how good the solder joints are). Cheaper than a commercial adapter? yep (assuming you already have the parts needed and don’t factor in the labor )

The system clock is based around a 32kHz clock crystal and one of the 8-bit timers provided by the AVR, it is possible to use the CPU frequency as a timer base as long as it’s a nice, dividable, frequency.
To be able to provide a stable 1 Hz clock but still have sub-second precision we divide 1 second into an arbitrary number of
system ticks. To minimize CPU usage the tick counter should be increased at each interrupt, this means that the number of
ticks per second we we choose determines our interrupt frequency and timer resolution.
How to choose number of ticks? It all depends on your required resolution, if you only want a 1 second resolution a tick and a second becomes equal.

The AVR timer is a 8-bit register that simply counts at the rate of its clock source. The clock source can be either the CPU
clock or an external oscillator, the clock source is also subject to a prescaler to further decrease the frequency.
The timer then generates an interrupt on overflow and/or when it hits a pre-configured value.
Since its a 8-bit timer, we have a maximum of 256 cycles before an interrupt is generated, a smaller interval can be achieved by using the comparator match to generate an interrupt at a specific value.

This example is creating a 1/32 second resolution timer (32 ticks per second) using an external 32768 Hz watch crystal.
comp is the comparator value, to avoid re-arming it with different values it should be limited to 128 or 256, otherwise it has to be changed at each interrupt.

The following to equations can be used to calculate either comp or the prescale value.
prescale is limited by the target device, but common values are powers of 2 (8,32,64,128,256,1024).

Using 128 as the comp value and inserting the other values into equation 2 yields the following prescaler

So, a prescaler of 8 gives us two interrupts per 256 cycles, one at 128 and one at 256 (overflow). Using 256 as comp would yield a perscaler of 4 but the target device I used didn’t have a TS/4 prescaler.

Complete source code for a 1/32 (or 31.25ms) second resolution timer for the ATmegaxx4 using Timer 2 and a 32kHz watch crystal connected to the pins TOSC1 and TOSC2.
Requires AVR libc.

#include <avr/io.h>
#include <avr/interrupt.h>
 
typedef uint32_t clock_time_t;
static clock_time_t global_system_ticks = 0;
 
/* ISR for the timer overflow */
ISR(TIMER2_OVF_vect)
{
    global_system_ticks++;
}
 
/* ISR for the comparator */
ISR(TIMER2_COMPA_vect)
{
    global_system_ticks++;
}
 
/* Return number of elapsed ticks */
clock_time_t clock_time()
{
    return (global_system_ticks);
}
 
/* Return number of elapsed seconds */
unsigned long clock_seconds(void)
{
    uint32_t tmp;
 
    TIMSK2 &= ~(1 << OCIE2A) | (1 << TOIE2);
    tmp = global_system_ticks / 32;
    TIMSK2 |= (1 << OCIE2A) | (1 << TOIE2);
    return (tmp);
}
 
void clock_init()
{
 
    /* Enable external oscillator (32 kHz crystal) connected to TOSC{1,2} */
    ASSR |= (1 << AS2);
 
    /* Reset timer */
    TCNT2 = 0;
 
    /* Set TS/8 prescaler, results in a 4096Hz clock */
    TCCR2B |= (1 << CS21);
 
    /* Compare at half counter value */
    OCR2A = 128;
 
    /*
     * Enable overflow and compare interrupt.
     * Triggers each 1/32 secs
     */
    TIMSK2 |= (1 << OCIE2A) | (1 << TOIE2);
}

Here is an Eagle PCB footprint I created for the MagJack (MJF13T36L-KF06B3GY-0808) from MoreThanAll sold by Sparkfun.

This one has been proven in production and it works, however you should double- (and triple-) check pin-outs and footprint of your device before sending a PCB for manufacturing using this part. Use at your own risk.

Magjack schematic

Magjack footprint

Download

morethanall_magjack.lbr

The “Authenticated sender” is added when a user has been authenticated by the MTA through SASL and the directive smtpd_sasl_authenticated_header have been set to yes.  The header_checks directive takes a file containing a regular expression which rewrites the header data and removes sensitive information.

This all works well – with IPv4. The regular expression posted on the pages mentioned above does not take IPv6 addresses into account, I modified it slightly to accept both IPv4 and IPv6 addresses.

/^Received: from (.* \(\[?[-._[:alnum:]]+\]? \[([\.0-9]{7,15}|IPv6[\:a-fA-F0-9]+)\]\))(.*)
\(Authenticated sender: ([^)]+)\)(.*)(by mx1\.example\.com) \(([^)]+)\) with (E?SMTPS?A?) id
 ([A-F[:digit:]]+)(.*)/ REPLACE Received: from smtp-auth.example.com (smtp-auth.example.com
 [127.0.0.1]) (Authenticated sender: hidden)$5$6 ($7) with $8 id $9 $10

Note that this should be one single line.

Put this in a file, for example /usr/local/etc/postfix/obscure_smtp_auth and add the following to your Postfix configuration (assuming you have SASL working).

header_checks = pcre:/usr/local/etc/postfix/obscure_smtp_auth
smtpd_sasl_authenticated_header = yes

The first header will now be rewritten, for both IPv4 and IPv6 clients and will look something like this.

Received: from smtp-auth.example.com (smtp-auth.example.com [127.0.0.1]) 127.0.0.1 (Authenticated sender: hidden)
	by mx1.example.com (Postfix) with ESMTPSA id 3677033C6F
	for &lthostmaster@example.se>; Wed, 10 Dec 2008 16:31:51 +0100 (CET)

instead of

Received: from [IPv6:2001:xxxx:xxxx:xxxx:xxxx:xxxx:fedd:7914] (unknown [IPv6:2001:xxxx:xxxx:xxxx:xxxx:xxxx::fedd:7914])
	(Authenticated sender: someuser@example.com)
	by mx1.example.com (Postfix) with ESMTPSA id 3677033C6F
	for  &lthostmaster@example.se>;  Wed, 10 Dec 2008 16:31:51 +0100 (CET)

The UUID generation algorithms are specified in RFC4122 and I’ve created a static PHP class that implements version 1 which is time based UUID, version 4 which is truly psuedo random UUID and version 3 and 5 which are named based UUID, using either MD5 (version 3) or SHA-1 (version 5).

I know there is a PECL package around the original uuidlib but this class have no external dependencies. The full source is available at the bottom of the page.

The class has the following public functions, their return value depends on the format argument.

$mixed generate($type, $fmt = self::FMT_BYTE, $node = "", $ns = "")
$mixed convert($uuid, $from, $to)

UUID version constants where each constant represents a specific UUID version

UUID_TIME, UUID_NAME_MD5, UUID_RANDOM, UUID_NAME_SHA1

UUID format constants

• FMT_BYTE is the default and returns a array of bytes that represents the 128-bit UUID.
• FMT_FIELD returns a associative array with individual fields as the format specified in RFC4122.
• FMT_STRING returns the familiar ASCII representation of a UUID (xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx).
• FMT_BINARY returns a raw 128-bit binary representation of the UUID.

The convert method can be used to convert between these representations.

UUID version 1 example

xxxxxxxxx-xxxx-1xxx-xxxx-xxxxxxxxxxxx
Version 1 UUIDs are timed based with the node constant. The name space argument is ignored and is not used in the generation. The node identifier (”abcdef” in this case) should identify the node or object, only 6 bytes (or characters) are used from the node id.

require_once('class.uuid.php');
$str = UUID::generate(UUID::UUID_TIME, UUID::FMT_STRING, "abcdef");
print "$str\n";

Example output, the last part is the node id and will remain constant (as long as the node id is the same), the first part will change according to time while the middle part is pseudo random.

1b55e723-578b-4e34-d5cf-616263646566

The class does not fully implement version 1 as described in the RFC. The clock sequence is generated from a psuedo random generator each time a new UUID is generated and not written and read from a stable storage as described in the RFC.

UUID version 4 example

xxxxxxxxx-xxxx-4xxx-xxxx-xxxxxxxxxxxx
Version 4 is a fully pseudo random where all fields are randomly generated.

require_once('class.uuid.php');
$str = UUID::generate(UUID::UUID_RANDOM, UUID::FMT_STRING);
print "$str\n";

Kind of useless example output

d8988842-43d5-42b3-9049-af4bbc694dbe

UUID version 3/5 example

xxxxxxxxx-xxxx-3/5xxx-xxxx-xxxxxxxxxxxx
The main difference is that a node id is unique within a name space and a UUID generated from the same node id and name space is always the same. The generation algorithm is the same for both version 3 and 5 with the only difference being the hash method used (MD5 versus SHA1).
This is the example from the RFC.

require_once('class.uuid.php');
 
/*
 * 6ba7b810-9dad-11d1-80b4-00c04fd430c8 is the DNS name space
 */
$md5 = UUID::generate(UUID::UUID_NAME_MD5, UUID::FMT_STRING,
    "www.widgets.com", '6ba7b810-9dad-11d1-80b4-00c04fd430c8');
$sha1  = UUID::generate(UUID::UUID_NAME_SHA1, UUID::FMT_STRING,
    "www.widgets.com", '6ba7b810-9dad-11d1-80b4-00c04fd430c8');
print "MD5: $md5\n";
print "SHA1: $sha1\n";

Example output

MD5: e902893a-9d22-3c7e-a7b8-d6e313b71d9f
SHA1: 13726f09-44a9-5eeb-8910-3525a23fb23b

Final notes

This class have been tested with PHP 5.2.x on both little and big endian machines. While there could be bugs in the algorithm implementation at least they are consistent across platforms

Get from github

• class.uuid.php

Links

• RFC4122

I previously used the service from addthis but never really liked it so I created my own, you’re free to use it if you like. It’s a simple script that is quite easy (guess it depends on who you ask…) to use and that generates bookmark/share buttons for popular services such as delicious, digg, facebook, twitter and more.

Supports Delicious, Digg, StumbleUpon, Twitter, Technorati Favorites, Google Bookmarks, Facebook, Reddit, Diigo, Blogmarks, Blinklist and Magnolia

This is a sample of how the bookmark bar could look, it’s possible to re-arrange and remove individual services if wanted. The icons are courtesy of http://fasticon.com/freeware/index.php/web-2-icons/

Follow this guide to get your own bookmark bar, you need some knowledge of how to edit HTML.

Step 1

• Download shareit.js
• Download the icon pack from here or here

Step 2

Extract the folder Icons from Web2Icons_PNG.zip, copy the Icons directory and the shareit.js file to your webhost.

Step 3

Integrate a piece of JavaScript code where you would like the bookmark bar to appear, the script will automatically get the current page title and page link and use that as the base for the submission.

There are three variables that affect the look.

• share_show – Selects which services to display
• share_icon_path – Absolute path to icon directory
• share_domain – Optional, if you want your domain to be replaced with something else.

Make sure to adjust the path to the Icons and shareit.js so that they point to the correct position on your webhost.

This will give you a bookmark bar like the one above.

<script type="text/javascript">
var share_show = ['delicious', 'digg', 'stumbleupon',
   'twitter', 'technorati', 'google', 'facebook', 'reddit',
   'diigo', 'blogmarks', 'blinklist', 'magnolia'];
var share_icon_path = '/Icons/';
</script>
<script type="text/javascript" src="/shareit.js"></script>

You can tweak share_show to select and order which services you would like to display, for example to only display FaceBook, Digg, Delicious and Twitter set it like this

<script type="text/javascript">
var share_show = ['facebook', 'digg', 'delicious', 'twitter'];
var share_icon_path = '/Icons/';
</script>
<script type="text/javascript" src="/shareit.js"></script>

If you have an IDN domain you need to use the share_domain setting, set it to your puny-encoded domain. Like this

<script type="text/javascript">
var share_show = ['delicious', 'digg', 'stumbleupon',
   'twitter', 'technorati', 'google', 'facebook', 'reddit',
   'diigo', 'blogmarks', 'blinklist', 'magnolia'];
var share_icon_path = '/Icons/';
var share_domain = 'www.xn--bcher-kva.ch';
</script>
<script type="text/javascript" src="/shareit.js"></script>

There you have, you should now have your very own bookmark bar!

If you use this on your blog/website consider putting a link back to here, thanks.

Consider the following scenario. Actions and inputs from a web page triggers something that might take a (very) long time to execute and if executed during the browser session which besides annoying the user who has to wait for a page to load, might cause a time out and interrupt the processing. How do we solve this?

I’ll use the term job to indicate some work that needs to be execution, in practice this is an isolated PHP function which takes an unspecified time to execute.

Some possible solutions

• Store job information somewhere (like a SQL database) have a separate script, PHP or in some other language, running on the host, either continuously or from crontab. The obvious flaw is that it requires additional administrative maintenance to keep the script running and if executed from crontab, will cause an additional delay before the job is executed.
• Use pcntl_fork from inside the (web page) PHP script and execute the job function in the child, problem is that this will fork an entire Apache (or whatever web server you’re using) processes which can cause all sorts of problems.
• Some other solution I didn’t think about?

My solution ended up being a mix of the two above. It consists of three major parts and, a bit simplified, works like this

1. Job information is stored in a SQL database.
2. An external CLI-based worker PHP-script is launched from inside the web page script using proc_open.
3. The worker script, which is running in an isolated PHP process calls pcntl_fork to create a duplicate of itself. The parent returns directly to the web page script.
4. The child process of the worker script now runs free of any webserver process and can now retrieve job information from the SQL database. When all jobs are executed it kills itself.

Here is a nice block diagram that shows the interaction of the components

And viola, we have asynchronous background execution of (almost) arbitrary PHP functions. There are a few problems left to solve, we’ll tackle those further down.

Requirements

• PHP 5 built with CLI support
• The PHP module pcntl
• The classes SQLConnector and Prefs published in earlier posts.

SQL database

First, we’ll need some database support. I’ve used MySQL, but any database should work fine. One simple table is required, it’s called jobs and looks like this

+-------+---------------------+------+-----+---------+----------------+
| FIELD | Type                | NULL | KEY | DEFAULT | Extra          |
+-------+---------------------+------+-----+---------+----------------+
| jid   | bigint(20) UNSIGNED | NO   | PRI | NULL    | AUTO_INCREMENT | 
| DATA  | text                | YES  |     | NULL    |                | 
+-------+---------------------+------+-----+---------+----------------+

The column jid is an arbitrary job id and the column data contains job information (more on what exactly job information is will be covered further down). You can create this table with the following SQL command

CREATE TABLE jobs (jid bigint UNSIGNED AUTO_INCREMENT,
    DATA text, PRIMARY KEY (jid))

PHP implementation

The implementation consists of three files, we’ll focus on the class jobs first. This leads us into what exactly the data column should contain. Since the job (PHP function) will execute in a new context the worker must be able to bring in PHP files so that required functions and classes can be resolved. So, in addition to the actual job function and an opaque argument we also need to store a list of PHP files to include at execution time.

The complete data structure stored in the data column looks like this.

$data = array(
    /* Version identifier of job structure */
    'version' => 1,
    /* Array of PHP files to include at execution time */
    'include' => array(),
    /* Name of actual job function */
    'callback' => '',
    /* Opaque argument passed to callback function */
    'args' => null,
);

A dedicated class calls jobs manages jobs and allows enqueuing and dequeuing of jobs. For clarity, only the function prototypes are shown here, the full source code can be found at the bottom of the pafe.

class jobs {
    private $m_con; /* SQL Connector */
    const jobversion = '1'; /* Job structure version */
 
    private $m_data = array(
        'version' => self::jobversion,
        'include' => array(),
        'callback' => '',
        'args' => null,
    );
 
    /*
     * Enqueue a job for execution
     *  cb - Name of PHP function to execute
     *  args - Arguments to pass to 'cb'
     *  incs - Array of PHP files to include before execution
     */
    public function enqueue($cb, $args, $incs);
 
    /*
     * Return the job identifier (integer) of
     * next available job or -1 if no new jobs are
     * available.
     */
    public function nextJob();
 
    /*
     * Dequeue job with id 'jid' for execution,
     * if no such job is available null is returned.
     *
     * An array with the following keys are returned
     *  (jid, include, callback, args)
     */
    public function dequeue($jid);
 
    /*
     * Attempt to launch worker process if not already running
     */
    private function startWorker();
}

The callback function ‘cb’ will be called like this, cb(args), before this call all files listed in the array incs will be included using require_once.
The functions enqueue, dequeue and nextJob are nothing special, they simply do normal SQL manipulation (inserts and removes job from the table). However the routine startWorker might deserve some attention (it’s called internally from enqueue)

/* Should be set to the path of the CLI PHP binary */
define("PHP_PATH",      "/usr/local/bin/php");
 
private function startWorker() {
    /*
     * Attempt to read the preference
     * 'worker_pid' to check if the worker is
     * already running.
     */
    $prefs = new Prefs($this->m_con, 'jobs');
    $pid = $prefs->worker_pid;
    if ($pid != null)
        return;
 
    /* Get our working directory */
    $cwd = getcwd();
 
    /*
     * Construct a command such as the following
     *  /usr/local/bin/php /working/path/worker.php
     */
    $cmd = escapeshellcmd(PHP_PATH ." $cwd/worker.php");
    $desc = array();
 
    /* Execute the command and wait for it to finish */
    $proc = proc_open($cmd, $desc, $pipes, NULL, NULL);
    proc_close($proc);
}

The startWorker function is executed in the context of a web page (from inside the Apache PHP module for example) and as you can see it doesn’t call pcntl_fork directly. Technically it still forks, otherwise it wouldn’t be able to execute another process. But we let the PHP module deal with that mess.

This leads us to worker.php
File: worker.php

require_once('jobs.inc');
require_once('prefs.inc');
require_once('db_mysql.inc');
define("PHP_PATH",      "/usr/local/bin/php");
 
if (!isset($argv)) {
    exit("Don't call me from a browser");
}
 
/*
 * Fork our self and let the parent return directly to startWorker
 */
$pid = pcntl_fork();
if ($pid == -1) {
   die("could not fork");
} else if ($pid) {
    /*
     * This is the parent process. Record the child pid so that we
     * don't launch more processes than wanted.
     */
    $con = new MySQLConnector('localhost', 'user', 'pwd', 'db');
    $prefs = new Prefs($con, 'jobs');
    $prefs->worker_pid = $pid;
    $prefs->Flush();
    /*
     * Exit the script, it runs in a separate
     * PHP process NOT inside Apache
     */
    exit;
}
 
$con = new MySQLConnector('localhost', 'user', 'pwd', 'db')
$jobs = new jobs($con);
 
$cwd = getcwd();
if ($cwd == false)
    exit("can't get working directory");
 
/*
 * Loop over all available jobs
 */
$pjid = -1;
for (;;) {
    $jid = $jobs->nextJob();
    if ($jid == -1 || $jid == $pjid)
        break;
 
    /*
     * Execute each job in a clean environment
     */
    $cmd = escapeshellcmd(PHP_PATH . " $cwd/job_execute.php $jid");
    $desc = array();
    $proc = proc_open($cmd, $desc, $pipes);
    proc_close($proc);
 
    $pjid = $jid;
}
 
/* Tell the world that we aren't executing any more */
$prefs = new Prefs($con, 'jobs');
$prefs->worker_pid = null;

The first thing that happens is that worker.php forks itself and lets the parent return directly to the caller, which in this case is startWorker. This allows startWorker and thus enqueue to finish and the calling PHP script can resume with what ever it was doing (creating a web page etc).

Also note that worker.php doesn’t execute the job functions directly, instead it hands the jid number to a second script called job_execute.php. There is a good reason for this, since we must include PHP files (with require_once) to be able to execute the job function the namespace of the worker would be contaminated quite fast and with that comes the risk of name collisions. By letting each job execute in a totally clean environment they can include only the files needed for their execution and thus avoid any name collisions.

job_execute.php takes a job id (jid) as argument, dequeues the job, includes all required files, calls the job function and then terminates.

File: job_execute.php

require_once('jobs.inc');
require_once('db_mysql.inc');
 
if (!isset($argv)) {
    exit("Don't call me from a browser");
}
 
if (!isset($argv[1])) {
    exit("Job missing");
}
$jid = $argv[1];
 
$con = new MySQLConnector('localhost', 'user', 'pwd', 'db');
$jobs = new jobs($con);
$job = $jobs->dequeue($jid);
if ($job == null)
    exit("Job dequeue failed");
 
/*
 * Include required files
 */
foreach ($job['include'] as $inc) {
    require_once($inc);
}
 
/* Execute the job handler */
$cb = $job['callback'];
$cb($job['args']);

Okay, that should be all. Are you still with me?

Using it

After all this…how do one use this then? First of all we need one or more job functions that we would like to be background executed. The prototype for such a function looks like this

function myBgFunc($args) { ... }

Where args can be anything you like (as long as PHP allows it). Put the function inside a file and include (preferably with require_once) any other files that you might need. For example like this, not the best example, but anyway.

File: myfunc.php

<?php
/* $args is an integer in this case */
function test1($args) {
    $j = 1;
    for ($i = 0; $i < $args; $i++) {
       $j *= 1.00001;
    }
    /* Report result somewhere */
}
?>

Now, from your run-of-the-mill PHP script, create a jobs object and enqueue the function test1.

require_once('db_mysql.inc');
require_once('jobs.inc');
$con = new MySQLConnector('localhost', 'user', 'pwd', 'db');
$jobs = new jobs($con);
/* Remember, "function", "argument", "files needed" */
$jobs->enqueue("test1", rand() * 1000, array('myfunc.php'));
print "Job enqueued";

Even if test1 takes ages to execute, the enqeue function will execute in a fraction of a second and report back directly to the web browser. You can enqueue any number of functions, they will be executed on a first-come first-serve basis.

Improvements

I’ve tested this a bit and it seems to work, however I haven’t put it through some real situation test yet. With some additional locking it should be possible to run multiple instances of worker.php to take advantage of SMP systems.

There is no built-in mechanism to know when a job has finished. The job function will need to report its results to a SQL database for example. If browser feedback is needed an AJAX solution could poll the database for the execution status.

Files

Files with complete source code

• jobs.inc
• worker.php
• job_execute.php
• prefs.inc
• db_iface.inc
• db_mysql.inc
• static.inc

If you found this useful consider putting a link back to this article from your own blog/website, thanks.