I’ve been running DSPAM for a long time and the spam classification is great, unfortunately the maintenance tools are not as it turns out. This is the tale of why and more especially HOW I moved from the hash driver to the mysql driver as a backend for my DSPAM installation.

It all began a few days ago when the dovecot-antispam plugin all of a sudden refused to re-classify mails. This was all very weird because I was possible to re-classify mails from the command line, but the same mail/signature failed from the plugin.

After some debugging and messing with the plugin code I established that it successfully called dspam to re-classify the mail but dspam failed. Hm, maybe a permission problem then…nope. Same command with same permissions worked from the command line.

At this point I was really puzzled (and quite annoyed), the error made no sense. So I hacked the plugin to call dspam using a system call tracer (in this case truss which is shipped with FreeBSD) to produce a call trace of the dspam processes within the context of the plugin.

After comparing the outputs from a working case and a non-working case, one line stood out.

mmap(0×0,263332768,PROT_READ|PROT_WRITE,MAP_SHARED,4,0×0) ERR#12 ‘Cannot allocate memory’

For those who don’t speak truss. This means that the mmap() system call (map a content of file into memory) failed because of an out of memory condition.

WHY!? Well it obviously hit a built-in mmap limit as the system had a lot of free memory. It was not vm.max_proc_mmap (tried bumping it, didn’t work). vm.kmem_size_max is large (64-bit system) and the uid had unlimited resources (ulimit). I’m still puzzled by this.

As the title reveals I’m running DSPAM with the hash driver, why? mostly because it was the default back-end driver in DSPAM when I installed it (and I was young and naive).

Wonder why it hit the limit in the first place, a quick look in /var/db/dspam/data reveals that the hash database with tokens for this particular user had grown to ~250MB.

Why so big? DSPAM comes with a cleaning utility called dspam_clean to clean out unused data, I’ve been running this nightly to keep stuff neat and tidy. Well, here comes the fun part.

It turns out that dspam_clean doesn’t actually do anything when you’re running the hash driver, it just looks like its working and takes ages to execute. There is no mentioning of this in any user documentation I found. With the hash driver you’re supposed to run cssclean…only that this program doesn’t work either – unless you patch it. I found a patch dubbed “cssclean is a redheaded stepchild” sent to the (now defunct, and moved to sourceforge) dspam-dev@ list at nuclearelephant.com that took care of this. The patch missed an include (at least on FreeBSD) that caused it to crash on 64-bit platforms (64/32-bit pointer truncation). The whole patch is included at the bottom of the page.

To summarize it, the maintenance tools are virtually non-existing when using the hash driver.

Probably a good opportunity to convert to the mysql driver instead (as I’ve been meaning to do for the last year or so).

Converting from the hash driver to mysql

There is a tool included with the DSPAM package called dspam_2sql, but of course this didn’t work properly either. Most important, it didn’t work with virtual users and just refused to run. Also timestamps on token data were ignored. I’ve hacked together a patch to fix theses issues, it’s available at the bottom of the page.

I first attempted to import non-cleaned data into the SQL database…big mistake. I aborted the progress when the token table had about 8 million entries (and weren’t even done with 1 user).

Quick recipe on how to do the conversion

First

• Grab the source package for dspam-3.8.0
• Download the patches at the bottom of this page

Patch and compile the sources

tar xzf dspam-3.8.0.tar.gz
cd dspam-3.8.0
patch -p1 < dspam_cssclean.patch
patch -p1 < dspam_2sql_virtuser.patch
cd src
make

The patched cssclean tools can be found in tools.hash_drv

Run tools.hash_drv/cssclean on all users (or users that receive a lot of mail). Backup your database (/var/db/dspam) first.

Example

tools.hash_drv/cssclean /var/db/dspam/user@domain.tld/user@domain.tld.css

You can check the number of entries with cssstat

Create a MySQL database and user for dspam, I’ve called mine dspam. The table defintions can be found in tools.mysql_drv

mysql -D dspam -u dspam -p < tools.mysql_drv/mysql_objects-speed.sql
mysql -D dspam -u dspam -p < tools.mysql_drv/virtual_users.sql

You should now have a database with tables and a cleaned hash database, the only thing left is to convert it. dspam_2sql will output SQL statements, just feed these into MySQL.

For example, the conversion can be done like this

tools/dspam_2sql | mysql -D dspam -u dspam -p

Beaware that it probably will take a lot of time.

You can convert one single user with

dspam_2sql –user=user@domain.tld

That’s it. Modify your dspam.conf to use the MySQL driver and restart dspam.

Performance

The MySQL driver IS slower than the hash driver. The processing time for a message with the hash driver was around 0.1-0.2 seconds on this hardware. The MySQL driver takes between 0.2-2 seconds for a message.

The slowdown is not a problem for me. Things like real database consistency (some of my users had duplicated tokens with the hash driver), better maintenance tools, easier manipulation of the data (and the benefits of my MySQL replication+backup configuration) makes it well worth it.

Patches

• dspam_cssclean – cssclean patch found on mailing list. Posted by Frank Cusack.
• dspam_2sql_virtuser – dspam_2sql patchset. Virtual user support etc.

Having a consistent interface will obviously help if you are using multiple types of database servers or someday would like to migrate to another SQL server (of course, weird and vendor specific SQL code could be a problem in this case). Other advantages are things like automatic disconnection on object destruction etc.

The foundation is an abstract class called SQLConnector and provides the basic interface. The class will connect to the database on creation and disconnect when the object is destroyed, it also implements __sleep and __wakeup so that it’s possible to serialize a SQL connection.

Because SQLConnector is abstract one can’t create objects directly from it, we’ll create child classes for each type of SQL-server we want to connect to. These objects will share the same interface and can be used interchangeably.

File db_iface.inc

<php
abstract class SQLConnector
{
    private $m_dbAddress; /* Address of SQL server */
    private $m_dbAccount; /* Account name (username) */
    private $m_dbPwd;     /* Password */
    private $m_dbName;    /* Initial database */
 
    /* Base constructor */
    public function __construct($address, $account, $pwd, $name) {
        $this->m_dbAddress = $address;
        $this->m_dbAccount = $account;
        $this->m_dbPwd = $pwd;
        $this->m_dbName = $name;
 
        /* Call the connect method in the child class */
        $this->connect($this->m_dbAddress, $this->m_dbAccount,
            $this->m_dbPwd, $this->m_dbName);
    }
 
    /* Let the child disconnect when the object is destroyed */
    public function __destruct() {
        $this->disconnect();
    }
 
    /* We are about to be serialized, disconnect
     * and return data needed for serialization
     */
    public function __sleep() {
        $this->disconnect();
        return array('m_dbAddress', 'm_dbAccount',
            'm_dbPwd', 'm_dbName');
    }
 
    /* We have been unserialized, re-connect */
    public function __wakeup() {
        $this->connect($this->m_dbAddress, $this->m_dbAccount,
            $this->m_dbPwd, $this->m_dbName);
    }
 
    /*
     * Abstract methods that needs to be implemented by child classes
     */
    abstract protected function connect($address, $account,
        $pwd, $name);
    abstract protected function disconnect();
    abstract function query($query);
    abstract function fetchRow();
    abstract function fetchAll();
    abstract function getNumRows();
    abstract function freeResult();
    abstract function getError();
}
?>

The following class implements MySQL connectivity using the interface above

File: db_mysql.inc

<?php
 
require_once('db_iface.inc');
 
class MySQLConnector extends SQLConnector
{
    private $m_dbHandle;
    private $m_Result;
 
    /* 
     * Connect function, also selects correct database
     * Returns 1 upon success, otherwise 0
     */
    function connect($address, $account, $pwd, $name) {
        $this->m_dbHandle = @mysql_connect($address, $account, $pwd);
 
        if ($this->m_dbHandle != 0) {
            if (mysql_select_db($name, $this->m_dbHandle)) {
                return 1;
            }
            else {
                return 0;
            }
        }
        else {
            return 0;
        }
    }
 
    /*
     * Disconnect function
     * Returns 1 upon success, otherwise 0
     */
    function disconnect() {
        if (@mysql_close($this->m_dbHandle) != 0) {
            return 1;
        }
        else {
 
            return 0;
        }
    }
 
    /*
     * Does a mysql-query, returns 1 upon sucess otherwise 0
     */
    function query($query) {
        $this->m_Result = mysql_query($query, $this->m_dbHandle);
 
        if ($this->m_Result != 0) {
            return 1;
        }
        else {
            return 0;
        }
    }
 
    /*
     * Fetches an array row
     */
    function fetchRow() {
        return mysql_fetch_array($this->m_Result);
    }
 
    function fetchAll() {
        while ($row = mysql_fetch_array($this->m_Result)) {
            $a_rs[] = $row;
        }
        mysql_free_result($this->m_Result);
        return $a_rs;
    }
    /*
     * Get number of rows
     */
    function getNumRows() {
        return mysql_num_rows($this->m_Result);
    }
 
    /*
     * Free resources allocated by a query 
     */
    function freeResult() {
        mysql_free_result($this->m_Result);
    }
 
    /*
     * Get errorstring
     */
    function getError() {
        return mysql_error($this->m_dbHandle);
    }
}

A (very) similar class for MSSQL connectivity
File: db_mssql.inc

<?php
require_once('db_iface.inc');
 
class MSSQLConnector extends SQLConnector
{
    private $m_dbHandle;
    private $m_Result;
 
    /*
     * Connect function, also selects correct database
     * Returns 1 upon success, otherwise 0
     */
    function connect($address, $account, $pwd, $name) {
        $this->m_dbHandle = @mssql_connect($address, $account, $pwd);
 
        if ($this->m_dbHandle != 0) {
            if (mssql_select_db($name, $this->m_dbHandle)) {
                return 1;
            }
            else {
                return 0;
            }
        }
        else {
            return 0;
        }
    }
 
    /*
     * Disconnect function
     * Returns 1 upon success, otherwise 0
     */
    function disconnect() {
        if (@mssql_close($this->m_dbHandle) != 0) {
            return 1;
        }
        else {
            return 0;
        }
    }
    /*
     * Does a mssql-query, returns 1 upon success otherwise 0
     */
    function query($query) {
        $this->m_Result = mssql_query($query, $this->m_dbHandle);
 
        if ($this->m_Result != 0) {
            return 1;
        }
        else {
            return 0;
        }
    }
 
    /*
     * Fetches an array row
     */
    function fetchRow()
    {
        return mssql_fetch_array($this->m_Result);
    }
 
    function fetchAll()
    {
        while ($row = mssql_fetch_array($this->m_Result)) {
            $a_rs[] = $row;
        }
        mssql_free_result($this->m_Result);
        return $a_rs;
    }
 
    /*
     * Get number of rows
     */
    function getNumRows()
    {
        return mssql_num_rows($this->m_Result);
    }
 
    /*
     * Free resources allocated by a query
     */
    function freeResult()
    {
        mssql_free_result($this->m_Result);
    }
    /*
     * Get errorstring
     */
    function getError()
    {
        return mssql_error($this->m_dbHandle);
    }
}
?>

So, how do one use this then? It’s quite simple

require_once('db_mysql.inc');
require_once('db_mssql.inc');
 
$c1 = new MySQLConnector('mysql-host', 'username', 'password', 'mydb');
$c2 = new MSSQLConnector('mssql-host', 'username', 'password', 'mydb');
 
$c1->query("select something from somewhere");
while ($row = $c1->fetchRow()) {
 ...
}
 
$c2->query("select somethingelse from somewhereelse");

Improvements, oh yes. The connect functions should really throw an error if things doesn’t work.
Also, wrappers for any SQL server should be quite easy to implement.

• db_iface.inc – Abstract interface
• db_mysql.inc – MySQL interface
• db_mssql.inc – MSSQL interface

If this article was usable or otherwise helpful for you, tell a friend or put a link back from your own website/blog.