Using a message queue can help to suspend heavy processes and execute them later so you won’t bother your visitors with long waiting times. There are a few solutions for queuing like Gearman, ActiveMQ and Zend Server Job Queue. For www.nd.nl (a Dutch newspaper) we wanted a simple and free queue mechanism that integrates with Zend Framework for handling a number of jobs. We found MemcacheQ.

MemcacheQ

MemcacheQ has some nice features that makes it an ideal candidate for our purpose:

  • damn simple
  • very fast
  • multiple queues
  • great concurrency
  • memcache compatible protocol

Especially the last item is very interesting. This means that you can use the native memcache API for communicating with MemcacheQ, so you don’t need additional PHP extensions.

The installation has some dependencies like libevent and BerkleyDB. The latter is used for the persistent storage of the data inside the message queue. If something kills MemcacheQ you won’t loose the queued data.

MemcacheQ will run as a daemon and communicates over the UDP protocol. It can handle concurrent connections and can be used in a load-balanced environment.

Arguments

There are some arguments to use when starting the service. We’re currently using the following on www.nd.nl:

memcacheq -d -r -H /data/memcacheq -N -R -v -L 1024 -u nobody &> \
/data/memcacheq/error.log

Explanation of the used arguments:

  • -d - Run as daemon
  • -r - Maximize core file limit
  • -v - Verbose logging
  • -u <user> - Run as user nobody
  • -H – Directory to store the BDB files
  • -N – Performance improvement
  • -R - Remove logfiles when no longer needed
  • -L - Log buffer size, default is 32KB, we want 1024KB

Other helpful arguments:

  • -h - Show the help with all the arguments
  • -vv - More verbose mode
  • Run without -d to directly see the output

Zend_Queue

Zend Framework provides an adapter to talk with MemcacheQ: Zend_Queue_Adapter_Memcacheq
This makes it easy to integrate the queue in your website.

Usage

We have a news site and want to show the number of visits to an article. When storing this number in a database, we need to execute a query every time a visitor reads an article. That’s something you don’t want because you can’t cache it and the visitor has to wait until the INSERT/UPDATE has finished. So we’ll have to store the hit temporarily in a fast queue and insert it later on in the database.

On the article page we add the new hit to the queue.

<?php
// MemcacheQ config
$queueOptions = array(
    'name' => 'example-queue',
    'driverOptions' => array(
        'host' => '127.0.0.1',
        'port' => 22201
    )
);

// Instantiate Zend_Queue
$queue = new Zend_Queue('MemcacheQ', $queueOptions);

// Find out if the queue exists
if (!$queue->getAdapter()->isExists($queueOptions['name']))
{
    // If not, create it
    $queue->createQueue($queueOptions['name']);
}

// Build a query string (key=val&key=val) because we need a scalar value
$params = http_build_query(
    array(
     'timestamp' => $timestamp,
     'page' => $page
    )
);
// Send the data to the queue
$queue->send($params);

And in a separate process (e.g. cronjob) we retrieve the queued hits and insert them in the database.

<?php
// MemcacheQ config
$queueOptions = array(
    'name' => 'example-queue',
    'driverOptions' => array(
        'host' => '127.0.0.1',
        'port' => 22201
    )
);

// Instantiate Zend_Queue
$queue = new Zend_Queue('MemcacheQ', $queueOptions);

// Retrieve 5 items from the queue
$messages = $queue->receive(5);

// $message is now a instance of Zend_Queue_Message_Iterator
// @TODO: Use a nice FilterIterator ;)
foreach ($messages as $job)
{
    if ('creating queue' == $job->body || false === $job->body)
    {
        // Skip message
        continue;
    }
    // Parse the query string to a array
    parse_str($job->body, $data);

    // Execute the heavy process
    $this->registerHit($data['timestamp'], $data['page']);
}

Known bugs

There’s currently (ZF version 1.11.2) one known bug in the ZF issue tracker: #ZF-9969.
This is a bug when using MemcacheQ 0.2.0.

Leave a Reply

Your email address will not be published. Required fields are marked *

6 thoughts on “Using MemcacheQ as message queue

  1. Pingback: Zend Framework in Action » Enrise: Using MemCacheQ as a Message Queue

  2. Interesting technique. I was wondering what it is that can’t be cached? When you say ‘it’, do you mean the the db query that updates the hits count? I assume that you use (partial?) output caching?

      • I don’t really get the point how/why you would want to cache a hits count update query. Why can’t a news articlebe cached, and update a hits count at the same time? (apart from the fact that the user has to wait for the hits update query to finish).

        • I now know what you mean.
          It’s not that I want to cache the query that updates the hit. I don’t want a update query being executed every time a visitor visits a page.
          That can cause a high load on the database on peak times. When using memcacheq we can bundle all queries and execute a query per page for all the visits, not per visit.
          And when a plane crashes and we get a major peak on the website, we can temporary turn the off the cronjob that processes the queue to take some load off the database.

  3. So by the term ‘caching’ in this context you mean saving up the queries and executing them periodically? That not really the definition of caching in my opinion.

    Very interesting that you put the real life examples here!

Volg ons

Twitter

Hey #designer! The new #Sketch3 is out now: http://t.co/nj9RsVLu7r http://t.co/MjDeXcuKDk
- Monday Apr 14 - 3:01pm

Next Enrise case: @carsomnl presentation for @DIA_awards. #DIA14 http://t.co/afTWUv6AXs
- Friday Apr 11 - 11:52am

Preparing for judge @DIA_awards #DIA14 http://t.co/FfP2vewZle
- Friday Apr 11 - 9:48am