MemcachedManager enables Zope to cache data in one or more shared memcached instances instead of in memory in each Zope process. This provides a reasonable tradeoff between added overhead and memory consumption. So far, we have been using python-memcached as a library for talking to memcached. python-memcached is compact and convenient, but can never compete with C-based libraries on speed.

pylibmc

While cmemcache never really took off as a memcached library for python, Ludvig Ericson has now made pylibmc which is a wrapper around libmemcached, and promises significant speedups.

Improving MemcachedManager

As we tested pylibmc and the retrieval of data had become significantly faster, it became obvious that some profiling and improvements to MemcachedManager itself was appropriate. The changes was mainly to remove unused code and avoid unnecessary invocation of DateTime.

Testing the improvements

We set up two tests. The first one is synthetic and returns a list of 25 dictionaries, each dictionary containing 3 elements with Title, Description and url. This is similar to caching a content listing, and it also uses pickling which adds some overhead but will usually be used when caching anything in Zope. The cached script was then called 1001 times from another script, for a total of 4004 calls to get content from memcached.

python-memcached and MemcachedManager 1.0rc1 took 1382m. When applying the MemcachedManager improvements, that time went down to 1014ms, and adding pylibmc to the mix gave a result of 812ms.

The second test is for a development site with plenty of content and Membrane based users. Membrane is mainly used for caching user info, which are rather largeish and complex objects. Viewing a page 10 times gave 620 calls to get content from memcached.

python-memcached and MemcachedManager 1.0rc1 took 789ms for those 620 calls, the improved MemcachedManager gave a barely noticeable improvement of 707ms, while pylibmc got a chance to really shine and only took 355ms.

Test for yourself

The improvements to MemcachedManager have been released as 1.0rc2. If you want to compile pylibmc with Python 2.4, you have to apply a patch and point to the correct libraries and include files. We have created a buildout for your convenience.

svn co http://svn.plone.org/svn/collective/Products.MemcachedManager/buildouts/pylibmc pylibmc
cd pylibmc
python bootstrap.py
bin/buildout

Then you have to start memcached with

bin/memcached

and start zope like you're used to.

MemcachedManager is available as a type in the ZMI, and can be used as a drop in replacement for RAMCache.

-- Helge Tesdal