Hungry for information

The world is a highly dynamic place: an amazing amount of things are happening around us at an incredible place. And we humans, being the curious beings that we are, want to know exactly what is happening. Our need for information is so large that we have build an army of tools to manage information. The latest trend in that area are  Web Feeds: channels with output little bits of information you can subscribe to.

Feeds have several good qualities: they are easy to access, and you can do so using any of the thousands libraries and applications that support the popular RSS and Atom formats. This also makes them very useful as a method to get data from one system to another system.

A feed on my own menu

One place where we want to be able to use feeds is on our websites. Particularly we have a customer who was interested in being able to accumulate data from several feeds and present them as a single listing of news items. Plone 3 already included a RSS portlet which could handle a single feed and present that in a portlet. Originally we planned to extend this portlet to be able to handle multiple feeds. This quickly turned out to be a bad idea: the RSS portlet was designed to act as a very simple to use window into a news source, and does a very good job at that: it is easy to configure, automatically picks up title and URL for the news source and outputs a standard Plone portlet using those. Extending that to handle multiple feeds would result in a something that was too complex for its original use case and not convenient for multiple feeds as well. The worst of both worlds.

So we scrapped that idea and implemented a new package: FeedMixer. Or collective.portlet.feedmixer if prefer the exact python package name. Just like the standard Plone 3 RSS portlet FeedMixer aims to be as simple as possible to use. You can specify the portlet title, number of items to show and as many URLs as you want and in return you will get a portlet with the latest news items from all your sources. And if you want to see more results there is also a handy page which shows the complete entry list from all feeds.

Do you serve fast food, euh, feed?

One problem with feeds is that they are generally not very fast: software which processes feeds regularly asks feed publishers for the latest which can put a huge load on publishers. On the client side you do not want to wait for a system to reload the feed data, process it and show you the results. You want your feed, and you want it fast.

FeedMixer accomplishes that by reusing the caching infrastructure in Plone 3. It uses the memoize caching infrastructure to find a global cache where all feed data can be stored. Normally this is a RAMCache, but if you are using multiple Zope instances you can easily change this to use a shared memcached cache instead. That means that if one of your Zope instances refreshes a feed all your instances will immediately be able to use it. Since the Zope caches may have a longer timeout setting then you want to use for your feed data you can also configure a maximum timeout directly in a FeedMixer portlet.

If FeedMixer needs to refresh data it will try to use the ETags and the HTTP Last-Modified header to only download the feed if it has changed on the server.

--
Wichert Akkerman