Jarn Blog

Plone performance sprint bristol 2008

Helge Tesdal - Jarn — 2009-01-05T18:12:07Z

A summary of the performance sprint in Bristol, 2008

It's been some time since the performance sprint in Bristol now, but it certainly deserves a summary.

Location

Turns out Bristol is a great city with a wonderful atmosphere. There is plenty of history and old buildings (unlike in Norway). Netsight also picked a great location for the sprint, and provided plenty of refreshments to keep the geeks doing long days.

Sprint topics

Automated performance tests

The most important result from the performance sprint is probably the automated funkload tests. In Copenhagen in 2007 we looked into using JMeter, but it was too much hassle to work with, and we didn't get beyond creating some test plan templates for Plone.

With collective.loadtesting we have something that is a lot easier to set up and use in a buildout, and we can have daily performance runs and keep track of the general performance of Plone (although it doesn't seem to be set up yet). This means we can easily and immediately spot changes that affect performance in a negative way, and make informed decisions whether the advantages of functionality or code changes is worth the performance loss.

Instrumentation

Instrumentation was the other important topic at the sprint. We always need more performance data, more accurate information, and better ways of presenting that data to enable us to understand and improve the system we're profiling.

Enter mr.bent. Mr. Bent knows his numbers, and is a framework for allowing profile data to be collected in a Python application and viewed at different logical levels.

One example of usage is collective.performancecolouriser which enables the developer to visually see timing info for different page components by color.

ExtendedPathIndex refactoring

While the ExtendedPathIndex has worked quite well for a long time, the implementation was rather obtuse, or obfuscated if you like.

Martijn Pieters has been planning a rewrite for quite some time, and at the sprint he finally had the opportunity to do so. He found several bugs in the old implementation, added tests and implemented all features for all scenarios, while also improving performance by being smarter about set operation ordering.

The improvements were released as version 2.5, and will be included in Plone at the earliest convenience.

Catalog

experimental.catalogqueryplan saw two improvements. The first one is faster set operations when intersecting a large and a small set. In Plone, especially with Membrane, there are some cases where you have a rather small result set from one index (like user or group ids) and a large result set from another (implemented interfaces or permissions). In those cases it is more efficient to check if the large set has a certain key by direct lookup than to scan all keys for a match. The size check is implemented in Python as a temporary monkeypatch used for the catalogqueryplan indexes, and gives a noticeable improvement. The exact improvement depends on whether all values in the small set are found at the beginning ('Low values' in graph) or end ('high values' in graph) of the large set. It might be as much as 20x faster to check for a key.

The graph shows the timing of an intersection between a set of 10000 items and a subset of 10 items, repeated 100 times. The green bar is the regular C implementation from IIBTree, the yellow bar is our python implementation using has_key with fallback to the C implementation, and the blue bar is using the builtin Python set (instead of the IISet from Zope).

Inside the indexes, we now sort the sets on length when doing intersection operations, to use the smallest sets first. When doing OR operations, we use multiunion instead of several unions. This should be noticeable for the permissions index (allowedRolesAndUsers) for example.

The improvements are released as version 1.1, and can easily be tested by including experimental.catalogqueryplan in your buildout.

The second improvement is the inclusion of a new index. BooleanIndex is a simplified FieldIndex which only stores True values and ignores everything else. This will lower the object count slightly for indexes like default page and folderish and will enable the use of the intersection improvements for smaller sets. This only works as long as there are fewer True than False values indexed.

-- Helge Tesdal

Saving the day: recovering lost objects

Martijn Pieters - Jarn — 2008-12-18T09:47:43Z

When a customer discovers over a week later that an important object was accidentially deleted, what do you do?

Oh noes!

A customer discovered that an important entire section of his site was missing and asked us to bring it back. This was in a heavily edited site, with loads of writes each day, but we quickly located the offending transaction: someone had deleted the object in question 9 days earlier.

Undo was no longer an option, though: too many things had changed, not least the catalog. Truncating the Data.fs (removing all transactions since, including the offending one) was not only undesirable, but impossible as the site stores the data in Oracle through RelStorage.

Time travel

So, instead of permanently removing transactions, we used a handy little package to do some time traveling: zc.beforestorage.

zc.beforestorage does require a ZODB version 3.8 or 3.9; the customer installation is on Plone 3.0, so a newer ZODB3 egg was necessary for this operation. A small additional buildout configuration file (saved as beforestorage.cfg) helps out:

[buildout]
extends =
    buildout.cfg
eggs +=
    zc.beforestorage
    ZODB3
    zope.proxy

[versions]
ZODB3 = 3.8.1
zope.proxy = 3.4.2

[relstorage-patch]
recipe = plone.recipe.command
command = 
    cd ${buildout:eggs-directory}/ZODB3-3.8.1-py2.4-linux-i686.egg/ZODB
    curl -s http://svn.zope.de/zope.org/relstorage/tags/1.1c1/poll-invalidation-1-zodb-3-8-0.patch | patch -N -p0
    cd ${buildout:directory}
update-command = ${relstorage-patch:command}

[instance]
zope-conf-additional +=
    enable-product-installation False

The relstorage-patch section in the above code ensures that our ZODB3 egg is patched with the RelStorage additions, and the zope.proxy egg is needed because ZODB 3.8 requires a newer version. The enable-product-installation line is required because zc.beforestorage puts your ZODB in read-only mode (understandibly); the option tells Zope not to try and write product information to the ZODB.

Once buildout has been run with this configuration (with the -c switch), you'll still need to edit the zope.conf file for your instance, usually in parts/instance/etc/zope.conf. You need to edit the <zodb_db main> section to wrap the storage in the beforestorage. Ours looked something like this:

<zodb_db main>
    # Main database
    cache-size 650000
%import zc.beforestorage
%import relstorage
    <before>
    before 2008-12-08T10:29:03
    <relstorage>
        <oracle>
            dsn RELSTORAGE_DSN
            password xxxxxxxxx
            user xxxxxxxx
        </oracle>
    </relstorage>
    </before>
    mount-point /
</zodb_db>

Any line with the word 'before' in it is new. The timestamp we learned from the undo log, simply converted to UTC. Now, when you start the instance, you are in the past. You can't alter this past (no killing of grandfathers), but you can read it. And lo and behold, the deleted object is back.

Recovery

Now that we have found the lost object, we can recover it. We simply exported it; in the ZMI, choose the Export/Import button, and save the export on the server. Remove the zc.beforestorage configuration (just run buildout with your regular buildout file), restart, import the .zexp file, done!

Note that you'll need to reindex the imported content and that any related data that lives outside of the object itself is gone. For example, its intid are gone and all relationships to it will have to be recreated etc. But you just saved your customers bacon, I'm sure they won't mind a little manual work!

--
Martijn Pieters

Listening for signals from dead Zopes

Wichert Akkerman - Jarn — 2008-10-22T09:54:08Z

Occasionally the impossible, or at least very unlikely, happens: your Zope instance hangs. You can see in the process listing that it is working very hard on something but it is refusing to show you what that is.

This phenomenon is commonly known as a deadlocked Zope. This is a bit of a misnomer since you might be dealing with infinite loops instead of a deadlock.

There are two popular tools to help you debug your Zope: DeadlockDebugger and z3c.deadlockdebugger. Both provide a magic URL you can visit and which will return an overview of all stackframes for all threads. This will show you exactly what Zope is hiding from you.

Both these tools have an achilles heel: If all your Zope threads are stuck there is nothing available to process your magic URL. We ran into exactly that situation for a customer project and had to find a solution. Luckily this turned out to be simple: instead of requesting and returning the stackframe data through the webserver why not do it directly on the console? UNIX already provides us with a very useful signalling system available for: signals. Signals also have the benefit of being more secure: they can only be sent by someone who has access to the server and the account used to run the Zope instance. And thus was born a new product: Products.signalstack.

Once you have installed signalstack in your Zope instance all your need to do is send a USR1 signal to the Zope process and it dumps the stackframes of all threads to its standard output.

Installing

Installing signalstack is simple: you only need to install the Products.signalstack package in your site. If you are using zc.buildout just add it to the eggs-line for your instance and run buildout.

[instance]
recipe = plone.recipe.zope2instance
zope2-location = ${zope2:location}
eggs =
    Plone
    PIL
    Products.signalstack

If you are not using buildout you can use easy_install to install it either globally or inside your Zope instance.

Using

First you need to figure out the pid (process id) of your Zope instance. You can find this in the var/instance.pid file in your buildout. If you can not find that file look for a zope process in your process listing. Once you have found the pid you can send a signal to it:

$ kill -USR1 4361

Zope will respond in kind by throwing a lot of data at you:

Threads traceback dump at 2008-10-21 11:34:47

Thread -1340051456:
  File "/Users/wichert/Library/Zope/Zope-2.10.6-final-py2.4/lib/python/ZServer/PubCore/ZServerPublisher.py", line 19, in __init__
    name, a, b=accept()
  File "/Users/wichert/Library/Zope/Zope-2.10.6-final-py2.4/lib/python/ZServer/PubCore/ZRendezvous.py", line 73, in accept
    l.acquire()

Thread -1340583936 (GET /Plone):
  File "/Users/wichert/Library/Zope/Zope-2.10.6-final-py2.4/lib/python/ZServer/PubCore/ZServerPublisher.py", line 25, in __init__
    response=b)
  File "/Users/wichert/Library/Zope/Zope-2.10.6-final-py2.4/lib/python/ZPublisher/Publish.py", line 401, in publish_module
    environ, debug, request, response)
  File "/Users/wichert/Library/Zope/Zope-2.10.6-final-py2.4/lib/python/ZPublisher/Publish.py", line 202, in publish_module_standard
    response = publish(request, module_name, after_list, debug=debug)

[ .. removed a lot of uninteresting frames here ..]

  File "/Users/wichert/Library/Zope/Zope-2.10.6-final-py2.4/lib/python/Products/PageTemplates/Expressions.py", line 123, in render
    ob = ob()
  File "/Users/wichert/Development/plone/plone3.2/src/Plone/Products/CMFPlone/browser/ploneview.py", line 287, in showEditableBorder
    request = self.request
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/bdb.py", line 48, in trace_dispatch
    return self.dispatch_line(frame)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/bdb.py", line 66, in dispatch_line
    self.user_line(frame)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/pdb.py", line 135, in user_line
    self.interaction(frame, None)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/pdb.py", line 158, in interaction
    self.cmdloop()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/cmd.py", line 130, in cmdloop
    line = raw_input(self.prompt)

Thread -1341648896:
  File "/Users/wichert/Library/Zope/Zope-2.10.6-final-py2.4/lib/python/ZServer/PubCore/ZServerPublisher.py", line 19, in __init__
    name, a, b=accept()
  File "/Users/wichert/Library/Zope/Zope-2.10.6-final-py2.4/lib/python/ZServer/PubCore/ZRendezvous.py", line 73, in accept
    l.acquire()

Thread -1341116416:
  File "/Users/wichert/Library/Zope/Zope-2.10.6-final-py2.4/lib/python/ZServer/PubCore/ZServerPublisher.py", line 19, in __init__
    name, a, b=accept()
  File "/Users/wichert/Library/Zope/Zope-2.10.6-final-py2.4/lib/python/ZServer/PubCore/ZRendezvous.py", line 73, in accept
    l.acquire()

End of dump

It is immediately obvious that this Zope has a problem: someone forgot to remove a pdb.set_trace() statement at line 287 of ploneview.py.

--
Wichert Akkerman

Catalog query plan

Helge Tesdal - Jarn — 2008-07-29T10:15:45Z

Making the catalog faster

While the catalog tool in Zope is immensely useful, we have seen some slowdowns in large Plone sites with a combination of additional indexes and lots of content.

Analyzing Catalog implementation

The catalog implementation is using BTree set operations like union, multiunion and intersection. Those operations are fairly fast, especially when everything is in memory. However, the catalog implementation is rather naïve which leads to lots of set operations on rather big sets.

Example

A typical simple scenario is an intranet with 150000 content items, most of them published, and local roles used to handle access to most of the content.

We want to look for pending Documents in a section of the site. In addition to the specified requirements, the catalog tool in Plone would add a search for allowedRolesAndUsers and effectiveRange to only return results the user is allowed to see.

There is no explicit ordering of the indexes, so in our first example our ordering is portal_type, allowedRolesAndUsers, review_state, effectiveRange, path. The first index, portal_type returns 90000 items. Then we continue on to allowedRolesAndUsers, which returns 40000 items. This is intersected with the the previous set, yielding a set of 20000. As all the content in this section is published, there are no items in the review_state=pending, which gives an empty set when intersected with the previous set. effectiveRange returns 140000 items, which is intersected with empty set. path returns 400 items which is once again intersected with the empty result set. The end result is an empty set and the query typically takes 0.15 seconds on my laptop.

As a basic first improvement, it would help to immediately return after getting an empty result set from an index, but we can go even further.

Query plan

Search engines and databases uses query optimizers to select query plans that will minimize the result set as early as possible, because working with large amounts of data is time consuming.

What we want to do is to search against the indexes giving the smallest result set first. However, for that to be useful, we need to pass that result along into the indexes to allow the indexes to limit the result set as soon as possible internally. When calculating a path search, there is no need to look in all 150000 results if the portal type index has already limited the possible result to 10000. If we have already limited the result to 10000 results, all set operations are going to be significantly faster.

Solution

We identify different searches by the list of indexes that are searched. If there are no query plans for a set of indexes, the query is run like normal while storing the number of results for each index. When all indexes have been checked, the list is sorted on number of results and stored as a query plan. Next time a search on the same indexes comes in, the query plan is looked up.

To get different query plans for similar queries, you can provide additional bogus index names. They will be ignored by the catalog, but will become part of the key. This means that if you search for Documents in draft state for a worklist, you can have a different ordering than when searching for published Documents, as there are likely to be very few items in draft state, but many in published state.

Results

I tested this in an example site locally with a 15GB Data.fs, 150000 content items, and running a JMeter test plan that logged in, viewed a couple of pages and logged out again.

When measuring in the regular catalog, there were 1220 queries taking 81 seconds with the regular catalog, which is 66ms per query. When using the query plan, the queries took 14.1 seconds, which is 11ms per query. The catalog implementation is also used for membrane and reference catalog to look up users and objects, but those have significantly fewer indexes than the regular catalog tool, so we also measure on the Catalog Tool.

When measuring only the catalog tool, there were 510 queries taking 80.9 seconds with the regular catalog, which is 159ms per query. With the query plan it took 14.3 seconds, which is 28ms per query. We consider 14.3 vs 14.1 to be within normal variations when measuring, and the numbers indicate that the time spent in membrane and reference catalog can be ignored compared to the time spent in the catalog tool.

Going from 159ms to 28ms for typical queries is a noticeable improvement. If you are mildly abusing the catalog, this might add up to a second or more on a single page view.

Interested?

You can try these performance improvements by installing the experimental.catalogqueryplan package in your Plone site. Just add it to the eggs list in your buildout configuration and either import experimental.catalogqueryplan from your code, or load its zcml.

Clean and fast indexing in Plone

Helge Tesdal - Jarn — 2008-04-01T07:33:33Z

Getting rid of redundant reindexing the right way.

Background and motivation

We have already established the fact that indexing can be improved in Plone in a previous blog posting, where we investigated the potential for improvement by applying an insane/inspired monkeypatch.

The monkeypatch made the Plone site temporarily ignore any indexing requests, and it had side effects. Like the fact that any additional indexing triggered by content creation in factory method or initializeArchetype would be ignored.

Moving forward

The fundamental problem with indexing from a performance point of view, is that indexing is performed instantly instead of at transaction boundary. By postponing indexing until the transaction commits, we are able to filter the indexing events. This enables us to

Ignore unnecessary indexing (add followed by delete)
Only do one (re)indexing instead of many on the same object

Technical details

We added an indexing queue, which was in turn controlled by a transaction manager. The transaction manager pattern was reused from enfold.solr.

On modified event, or when (re)indexObject is called, the indexing requests are put in the queue. When the transaction commits, the queue is processed. There is a default reducer which filters duplicates (which can be overridden by registering a new adapter). Then the requests are dispatched to any queue processors registered. The default queue processor is for adding content in portal_catalog, and dispatches to CatalogAware and CatalogMultiplex. You can easily add queue processors for asynchronous queue processing, or for external indexing.

Results

When measuring improvements, we created 100 news articles using the JMeter Test Plan from the collective. Unmodified is plain Plone. Indexing refers to a site with the collective.indexing extension profile applied. Experimental refers to experimental.contentcreation (without redundant reindex hack), and E+I is experimental.contentcreation and collective.indexing together.

Note that the test results are not useful as a measure of absolute performance or to be compared with other tests. We can see that the performance improves when avoiding redundant reindexing, and now it is done in a clean and efficient way.

In a real life deployment with more indexes and metadata, and possibly file conversion as well, the improvement will be bigger.

Try it out for yourself

You can use the buildout for testing this yourself.

svn co http://svn.plone.org/svn/collective/collective.indexing/buildout indexing
cd indexing
python bootstrap.py
bin/buildout
bin/instance start

Create a new Plone site, using the collective.indexing extension profile.

Future plans

While this component is fully usable, it is also about exploring indexing in Zope in general. During the Sorrento Sprint, Malthe Borch and Sylvain Viollon were looking into Xapian integration, and found the time to start splitting collective.indexing into z3c.indexing.dispatcher, which will in turn dispatch to components like z3c.indexing.zcatalog, external indexers or asynchronous queues similar to the one in ore.xapian. For extracting data from content in Plone, there will be a plone namespace packages with adapters for data extraction.

Because collective.indexing is about adapters and utilities and not data structures, it is simple to switch parts of it as it becomes available in z3c packages, and when that's done, move it into the plone namespace and hopefully also towards inclusion in Plone core...

Explore portlet

Wichert Akkerman - Jarn — 2008-03-07T10:51:15Z

Navigating the seas of Plone

One of the most important things you need to do when using a content management system such as Plone is to find your way to your data. Typically this involves navigating through a site using the navigation portlet: for every step your browser needs to load another page so you can find the next step on your path. This can be a slow procedure since you need a full page load at each step.

A faster way to travel

To help you get to your destination faster we wrote the Explore portlet. This is an alternative to the standard Plone 3 navigation portlet which features AJAX-based navigation: simply click on the triangle next to a folder in the navigation tree and its contents will be loaded into the tree. Click again and the tree collapses again. This gives you a very fast and intuitive method to explore the contents of your site.

Getting rid of redundant reindexing

Helge Tesdal - Jarn — 2008-02-17T22:11:31Z

experimental.contentcreation cuts amount of reindexing.

Redundant reindexing

While creating a new object using the factory tool, the new object would be reindexed four times.

ObjectAddedEvent event handler
Finish construction in factory method
processForm in Archetypes.BaseObject
Rename after creation

This is 3 times more than necessary. The first two are called by factory_tool.doCreate, and the two last are in processForm.

Avoiding reindexing

All the reindexing was happening in methods called by content_edit_impl, and all reindexing went through CatalogMultiplex. In experimental.contentcreation we already take advantage of the fact that CatalogMultiplex gets a list of all catalogs to index in, and by returning an empty list, the object is not indexed. The strategy would be to let content_edit_impl change the result of the catalog listing while the reindexing method was called, then finally do 1 reindex afterwards.

First of all we needed to override the content_edit_impl script. In Plone, it is implemented as a skin script, which enables easy customization. As we needed trusted code, we added it to the patch in experimental.contentcreation and set it as an attribute to the Portal class, which prevents the use of the script in portal_skins.

Before calling doCreate and processForm from content_edit_impl, we create a lambda function that returns an empty list and set the getCatalogsByType method of the Archetypes tool object to the newly created lambda function. In addition, we set _p_changed to False on the Archetypes tool to avoid the transaction machinery trying to save the function during savepoint.

After doCreate and processForm, we delete the function object and reset _p_changed again, before doing a single and final reindexObject.

The result

No performance blog posting without a graph. We tested the setup with JMeter on a laptop, 10 loops for each scenario. The blue bar is content creation with experimental.contentcreation with 1 reindex.

Our tests show that time went down from 567ms in regular Plone to 203ms with experimental.contentcreation and the redundant reindexing removal. experimental.contentcreation without reindexing removal was 284ms.

If you have converters to handle uploaded files and more fields and indexes in the catalog, the time difference will be even bigger.

Future

We also created a plone.app.content based content type with formlib forms to compare response times compared to regular Plone. While not directly comparable and not feature complete, the numbers give an indication of what to expect in the future, and why people are looking into Zope 3 technology.

Try it out for yourself

To try it out for yourself, you can check out experimental.contentcreation from SVN, add it to your buildout (don't forget the ZCML slug) and restart Zope.

-- Helge Tesdal

Membrane performance

Helge Tesdal - Jarn — 2008-02-17T21:32:19Z

Optimization of the object_implements method in Membrane

The challenge

While looking at content creation in a Membrane scenario, we noticed that object_implements took a lot of time. object_implements is a catalog index wrapper that lists both the directly implemented interfaces and any interfaces that can be adapted into. This is used when looking up user objects, as you normally associate adapters with your member types to make them behave like users.

There was a note in object_implements about it being slow. Timing the method call while running tests indicated that the method typically took 1.2 seconds on a MBP laptop. It also did not help that objects are reindexed 4 times when creating content, but that's another story.

The problem

Previously, object_implements was using the getRequiredAdapters method from zope.app.apidoc.component to list all required adapters for an interface. getRequiredAdapters would check all adapters and instantiate a registration info object for each matching adapter. It would be called once for each interface implemented by the object currently being examined. Judging by the package name, this info object was originally intended to be used for API documentation, and contains more info than we need for this use case.

Poking the adapter registry

As we were not aware of any current methods in the adapter registries that would give the result we were looking for, we started poking the internals of the registry from membrane. The registry has a dictionary of interfaces and their corresponding adapters, and we used that to look up the adapters. If an exception occurs in the new code, we have a fallback to the old code.

Poking the internals of the adapter registry is not future proof, as we are circumventing interfaces and depending on the internal data structures of the adapter registry to stay the same. While not very likely to change, the data structure might change between Zope versions - requiring us to update Membrane.

Measuring improvement

After testing again, the time spent in object_implements was 0.0006 seconds, and the set of interfaces returned were exactly the same with both methods.

The difference was difficult to express in our regular go-faster-graphs, so we used an area plot instead. The four pixels on the right indicate the time after optimization relative to before the optimization.

This change is checked in on membrane trunk, and will be part of the next release unless there are any unforeseen problems.

--
Helge Tesdal

Plone 2.5 performance

Helge Tesdal - Jarn — 2008-02-08T14:57:14Z

Profiling of content creation and teamspace scenario in Plone 2.5, and publishing contentmenu and factoryhack.

We have blogged about performance before, after doing profiling for Plone 2.5 and during the Copenhagen Performance Sprint where we made experimental.contentcreation. Prior to experimental.contentcreation we made two Plone 2.5 products named factoryhack and contentmenu that are now available in the collective.

Performance improvements

First of all we'll take a look at what you'll get from these products by setting up some tests with jmeter. Each test is run 10 times after warming up 2 times. Neither of the tests were done on plain Plone, they have several add on products installed, and other factors that affect performance. The results can not be used as an absolute benchmark on Plone performance, but are useful for showing relative results with and without the optimizations.

The first test shows results for content creation. Listing is listing folder contents, Addform is showing the addform to add content in the folder, Create is pressing the save button and creating the content, and show is displaying the newly created content.

The second test is for a teamspace scenario. Teamspaces is viewing a listing of teamspaces, Teamspace is viewing one teamspace, Subfolder 1 is listing contents in a subfolder of Teamspace, and Subfolder 2 is another folder, parallell to Subfolder 1.

Install

If you want to try this for yourself, you can check out the products into your Products directory using command line svn or a client like TortoiseSVN:

svn co https://svn.plone.org/svn/collective/experimental.contentcreation/plone2.5/factoryhack factoryhack

svn co https://svn.plone.org/svn/collective/experimental.contentcreation/plone2.5/contentmenu contentmenu

Content menu then has to be installed in the install products control panel.

Technical

The technical details are covered pretty well in the two previous blog postings [1][2].

Factoryhack is mainly about optimizing the portal factory tool, but also the type listings. It is implemented as a monkey patch, meaning that it it automatically applies itself when Zope starts, and is completely gone as soon as you remove it from the Products directory.

Contentmenu is about improving the type listings further. It is installed in the control panel and provides a skin override. It does not alter the data in any way, and will be gone as soon as you uninstall in control panel and remove from Products directory.

Disclaimer

Note that the type listings are optimized by bypassing the regular security machinery and trying to emulate it in a more efficient way by gathering all local roles first, then check against roles required for adding content. This is working in production, but if you experience unexpected results in the add content dropdown after installing these products, you will have to uninstall them again. It does not affect actual create permissions, only the drop down menu that lists the types available. The worst that can happen is that you don't see all content types you are allowed to create, or get an error message because you are trying to add a content type without really being allowed to.

-- Helge

Announcing Plone Enterprise Forum

Geir Bækholt - Jarn — 2008-01-28T22:43:04Z

We are proud and happy to announce a new interest group and mailing list: "Plone in the Enterprise".
There is a growing interest in features for using Plone in Enterprise-level deployments like:

scaling to handle hundreds or thousands of parallel authenticated users
Enterprise search (being able to quickly search through millions of objects)
Integration
Clustering and hosting solutions
High availability architcture
more…

We are now launching a separate forum for the discussion of Plone in the Enterprise.
There are already many interesting projects happening, and the sooner we can join our efforts the better it will be for the world of large-scale Plone. Let's share experiences, needs, war stories and help each other on development of large-scale features:
The mailing list is available at
http://lists.plone.org/mailman/listinfo/enterprise

It is available in Gmane for those of us that prefer a newsgroup interface:
gmane.comp.web.zope.plone.enterprise

and as a web-forum at Nabble:
http://www.nabble.com/Enterprise-f30548.html

If you care about enterprise level Plone, Please join us!

Geir Bækholt

Zope Town, Liberia

Geir Bækholt - Jarn — 2007-11-17T21:56:52Z

By accident, we stumbled across this remote little town in Liberia, called Zope Town (sometimes also referred to as Zopita).

Fun :)

The classic portlet, a first-class Plone 3 citizen

Geir Bækholt - Jarn — 2007-11-10T23:58:26Z

Reading a recent comment about portlets in Plone 3, i was baffled by the negativity, until i realized that the man is simply suffering from the slight misconception that the classic portlet is something he is not supposed to use. (probably due to me not writing the tutorial Martin rightfully expects me to write)

Complexity

Roche is unhappy that portlets are now more complex than they were before.
Fair enough. This is a pattern we have seen for thousands of years. To accomodate new needs or desires, growth or increasing complexity of society, technology grows more complex.

Let's use flight as an example: A hot air balloon is a fairly simple instrument, and can be built by almost anyone with motivation and a bit of resources. But even though hot air balloons can be perfectly good (even the best) for many use cases, they also have limitations; they are slow, they have limited cargo capabilities and are hard to navigate. Upgrading to a simple aeroplane may be tempting as it can help resolve these issues. But upgrading from a balloon to a plane will also lead to more complex technology; aerodynamics, combustion engines, airfield construction… Moving another step up the ladder into organised passenger traffic, you'll end up with airports, air traffic controllers, ticket management systems and lots of organizational issues too.

The neat thing is that even in today's world of organized passenger air traffic, there is room for the hot air balloon. In many scenarios, the simple, flexible and agile tool (not to mention cheap), is preferable to complex engineering.

Give me a classic!

Many of us have been suffering from the lack of a proper portlet management system in Plone. Given a deployment of any reasonable complexity, it was just too much to handle with the tools we had.
The reason i am writing this is that some people seem to be suffering from the misconception that the old-style portlet, the classic portlet, — the hot air balloon of portlets, is deprecated and not recommended in real-life use. This is completely wrong! Given the late trend of integrators being suspicious of the added complexity and allegedly lacking TTW-support of Plone 3, i feel it is necessary to clarify this bit for the non-developers out there:

The classic portlet, a first-class Plone citizen:

A lot of effort (Kudos to Martin) has been put into making it simple to work with portlets for integrators. And even better: in addition to being able to continue working with the portlets as before you get to combine it with the powers the new portlet engine brings, and add extra free superpowers without having to break a sweat: Even with single-template-based classic portlets you get new stuff from plone.portlets and plone 3:

Your portlet can be assigned (and unassigned or blocked) in the UI, in Plone, not just in ZMI valley where no real user will ever venture.
Your portlet can now be assigned to users' dashboards
Your portlet can now be assigned to groups of users
Your portlet can now be assigned to content types

In exchange for all these new features, you have to sacrifice exactly *nothing*. You don't even have to rewrite your template. You don't have to touch three python classes or any zcml files. — Simply add a classic portlet in the plone interface, paste the path to your template and you are good to go!

In addition to all this, developers will also soon start distributing usable portlets as eggs, so you can plug all sorts of extra functionality on top — and you'll still not be chewing on the extra complexity.

--
Geir Bækholt

One cookie please, but hold the pickles

Martijn Pieters - Jarn — 2007-11-09T23:08:33Z

The python pickle module is dangerous, didn't you know?

All your base are belong to us

By now you all should have installed last Tuesday's Hotfix. If you haven't yet, but are running Plone 2.5 or Plone 3.0 websites, you should do so yesterday, or at least as soon as humanly possible.

The Hotfix patches a serious security problem in the statusmessages and linkintegrity modules, where network-supplied data was interpreted as pickles. "Network-supplied" data in this case means both cookies and form data, and no authentication is required to exploit the holes.

What happen ?

The basic problem with the holes is that the Plone community was totally unaware of how dangerous the pickle module really is. Hanno Schlichting did file a report a few months ago stating that the code was potentially dangerous, but even he didn't fully appreciate that pickles are a security hole only waiting for attacker input. The scary thing here is that the code in question was written by extremely capable and experienced developers, but none of them were aware of the fact that you cannot ever use pickles to load user-supplied data.

What is needed then, is education. This is my contribution.

You are on the way to destruction

So what is wrong with pickles? They are just a damn handy way to serialize arbitrary data into binary strings and back again, right?

Yes, they are that, but the pickle format used is also a simple stack language that allows the creation of arbitrary python structures, and execute them. This stack language allows you to import modules (the 'c' symbol), and apply arguments to callables (the 'R' symbol), thus causing code to be run. Combine this with the python built-in methods eval and compile and you have the perfect vehicle for an attacker to have the pickle loader routine execute arbitrary python code when loading a well-crafted pickle. Just image what an attacker could do with that to your Zope server. Do you think you'll ever be sure you got all the backdoors out of your Data.fs?

We get signal

So next time you need to preserve data across HTTP requests, please do not be tempted to use the pickle module to create strings for you. Rarely will you have anything more than a handful of simple datatypes to pass along anyway, so just invent a simple dataformat and use that instead. (No, using a subclass of the python implementation of pickle is not a simpler solution).

With statusmessages for example, each message consists of a message and a type string, both unicode. So we changed to a hand-rolled format using a 2 byte length header (11 bits of message length, 5 for the type) directly followed by the message and type strings (encoded to utf-8). When reading this from a cookie again later, the decoder simply has to read the lengths from the first 2 bytes, then read the right amount of characters to get the message and type back. A similar method was used to encode the linkintegrity data. Simple, effective, and impervious to attacks.

--
Martijn Pieters

Congratulation!!
A.D.2111
All bases of CATS were destroyed.
It seems to be peaceful.
But it is incorrect. CATS is still alive.
ZIG-01 must fight against CATS again.
And down with them completely !
Good luck.
-- Zero Wing, 1989

FeedMixer

Wichert Akkerman - Jarn — 2007-11-05T21:54:40Z

Hungry for information

The world is a highly dynamic place: an amazing amount of things are happening around us at an incredible place. And we humans, being the curious beings that we are, want to know exactly what is happening. Our need for information is so large that we have build an army of tools to manage information. The latest trend in that area are Web Feeds: channels with output little bits of information you can subscribe to.

Feeds have several good qualities: they are easy to access, and you can do so using any of the thousands libraries and applications that support the popular RSS and Atom formats. This also makes them very useful as a method to get data from one system to another system.

A feed on my own menu

One place where we want to be able to use feeds is on our websites. Particularly we have a customer who was interested in being able to accumulate data from several feeds and present them as a single listing of news items. Plone 3 already included a RSS portlet which could handle a single feed and present that in a portlet. Originally we planned to extend this portlet to be able to handle multiple feeds. This quickly turned out to be a bad idea: the RSS portlet was designed to act as a very simple to use window into a news source, and does a very good job at that: it is easy to configure, automatically picks up title and URL for the news source and outputs a standard Plone portlet using those. Extending that to handle multiple feeds would result in a something that was too complex for its original use case and not convenient for multiple feeds as well. The worst of both worlds.

So we scrapped that idea and implemented a new package: FeedMixer. Or collective.portlet.feedmixer if prefer the exact python package name. Just like the standard Plone 3 RSS portlet FeedMixer aims to be as simple as possible to use. You can specify the portlet title, number of items to show and as many URLs as you want and in return you will get a portlet with the latest news items from all your sources. And if you want to see more results there is also a handy page which shows the complete entry list from all feeds.

Do you serve fast food, euh, feed?

One problem with feeds is that they are generally not very fast: software which processes feeds regularly asks feed publishers for the latest which can put a huge load on publishers. On the client side you do not want to wait for a system to reload the feed data, process it and show you the results. You want your feed, and you want it fast.

FeedMixer accomplishes that by reusing the caching infrastructure in Plone 3. It uses the memoize caching infrastructure to find a global cache where all feed data can be stored. Normally this is a RAMCache, but if you are using multiple Zope instances you can easily change this to use a shared memcached cache instead. That means that if one of your Zope instances refreshes a feed all your instances will immediately be able to use it. Since the Zope caches may have a longer timeout setting then you want to use for your feed data you can also configure a maximum timeout directly in a FeedMixer portlet.

If FeedMixer needs to refresh data it will try to use the ETags and the HTTP Last-Modified header to only download the feed if it has changed on the server.

--
Wichert Akkerman

Small change, big effect

Martijn Pieters - Jarn — 2007-11-04T12:54:08Z

How changing one line halved the time it took to rename a Plone folder.

Here at the Plone Performance sprint, Matt Hamilton and Sasha Vincic are homing in on the Catalog and folder renaming. As Sasha already reported earlier, they identified the object_provides index as a potential bottleneck.

The object_provides index

The index is filled with interface identifiers, strings representing the actual interfaces. The data for the index comes from a small method in Products.CMFPlone.CatalogTool, object_provides, which looked like this:

def object_provides(object, portal, **kw): 
    return [interfaceToName(portal, i) for i in providedBy(object).flattened()]

So, for each interface declared by an object, interfaceToName is invoked. The purpose of interfaceToName is to provide a way to turn an interface to a string that can be used to later turn that string back into an interface, through the queryInterface method, interfaceToName's sibling.

Now the problem with interfaceToName is that it has to iterate over the whole utilities registry to find all interfaces registered as utilities just to find out what name it was registered with. This is a slow process, but a necessary one; although the default name for an interface is it's dotted name (the python identifier path to their definition), some special classes of interfaces are registered with a different name. For example, when registering a Zope3-style browser menu, an interface is generated for the menu, and registered with a zope.app.menus prefix.

No need for interfaceToName

Luckily, for the object_provides index use-case, interfaceToName is overkill. First of all, object_provides indexes declared interfaces on content objects only, and therefore will never encounter any of the "special" interfaces.

But more importantly, the index contents are never used to find the original interfaces again. Quite the contrary, it is only used to search what objects provide a given interface, and the developer querying the catalog will have to generate the same string format every time they search. So, with the index using interfaceToName to fill the index, searching the index also requires developers to use interfaceToName to query the index. Search for IATFolder? Pass in interfaceToName(IATFolder) and hit the same performance problem.

Unique identifier

So if interfaceToName is overkill, what unique identifier should we use then? As we already mentioned, when you register an interface in the first place, the default name is the dotted name of the interface. It's a unique identifier, as it's the name under which python stores it in memory. It is available as the __identifier__ attribute on the interface. As it's unique, and available directly from the interfaces themselves, it's ideally suited for both indexing and searching.

Of course, this means that if we use __identifier__ then you should use the same attribute when querying the index. Because __identifier__ (or __module__ + '.' + __name__, which is the same) is already the default for interfaceToName, this is what Plone developers have been using anyway.

So we changed the indexing method to:

def object_provides(object, portal, **kw): 
    return [i.__identifier__ for i in providedBy(object).flattened()]

and presto, indexing was more than twice as fast, as shown by the pretty graph on the right. We tested this by having JMeter rename a folder with 20 documents in it, 40 times.

Not bad for a one-line change.

--
Martijn Pieters

Jarn Blog

Plone performance sprint bristol 2008

Location

Sprint topics

Automated performance tests

Instrumentation

ExtendedPathIndex refactoring

Catalog

Saving the day: recovering lost objects

Oh noes!

Time travel

Recovery

Listening for signals from dead Zopes

Installing

Using

Catalog query plan

Analyzing Catalog implementation

Example

Query plan

Solution

Results

Interested?

Sponsored by

Clean and fast indexing in Plone

Background and motivation

Moving forward

Technical details

Results

Try it out for yourself

Future plans

Sponsored by

Explore portlet

Navigating the seas of Plone

A faster way to travel

Sponsored by

Getting rid of redundant reindexing

Redundant reindexing

Avoiding reindexing

The result

Future

Try it out for yourself

Membrane performance

The challenge

The problem

Poking the adapter registry

Measuring improvement

Plone 2.5 performance

Performance improvements

Install

Technical

Disclaimer

Announcing Plone Enterprise Forum

Zope Town, Liberia

The classic portlet, a first-class Plone 3 citizen

Complexity

Give me a classic!

The classic portlet, a first-class Plone citizen:

One cookie please, but hold the pickles

All your base are belong to us

What happen ?

You are on the way to destruction

We get signal

FeedMixer

Hungry for information

A feed on my own menu

Do you serve fast food, euh, feed?

Small change, big effect

The object_provides index

No need for interfaceToName

Unique identifier