http://www.jarn.com News, happenings and rants from the Jarn team. daily 1 2007-08-20T10:53:11Z http://www.jarn.com/blog/plone-cathedral-sprint-after-action-report Short report from the Plone 4.0 and 4.1 Cathedral sprint in Cologne, Germany. <p>A small delegation from Jarn, Denys Mishunov, our German freelancers Tom Lazar and Andreas Zeidler, and myself, participated in the Cathedral Sprint in Cologne, Germany, March 15-19 2010, generously sponsored and hosted by <a class="external-link" href="http://www.gfu.net/">GfU Cyrus AG</a>. We had a great time, a week with awesome Plone people, great mindshare, wonderful productivity and German hospitality, food and beer.</p> <p>The sprint focused on features and finishing for the upcoming Plone 4 and 4.1 releases and saw incredible results in the 5 days of hard work from 25 sprinters.</p> <p><img class="image-left" src="../images/cathedralsprinters/image_preview" alt="Cathedralsprinters" /></p> <p style="clear: left;" class="discreet">Photo © Armin Carl Stroß-Radschinski.</p> <p>Here are some highlights based on <a class="external-link" href="http://www.acsr.de/archive/plone-cathedral-sprint-in-koln-war-extrem-produktiv">Armin Carl Stroß-Radschinski's sprint summary in German</a>, generously translated to English by <a class="external-link" href="http://www.matthewwilkes.co.uk/">Matthew Wilkes</a> and crippled for brevity by me. Read the <a class="external-link" href="http://www.acsr.de/archive/plone-cathedral-sprint-in-koln-war-extrem-produktiv-en">original summary (english) for more details here</a>.</p> <h2>Results<br /></h2> <p>I have tried listing some of the achievements from the sprint — beyond getting to spend time with the best Open Source community on the planet. I apologize for any names or features i have missed (and i'll be happy to take corrections).</p> <h3>Reimplementation of Plone collections <br /></h3> <p class="discreet">Eric Steele, Matthew Wilkes, Roel Bruggink, Maarten Kling, Ralph Jacobs , Rob Gietema, Martijn Jacobs and Geir Bækholt</p> <p>Collections in Plone are saved searches with complex criteria whose results are shown with a given ordering. The improvements to the interface of these 'smart folders', which will also apply to searches, has made their usage significantly more intuitive.<br />The interface for entering search criteria shows search results in live time as the query is edited, in the same way as the Live-Search feature does for simple searches in Plone 3. More criteria can be added step-by-step to narrow the search even further. The results are updated real-time for instant user feedback.<a class="external-link" href="http://blip.tv/file/3367082/"><br /><img class="image-inline" src="../images/film-icon/image_icon" alt="film icon" /> Screencast of the new collections</a></p> <h3 style="clear: left;">Search improvements</h3> <p class="discreet">Denys Mishunov, Israel Saeta Pérez, Nejc Zupan</p> <p>Great improvements to search results display, and a new, improved advanced search form with instant Ajax gratification.<br /><a class="external-link" href="http://blip.tv/file/3367086/"><img class="image-inline" src="../images/film-icon/image_icon" alt="film icon" /> Screencast on search improvements</a></p> <h3>Event upgrades</h3> <p class="discreet">Andreas Jung, Vincent Fretin, Johannes Raggam, Tom Gross and Rodrigo Ristow</p> <p> Further improvements to the plone calendaring system, incorporating features like new (i18n) widget, full-day-events, recurring events, iCal export and more.<br /><a class="external-link" href="http://blip.tv/file/3362551/"><img class="image-inline" src="../images/film-icon/image_icon" alt="film icon" /> Screencast of event improvements</a> or r<a class="external-link" href="http://www.zopyx.de/blog/plone.app.event">ead more at Andreas Jung's blog. </a></p> <h3>Content Browsing<br /></h3> <p class="discreet">Robert Niederreither</p> <p>A new Java-Script based interface for fast and efficient navigation of the content structure of a site. The view, based on the split-pane view of MacOSX's finder is, through direct queries to the Plone catalogue, extremely fast compared to the conventional method of loading each new page in turn, and is displayed using a CSS overlay.<br />The new interface can be called up from any page in the site - you immediately get an overview of the viewed content's context.<br /><a class="external-link" href="http://www.zope.de/redaktion/dzug/nachrichten/plone-cathedral-sprint-ein-finder-fuer-plone"><img class="image-inline" src="../images/film-icon/image_icon" alt="film icon" /> Screencast of Plone Finder<br />German article from Jan Ulrich Hasecke </a></p> <h3>New content migration strategy for Plone 4</h3> <p class="discreet">Tom Lazar and Carsten Senger</p> <p>A combination of Plone 3 and Plone 4 extensions that use the content migration pipelines from Transmogrifier. With this the content of an existing site can be exported in a Plone-version agnostic form and then reconstructed in a fresh Plone 4 instance.</p> <h3>Translations</h3> <p class="discreet">Jan Ulrich Hasecke, Martijn Schenk</p> <p>German and Dutch Translations for Plone 4</p> <h3>Plone Discussions</h3> <p class="discreet">Timo Stollenwerk and Florian Friesdorf</p> <p>The new commenting engine for Plone, with moderation and other useful improvements.<a class="external-link" href="http://blip.tv/file/3365701/"><br /><img class="image-inline" src="../images/film-icon/image_icon" alt="film icon" /> Screencast of Plone discussions</a></p> <h3>Performance / image scales<br /></h3> <p class="discreet">Andreas Zeidler,Simon Pamies</p> <p>Performance improvements including handling of image data and a complete revamp of image scales.&nbsp;</p> &nbsp; <p>For more info on the sprint, <a class="external-link" href="http://www.hasecke.com/nachrichten/plone-cathedral-sprint-2013-personal-wrap-up">Jan Ulrich Hasecke has also written a good personal summary of events</a> while <a class="external-link" href="http://www.flickr.com/photos/12317756@N08/sets/72157623659097492/">Armin Carl Stroß-Radshinski has posted a lot of photos from the sprint on Flickr</a>.</p> -- <a class="external-link" href="../people/baekholt"><br />Geir Bækholt</a> <p>&nbsp;</p> <p><a class="external-link" href="http://www.flickr.com/photos/12317756@N08/sets/72157623659097492/"><img class="image-left" src="http://farm5.static.flickr.com/4068/4460527505_f2e2d72c24.jpg" alt="Cathedralsprinters in action" /></a></p> <p style="clear: left;" class="discreet">Photo © Armin Carl Stroß-Radschinski.</p> No publisher Geir Bækholt - Jarn 2010-03-26T21:58:27Z Blog Posting http://www.jarn.com/blog/linguaplone-3.0 After one year of work a new final LinguaPlone release is out. <p><img title="Norden och EU - by Johannes Jansson/norden.org" class="image-right" src="../images/norden-och-eu/image_mini" alt="Norden och EU - by Johannes Jansson/norden.org" />After more than a year of work, we released a new final version of LinguaPlone a couple of days ago. LinguaPlone is the multi-lingual solution for Plone.</p> <p>While we have released many preview releases over the last months, we'd like to take the final release as an opportunity to give an overview of all the work.</p> <p>Note that LinguaPlone 3.0 only works with Plone 3.3 and the upcoming 4.0. The Plone 4.0 support is still preliminary, as Plone itself is still in early alpha stages. If you use an older Plone 3 release, you will need to use LinguaPlone 2.4 or upgrade your Plone version.</p> <p>Upgrading from earlier versions of LinguaPlone to the new 3.0 release requires an upgrade. Please visit the Add-on products control panel and execute the upgrade for LinguaPlone. Note that this requires a full catalog update and can take a substantial amount of time depending on the amount of content in your site.</p> <p>The following list gives a summary of the main changes in 3.0. A full list of all changes can be found on the <a class="external-link" href="http://pypi.python.org/pypi/Products.LinguaPlone#changelog">product page</a>.</p> <h3>Performance</h3> <p>We substantially increased performance most notably in large sites. There's a huge number of internal technical changes mostly related to the use of the Archetypes reference engine. These improve both general rendering time of all pages as well as content editing operations. LinguaPlone will now use the catalog much more effectively and require less objects to be loaded from the database.</p> <h3>Multi-lingual references</h3> <p>So far reference fields or references in general haven't been&nbsp;multi-lingual aware.</p> <p>Imagine creating a page at /source and one at&nbsp;/target. Now add "target" as a related item to "source". If you translate both pages into German, you end up with /source-de and /target-de. But "source-de" would point to "target" as a related item and not "target-de".</p> <p>The same problem occurs if you use the visual editors "link by UID" feature and have links inside the body text of your pages.</p> <p>In LinguaPlone 3.0 these references now point to the expected target. So "source-de" points to "target-de" if "target-de" exists. If the target is not translated into the translations language, the reference points to the target in the source language.</p> <p>This behavior applies to all reference fields marked as language neutral.</p> <h3>Updated language selector</h3> <p>LinguaPlone's language selector has been updated considerably. Both using language names and flags are supported, using names per default.&nbsp;The names are now always in the native language. So you'll see "English", "Deutsch", "Norsk" and no longer English only names.</p> <p>When switching languages the current context will be preserved as much as possible. This includes proper handling of default pages, different views and query string arguments.</p> <p>If the current content item has no translation into the desired language, it will try to find the "nearest" translation in the hierarchy, instead of staying at the current page.</p> <p>A new content language selector is enabled by default, which ensures that the UI language always reflects the language of the content item.</p> <h3>Editing</h3> <p>The "Translate into" menu no longer contains an entry pointing to the language control panel. The entry has repeatedly been mistaken as affecting the individual page, instead of the entire site. The control panel is now only available from the site setup screens (already in 2.4).</p> <p>The split-screen translation edit view hides the right column, giving more space for the actual content editing. The behavior can be turned off via a setting in portal_properties.</p> <p>Language independent fields are shown in view mode in the translate edit screens. This prevents editors from mistakenly changing these fields affecting the canonical and all translations (already in 2.2).</p> <p>The metadata language in the edit screens only shows the supported languages in the site. So far you could accidentally change a content item to an unsupported language.</p> <h3>Technical changes</h3> <p>archetypes.schemaextender has been updated to seamlessly support LinguaPlone. Fully functioning support for language independent fields requires version 1.3 of schemaextender.</p> <p>The latest versions of plone.app.blob are fully multi-lingual aware. In standard LinguaPlone the content of language independent fields is copied to all translations, duplicating the entire data. Blobs are able to avoid this issue and share the same underlying blob data for all translations. So a large pdf document of 100mb will only be stored once in the blob storage on the filesystem, instead of once for each language in the ZODB.</p> <p>There's quite a number of internal changes, exposing various aspects of the translation logic as separate adapters or making certain features configurable via environment variables. A number of outdated code like the custom translation cache has been removed.</p> <h3>Contributions</h3> <p>Most of the work done by the Jarn team has been sponsored by the <a class="external-link" href="http://www.norden.org/en/">Nordic Council and Nordic Council of Ministers</a>. Many thanks to the organization and the communications department in particular.</p> <p>Significant contributions have been made by Jens Klein helping with the dark corners of the Archetypes reference engine and by Maurits van Rees helping with Plone 4.0 compatibility.&nbsp;</p> <p>We have many more improvements planned for LinguaPlone in the coming months. Stay tuned while Plone's multi-lingual support gets even better.</p> <p><a class="external-link" href="../people/hanno">Hanno Schlichting</a>, Jarn AS</p> No publisher Hanno Schichtling - Jarn 2009-12-23T18:05:41Z Blog Posting http://www.jarn.com/blog/jarn-pledges-super-effort-for-the-22nd-plone-tune-up <p><img class="image-right" src="../images/sledgehammer/image_mini" alt="Sledgehammer" />Plone Tune-Ups are regular online events that aim at addressing tickets in the Plone issue tracker and advancing the Plone open source Content Management System in general.</p> <p>For the <a class="external-link" href="http://plone.org/events/plone-tuneups/tuneup-22">22nd Tune-Up</a> the tune-up-team is asking Plone companies to pledge development time to make a huge push forward in bugfixing and also boost participation in general.&nbsp;</p> <p>The Jarn team of employees and freelancers makes an extra effort and pledges 30 man-hours of expert, veteran Plone developer time!&nbsp;</p> <ul><li><span class="Apple-style-span">Helge Tesdal</span></li><li><span class="Apple-style-span">Martin Opstad Reistadbakk</span></li><li><span class="Apple-style-span">Martijn Pieters</span></li><li><span class="Apple-style-span">Geir Bækholt</span></li><li><span class="Apple-style-span">Andreas Zeidler</span></li><li><span class="Apple-style-span">Stefan Holek</span></li><li><span class="Apple-style-span">Florian Schulze</span></li></ul> <p>From <a class="external-link" href="http://plone.org/events/plone-tuneups/tuneup-22">friday morning (October 23rd 2009)</a> Central European Time we will be working on plone-bugs in the #plone-tuneup channel on the freenode irc network.&nbsp;</p> <p>&nbsp;</p> <p>&nbsp;</p> <p>Join us!</p> <p>Geir Bækholt, Jarn</p> No publisher Geir Bækholt - Jarn 2009-10-22T16:48:16Z Blog Posting http://www.jarn.com/blog/faster-memcachedmanager MemcachedManager goes to 11 with refactoring and pylibmc <p><a class="external-link" href="http://pypi.python.org/pypi/Products.MemcachedManager/">MemcachedManager</a> enables Zope to cache data in one or more shared <a class="external-link" href="http://www.danga.com/memcached/">memcached</a> instances instead of in memory in each Zope process. This provides a reasonable tradeoff between added overhead and memory consumption. So far, we have been using python-memcached as a library for talking to memcached. python-memcached is compact and convenient, but can never compete with C-based libraries on speed.</p> <h3>pylibmc</h3> <p>While <a class="external-link" href="http://gijsbert.org/cmemcache/">cmemcache</a> never really took off as a memcached library for python, <a class="external-link" href="http://lericson.blogg.se/code/category/pylibmc.html">Ludvig Ericson</a> has now made <a class="external-link" href="http://pypi.python.org/pypi/pylibmc">pylibmc</a> which is a wrapper around <a class="external-link" href="http://tangent.org/552/libmemcached.html">libmemcached</a>, and promises significant speedups.</p> <h3><img class="image-right" src="../images/memcachedmanager-and-pylibmc/image_preview" alt="MemcachedManager and pylibmc" />Improving MemcachedManager</h3> <p>As we tested pylibmc and the retrieval of data had become significantly faster, it became obvious that some profiling and improvements to MemcachedManager itself was appropriate. The changes was mainly to <a class="external-link" href="http://dev.plone.org/collective/changeset/82120">remove unused code and avoid unnecessary invocation of DateTime</a>.</p> <h3>Testing the improvements</h3> <p>We set up two tests. The first one is synthetic and returns a list of 25 dictionaries, each dictionary containing 3 elements with Title, Description and url. This is similar to caching a content listing, and it also uses pickling which adds some overhead but will usually be used when caching anything in Zope. The cached script was then called 1001 times from another script, for a total of 4004 calls to get content from memcached.</p> <p>python-memcached and MemcachedManager 1.0rc1 took 1382m. When applying the MemcachedManager improvements, that time went down to 1014ms, and adding pylibmc to the mix gave a result of 812ms.</p> <p>The second test is for a development site with plenty of content and Membrane based users. Membrane is mainly used for caching user info, which are rather largeish and complex objects. Viewing a page 10 times gave 620 calls to get content from memcached.</p> <p>python-memcached and MemcachedManager 1.0rc1 took 789ms for those 620 calls, the improved MemcachedManager gave a barely noticeable improvement of 707ms, while pylibmc got a chance to really shine and only took 355ms.</p> <h3>Test for yourself<br /></h3> <p>The improvements to MemcachedManager have been released as 1.0rc2. If you want to compile pylibmc with Python 2.4, you have to apply a patch and point to the correct libraries and include files. We have created a buildout for your convenience.</p> <pre>svn co http://svn.plone.org/svn/collective/Products.MemcachedManager/buildouts/pylibmc pylibmc cd pylibmc python bootstrap.py bin/buildout</pre> <p>Then you have to start memcached with</p> <pre>bin/memcached</pre> <p>and start zope like you're used to.</p> <p>MemcachedManager is available as a type in the ZMI, and can be used as a drop in replacement for RAMCache.</p> <p>-- Helge Tesdal</p> No publisher Helge Tesdal - Jarn 2009-03-25T12:50:10Z Blog Posting http://www.jarn.com/blog/berlinale-sprint-2009 Spent your 10% time with us in Berlin! <img title="Berlin TV tower" style="padding: 0pt 0pt 1em 1em; float: right;" src="berlin-tv-tower/image_preview" alt="Berlin TV tower" height="400" width="162" /> <p>Ever since Jarn — or rather Plone Solutions at the time — <a class="external-link" href="the-10-plone-manifesto">declared the 10% Manifesto</a>, setting aside time to work on Plone while not interfering with ongoing projects has been somewhat tricky. Many times 10% time ended up being deferred or piled up over time. Of course, another reason for this was that projects at Jarn are usually a challenging and fun thing to do. So spending time on Plone all by yourself sometimes just didn't seem as appealing as working on things together with your team mates.</p> <p>Overall following up on the 10% Manifesto didn't quite work as we had hoped for. So we came up with another approach: accumulate time and then once in a while schedule a few days together in order to work on and improve Plone. In other words: sprint! We also decided it would be even more fun to invite people to stop by and join in. That is, have small but <a class="external-link" href="http://plone.org/events/sprints">proper Plone sprints</a> instead of just internal ones.</p> <p>And here we go: the first one will be held in the somewhat <a class="external-link" href="http://maps.google.com/maps/ms?ie=UTF8&amp;hl=en&amp;msa=0&amp;ll=52.492511,13.42355&amp;spn=0.000908,0.001776&amp;t=h&amp;z=19&amp;msid=108583178989000445492.00046162eba6239e437c7">semi-official new Jarn office</a> in Berlin. Actually it's just a shared office space where everybody happens to work for Jarn most of their time, but that's another story.</p> <p>The first Jarn 10% Sprint will run February 9th—11th, in Berlin Neukölln. Not quite accidentally it was scheduled in parallel with the Berlinale Film Festival and hence dubbed "Berlin(ale) Sprint". That's also the reason&nbsp;<a class="external-link" href="http://www.openplans.org/projects/berlinale-sprint/participants">most of the sprinters will arrive a bit earlier</a>...</p> <p>So if you happen to be in the area, or if you want to follow up this year's UI sprint in Baarn with some actual coding :) — not to mention having cake &amp; coffee on top of <a class="external-link" href="http://en.wikipedia.org/wiki/Fernsehturm">Berlin's TV tower</a>, <a class="external-link" href="http://play-berlin.de/">playing games</a> and taking a tour of some of our favourite pubs — <a class="external-link" href="http://www.openplans.org/projects/berlinale-sprint/">let us know</a> and join the fun!</p> <p>-- Andreas Zeidler, Tom Lazar and Stefan Holek</p> No publisher Andreas Zeidler - Jarn Blog Plone Jarn 2009-01-26T18:50:34Z Blog Posting http://www.jarn.com/blog/plone-performance-sprint-bristol-2008 A summary of the performance sprint in Bristol, 2008 <p>It's been some time since the performance sprint in Bristol now, but it certainly deserves a summary.</p> <h3><img class="image-right" src="../images/castle-park-bristol/image_mini" alt="Castle Park, Bristol" />Location</h3> <p>Turns out Bristol is a great city with a wonderful atmosphere. There is plenty of history and old buildings (unlike in Norway). Netsight also picked a great location for the sprint, and provided plenty of refreshments to keep the geeks doing long days.</p> <h2><a class="external-link" href="http://www.openplans.org/projects/plone-performance-sprint-2008/topics">Sprint topics</a></h2> <h3>Automated performance tests</h3> <p>The most important result from the performance sprint is probably the <a class="external-link" href="http://www.openplans.org/projects/plone-performance-sprint-2008/standard-performance-scalability-test-suite-buildout">automated funkload tests</a>. In Copenhagen in 2007 we looked into using JMeter, but it was too much hassle to work with, and we didn't get beyond creating some <a class="external-link" href="http://dev.plone.org/collective/browser/JMeterTestPlans/trunk">test plan templates</a> for Plone.</p> <p>With <a class="external-link" href="http://dev.plone.org/collective/browser/collective.loadtesting">collective.loadtesting</a> we have something that is a lot easier to set up and use in a buildout, and we can have daily performance runs and keep track of the general performance of Plone (although it doesn't seem to be set up yet). This means we can easily and immediately spot changes that affect performance in a negative way, and make informed decisions whether the advantages of functionality or code changes is worth the performance loss.</p> <h3>Instrumentation</h3> <p><a class="external-link" href="http://www.openplans.org/projects/plone-performance-sprint-2008/instrumentation-zodb-transactions-and-catalogging">Instrumentation</a> was the other important topic at the sprint. We always need more performance data, more accurate information, and better ways of presenting that data to enable us to understand and improve the system we're profiling.</p> <p>Enter <a class="external-link" href="http://pypi.python.org/pypi/mr.bent/">mr.bent</a>. Mr. Bent <a class="external-link" href="http://wiki.lspace.org/wiki/Mavolio_Bent">knows his numbers</a>, and is a framework for allowing profile data to be collected in a Python application and viewed at different logical levels.</p> <p>One example of usage is <a class="external-link" href="http://dev.plone.org/collective/browser/collective.performancecolouriser">collective.performancecolouriser</a> which enables the developer to visually see timing info for different page components by color.</p> <h3>ExtendedPathIndex refactoring</h3> <p>While the ExtendedPathIndex has worked quite well for a long time, the implementation was rather obtuse, or obfuscated if you like.</p> <p>Martijn Pieters has been planning a rewrite for quite some time, and at the sprint he finally had the opportunity to do so. He found several bugs in the old implementation, added tests and implemented all features for all scenarios, while also improving performance by being smarter about set operation ordering.</p> <p>The improvements were released as version <a class="external-link" href="http://pypi.python.org/pypi/Products.ExtendedPathIndex/">2.5</a>, and will be included in Plone at the earliest convenience.</p> <h3>Catalog</h3> <p><a class="external-link" href="http://pypi.python.org/pypi/experimental.catalogqueryplan">experimental.catalogqueryplan</a> saw two improvements. The first one is faster set operations when intersecting a large and a small set. In Plone, especially with Membrane, there are some cases where you have a rather small result set from one index (like user or group ids) and a large result set from another (implemented interfaces or permissions). In those cases it is more efficient to check if the large set has a certain key by direct lookup than to scan all keys for a match. The size check is implemented in Python as a temporary monkeypatch used for the catalogqueryplan indexes, and gives a noticeable improvement. The exact improvement depends on whether all values in the small set are found at the beginning ('Low values' in graph) or end ('high values' in graph) of the large set. It might be as much as 20x faster to check for a key.</p> <p><img class="image-right" src="../images/intersection-performance/image_preview" alt="Intersection performance" />The graph shows the timing of an intersection between a set of 10000 items and a subset of 10 items, repeated 100 times. The green bar is the regular C implementation from IIBTree, the yellow bar is our python implementation using has_key with fallback to the C implementation, and the blue bar is using the builtin Python set (instead of the IISet from Zope).</p> <p>Inside the indexes, we now sort the sets on length when doing intersection operations, to use the smallest sets first. When doing OR operations, we use multiunion instead of several unions. This should be noticeable for the permissions index (allowedRolesAndUsers) for example.</p> <p>The improvements are released as version <a class="external-link" href="http://pypi.python.org/pypi/experimental.catalogqueryplan/">1.1</a>, and can easily be tested by including experimental.catalogqueryplan in your buildout.</p> <p>The second improvement is the inclusion of a new index. <a class="external-link" href="http://dev.plone.org/collective/browser/Products.BooleanIndex/trunk">BooleanIndex</a> is a simplified FieldIndex which only stores True values and ignores everything else. This will lower the object count slightly for indexes like default page and folderish and will enable the use of the intersection improvements for smaller sets. This only works as long as there are fewer True than False values indexed.</p> <p>-- Helge Tesdal</p> No publisher Helge Tesdal - Jarn 2009-01-05T18:12:07Z Blog Posting http://www.jarn.com/blog/saving-the-day-recovering-lost-objects When a customer discovers over a week later that an important object was accidentially deleted, what do you do? <h3>Oh noes!</h3> <p>A customer discovered that an important entire section of his site was missing and asked us to bring it back. This was in a heavily edited site, with loads of writes each day, but we quickly located the offending transaction: someone had deleted the object in question 9 days earlier.</p> <p>Undo was no longer an option, though: too many things had changed, not least the catalog. Truncating the Data.fs (removing all transactions since, including the offending one) was not only undesirable, but impossible as the site stores the data in Oracle through <a href="http://wiki.zope.org/ZODB/RelStorage">RelStorage</a>.</p> <img class="image-right" src="resolveuid/9bad330fe62bf3ce4053922ab7ce61ca" alt="Antique timepiece" /><h3>Time travel</h3> <p>So, instead of permanently removing transactions, we used a handy little package to do some time traveling: <a href="http://pypi.python.org/pypi/zc.beforestorage">zc.beforestorage</a>.</p> <p><code>zc.beforestorage</code> does require a ZODB version 3.8 or 3.9; the customer installation is on Plone 3.0, so a newer ZODB3 egg was necessary for this operation. A small additional buildout configuration file (saved as beforestorage.cfg) helps out:</p> <pre>[buildout]<br />extends =<br /> buildout.cfg<br />eggs +=<br /> zc.beforestorage<br /> ZODB3<br /> zope.proxy<br /><br />[versions]<br />ZODB3 = 3.8.1<br />zope.proxy = 3.4.2<br /><br />[relstorage-patch]<br />recipe = plone.recipe.command<br />command = <br /> cd ${buildout:eggs-directory}/ZODB3-3.8.1-py2.4-linux-i686.egg/ZODB<br /> curl -s http://svn.zope.de/zope.org/relstorage/tags/1.1c1/poll-invalidation-1-zodb-3-8-0.patch | patch -N -p0<br /> cd ${buildout:directory}<br />update-command = ${relstorage-patch:command}<br /><br />[instance]<br />zope-conf-additional +=<br /> enable-product-installation False</pre> <p>The <code>relstorage-patch</code> section in the above code ensures that our ZODB3 egg is patched with the RelStorage additions, and the zope.proxy egg is needed because ZODB 3.8 requires a newer version. The <code>enable-product-installation</code> line is required because <code>zc.beforestorage</code> puts your ZODB in read-only mode (understandibly); the option tells Zope not to try and write product information to the ZODB.</p> <p>Once buildout has been run with this configuration (with the <code>-c</code> switch), you'll still need to edit the zope.conf file for your instance, usually in parts/instance/etc/zope.conf. You need to edit the &lt;zodb_db main&gt; section to wrap the storage in the beforestorage. Ours looked something like this:</p> <pre>&lt;zodb_db main&gt;<br />&nbsp;&nbsp;&nbsp; # Main database<br />&nbsp;&nbsp;&nbsp; cache-size 650000<br />%import zc.beforestorage<br />%import relstorage<br /> &lt;before&gt;<br /> before 2008-12-08T10:29:03<br />&nbsp;&nbsp;&nbsp; &lt;relstorage&gt;<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &lt;oracle&gt;<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; dsn RELSTORAGE_DSN<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; password xxxxxxxxx<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; user xxxxxxxx<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &lt;/oracle&gt;<br />&nbsp;&nbsp;&nbsp; &lt;/relstorage&gt;<br /> &lt;/before&gt;<br />&nbsp;&nbsp;&nbsp; mount-point /<br />&lt;/zodb_db&gt;</pre> <p>Any line with the word 'before' in it is new. The timestamp we learned from the undo log, simply converted to UTC. Now, when you start the instance, you are in the past. You can't alter this past (no killing of grandfathers), but you <em>can</em> read it. And lo and behold, the deleted object is back.</p> <h3>Recovery</h3> <p>Now that we have found the lost object, we can recover it. We simply exported it; in the ZMI, choose the Export/Import button, and save the export on the server. Remove the zc.beforestorage configuration (just run buildout with your regular buildout file), restart, import the .zexp file, done!</p> <p>Note that you'll need to reindex the imported content and that any related data that lives outside of the object itself is gone. For example, its intid are gone and all relationships to it will have to be recreated etc. But you just saved your customers bacon, I'm sure they won't mind a little manual work!</p> <p>--<br />Martijn Pieters</p> <p>&nbsp;</p> No publisher Martijn Pieters - Jarn 2008-12-18T09:47:43Z Blog Posting http://www.jarn.com/blog/listening-for-signals-from-dead-zopes Occasionally the impossible, or at least very unlikely, happens: your Zope instance hangs. You can see in the process listing that it is working very hard on something but it is refusing to show you what that is. <p>This phenomenon is commonly known as a <em>deadlocked Zope</em>. This is a bit of a misnomer since you might be dealing with infinite loops instead of a deadlock.</p> <p>There are two popular tools to help you debug your Zope: <a class="generated" href="http://www.zope.org/Members/nuxeo/Products/DeadlockDebugger">DeadlockDebugger</a> and <a class="generated" href="http://pypi.python.org/pypi/z3c.deadlockdebugger">z3c.deadlockdebugger</a>. Both provide a magic URL you can visit and which will return an overview of all stackframes for all threads. This will show you exactly what Zope is hiding from you.</p> <p><img class="image-right" src="resolveuid/65bd836bd3c6ced0934fd81baabe8f49/image_mini" alt="internet heartbeat" />Both these tools have an achilles heel: If all your Zope threads are stuck there is nothing available to process your magic URL. We ran into exactly that situation for a customer project and had to find a solution. Luckily this turned out to be simple: instead of requesting and returning the stackframe data through the webserver why not do it directly on the console? UNIX already provides us with a very useful signalling system available for: signals. Signals also have the benefit of being more secure: they can only be sent by someone who has access to the server and the account used to run the Zope instance. And thus was born a new product: <a class="generated" href="http://pypi.python.org/pypi/Products.signalstack">Products.signalstack</a>.</p> <p>Once you have installed signalstack in your Zope instance all your need to do is send a USR1 signal to the Zope process and it dumps the stackframes of all threads to its standard output.</p> <h3>Installing</h3> <p>Installing signalstack is simple: you only need to install the Products.signalstack package in your site. If you are using <a class="generated" href="http://pypi.python.org/pypi/zc.buildout">zc.buildout</a> just add it to the eggs-line for your instance and run buildout.&nbsp;</p> <pre>[instance]<br />recipe = plone.recipe.zope2instance<br />zope2-location = ${zope2:location}<br />eggs =<br /> Plone<br /> PIL<br /> Products.signalstack</pre> <p>If you are not using buildout you can use easy_install to install it either globally or inside your Zope instance.</p> <h3>Using</h3> First you need to figure out the pid (process id) of your Zope instance. You can find this in the <em>var/instance.pid</em> file in your buildout. If you can not find that file look for a zope process in your process listing. Once you have found the pid you can send a signal to it: <pre>$ kill -USR1 4361</pre> <p>Zope will respond in kind by throwing a lot of data at you:</p> <pre>Threads traceback dump at 2008-10-21 11:34:47<br /><br />Thread -1340051456:<br /> File "/Users/wichert/Library/Zope/Zope-2.10.6-final-py2.4/lib/python/ZServer/PubCore/ZServerPublisher.py", line 19, in __init__<br /> name, a, b=accept()<br /> File "/Users/wichert/Library/Zope/Zope-2.10.6-final-py2.4/lib/python/ZServer/PubCore/ZRendezvous.py", line 73, in accept<br /> l.acquire()<br /><br />Thread -1340583936 (GET /Plone):<br /> File "/Users/wichert/Library/Zope/Zope-2.10.6-final-py2.4/lib/python/ZServer/PubCore/ZServerPublisher.py", line 25, in __init__<br /> response=b)<br /> File "/Users/wichert/Library/Zope/Zope-2.10.6-final-py2.4/lib/python/ZPublisher/Publish.py", line 401, in publish_module<br /> environ, debug, request, response)<br /> File "/Users/wichert/Library/Zope/Zope-2.10.6-final-py2.4/lib/python/ZPublisher/Publish.py", line 202, in publish_module_standard<br /> response = publish(request, module_name, after_list, debug=debug)<br /><br />[ .. removed a lot of uninteresting frames here ..]<br /><br /> File "/Users/wichert/Library/Zope/Zope-2.10.6-final-py2.4/lib/python/Products/PageTemplates/Expressions.py", line 123, in render<br /> ob = ob()<br /> File "/Users/wichert/Development/plone/plone3.2/src/Plone/Products/CMFPlone/browser/ploneview.py", line 287, in showEditableBorder<br /> request = self.request<br /> File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/bdb.py", line 48, in trace_dispatch<br /> return self.dispatch_line(frame)<br /> File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/bdb.py", line 66, in dispatch_line<br /> self.user_line(frame)<br /> File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/pdb.py", line 135, in user_line<br /> self.interaction(frame, None)<br /> File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/pdb.py", line 158, in interaction<br /> self.cmdloop()<br /> File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/cmd.py", line 130, in cmdloop<br /> line = raw_input(self.prompt)<br /><br />Thread -1341648896:<br /> File "/Users/wichert/Library/Zope/Zope-2.10.6-final-py2.4/lib/python/ZServer/PubCore/ZServerPublisher.py", line 19, in __init__<br /> name, a, b=accept()<br /> File "/Users/wichert/Library/Zope/Zope-2.10.6-final-py2.4/lib/python/ZServer/PubCore/ZRendezvous.py", line 73, in accept<br /> l.acquire()<br /><br />Thread -1341116416:<br /> File "/Users/wichert/Library/Zope/Zope-2.10.6-final-py2.4/lib/python/ZServer/PubCore/ZServerPublisher.py", line 19, in __init__<br /> name, a, b=accept()<br /> File "/Users/wichert/Library/Zope/Zope-2.10.6-final-py2.4/lib/python/ZServer/PubCore/ZRendezvous.py", line 73, in accept<br /> l.acquire()<br /><br />End of dump</pre> <p>It is immediately obvious that this Zope has a problem: someone forgot to remove a pdb.set_trace() statement at line 287 of ploneview.py.</p> <p>&nbsp;</p> <p>--<br />Wichert Akkerman</p><p></p> No publisher Wichert Akkerman - Jarn 2008-10-22T09:54:08Z Blog Posting http://www.jarn.com/blog/catalog-query-plan Making the catalog faster <p>While the catalog tool in Zope is immensely useful, we have seen some slowdowns in large Plone sites with a combination of additional indexes and lots of content.</p> <h2>Analyzing Catalog implementation</h2> <p>The catalog implementation is using BTree set operations like union, multiunion and intersection. Those operations are fairly fast, especially when everything is in memory. However, the catalog implementation is rather naïve which leads to lots of set operations on rather big sets.</p> <h3>Example</h3> <p>A typical simple scenario is an intranet with 150000 content items, most of them published, and local roles used to handle access to most of the content.</p> <p>We want to look for pending Documents in a section of the site. In addition to the specified requirements, the catalog tool in Plone would add a search for allowedRolesAndUsers and effectiveRange to only return results the user is allowed to see.</p> <p>There is no explicit ordering of the indexes, so in our first example our ordering is <em>portal_type,&nbsp;</em><em>allowedRolesAndUsers</em>, <em>review_state</em>, <em>effectiveRange</em>, <em>path</em>. The first index, <em>portal_type</em> returns 90000 items.&nbsp; Then we continue on to <em>allowedRolesAndUsers</em>, which returns 40000 items. This is intersected with the the previous set, yielding a set of 20000. As all the content in this section is published, there are no items in the <em>review_state=pending</em>, which gives an empty set when intersected with the previous set.<em> effectiveRange</em> returns 140000 items, which is intersected with empty set. <em>path</em> returns 400 items which is once again intersected with the empty result set. The end result is an empty set and the query typically takes 0.15 seconds on my laptop.</p> <p>As a basic first improvement, it would help to immediately return after getting an empty result set from an index, but we can go even further.</p> <h2>Query plan</h2> <p>Search engines and databases uses <a href="http://en.wikipedia.org/wiki/Query_optimizer">query optimizers</a> to select <a href="http://en.wikipedia.org/wiki/Query_plan">query plans</a> that will minimize the result set as early as possible, because working with large amounts of data is time consuming.</p> <p>What we want to do is to search against the indexes giving the smallest result set first. However, for that to be useful, we need to pass that result along into the indexes to allow the indexes to limit the result set as soon as possible internally. When calculating a path search, there is no need to look in all 150000 results if the portal type index has already limited the possible result to 10000. If we have already limited the result to 10000 results, all set operations are going to be significantly faster.</p> <h2>Solution</h2> <p>We identify different searches by the list of indexes that are searched. If there are no query plans for a set of indexes, the query is run like normal while storing the number of results for each index. When all indexes have been checked, the list is sorted on number of results and stored as a query plan. Next time a search on the same indexes comes in, the query plan is looked up.</p> <p>To get different query plans for similar queries, you can provide additional bogus index names. They will be ignored by the catalog, but will become part of the key. This means that if you search for Documents in <em>draft</em> state for a worklist, you can have a different ordering than when searching for <em>published</em> Documents, as there are likely to be very few items in draft state, but many in published state.</p> <h2>Results</h2> <p><img class="image-right" src="resolveuid/725ef1a64e34a3e72f552bd09714afcd" alt="Catalog query plan" />I tested this in an example site locally with a 15GB Data.fs, 150000 content items, and running a JMeter test plan that logged in, viewed a couple of pages and logged out again.</p> <p>When measuring in the regular catalog, there were 1220 queries taking 81 seconds with the regular catalog, which is 66ms per query. When using the query plan, the queries took 14.1 seconds, which is 11ms per query. The catalog implementation is also used for membrane and reference catalog to look up users and objects, but those have significantly fewer indexes than the regular catalog tool, so we also measure on the Catalog Tool.</p> <p>When measuring only the catalog tool, there were 510 queries taking 80.9 seconds with the regular catalog, which is 159ms per query. With the query plan it took 14.3 seconds, which is 28ms per query. We consider 14.3 vs 14.1 to be within normal variations when measuring, and the numbers indicate that the time spent in membrane and reference catalog can be ignored compared to the time spent in the catalog tool.</p> <p>Going from 159ms to 28ms for typical queries is a noticeable improvement. If you are mildly abusing the catalog, this might add up to a second or more on a single page view.</p> <h2>Interested?</h2> <p>You can try these performance improvements by installing the <a href="http://pypi.python.org/pypi/experimental.catalogqueryplan">experimental.catalogqueryplan</a> package in your Plone site. Just add it to the eggs list in your buildout configuration and either import experimental.catalogqueryplan from your code, or load its zcml.</p> <h2>Sponsored by</h2> <p><a href="http://www.elkjop.no/"><img class="image-inline" src="resolveuid/4e61c35d8a124ab01cbf776feffb5ae6" alt="Elkjøp logo" /></a></p> <p>--<br /> Helge Tesdal</p> No publisher Helge Tesdal - Jarn 2008-07-29T10:15:45Z Blog Posting http://www.jarn.com/blog/plone-indexing-performance Getting rid of redundant reindexing the right way. <h2>Background and motivation</h2> <p>We have already established the fact that indexing can be improved in Plone in a <a title="Getting rid of redundant reindexing" href="resolveuid/36e1ea51b7421a9435fdf93a40d3483d">previous blog posting</a>, where we investigated the potential for improvement by applying an insane/inspired monkeypatch.</p> <p>The monkeypatch made the Plone site temporarily ignore any indexing requests, and it had side effects. Like the fact that any additional indexing triggered by content creation in factory method or initializeArchetype would be ignored.</p> <h2>Moving forward</h2> <p>The fundamental problem with indexing from a performance point of view, is that indexing is performed instantly instead of at transaction boundary. By postponing indexing until the transaction commits, we are able to filter the indexing events. This enables us to</p> <ul><li>Ignore unnecessary indexing (add followed by delete)</li><li>Only do one (re)indexing instead of many on the same object</li></ul> <h2>Technical details</h2> <p>We added an indexing queue, which was in turn controlled by a transaction manager. The transaction manager pattern was reused from <em>enfold.solr</em>.</p> <p>On modified event, or when (re)indexObject is called, the indexing requests are put in the queue. When the transaction commits, the queue is processed. There is a default reducer which filters duplicates (which can be overridden by registering a new adapter). Then the requests are dispatched to any queue processors registered. The default queue processor is for adding content in portal_catalog, and dispatches to CatalogAware and CatalogMultiplex. You can easily add queue processors for asynchronous queue processing, or for external indexing.</p> <h2>Results</h2> <p><img class="image-right" src="resolveuid/f28ad2f2114b99ff3d0fa6ad11757fd3" alt="collective.indexing performance improvement" />When measuring improvements, we created 100 news articles using the <a href="http://dev.plone.org/collective/browser/JMeterTestPlans/trunk">JMeter Test Plan</a> from the collective. Unmodified is plain Plone. Indexing refers to a site with the <a href="http://plone.org/products/collective.indexing/">collective.indexing</a> extension profile applied. Experimental refers to <a href="http://dev.plone.org/collective/browser/experimental.contentcreation/trunk">experimental.contentcreation</a> (without redundant reindex hack), and E+I is <a href="http://dev.plone.org/collective/browser/experimental.contentcreation/trunk">experimental.contentcreation</a> and <a href="http://plone.org/products/collective.indexing/">collective.indexing</a> together.</p> <p>Note that the test results are not useful as a measure of absolute performance or to be compared with other tests. We can see that the performance improves when avoiding redundant reindexing, and now it is done in a clean and efficient way.</p> <p>In a real life deployment with more indexes and metadata, and possibly file conversion as well, the improvement will be bigger.</p> <h2>Try it out for yourself</h2> <p>You can use the buildout for testing this yourself.</p> <pre>svn co http://svn.plone.org/svn/collective/collective.indexing/buildout indexing<br />cd indexing<br />python bootstrap.py<br />bin/buildout<br />bin/instance start</pre> <p>Create a new Plone site, using the <a href="http://plone.org/products/collective.indexing/">collective.indexing</a> extension profile.</p> <h2>Future plans</h2> <p>While this component is fully usable, it is also about exploring indexing in Zope in general. During the Sorrento Sprint, Malthe Borch and Sylvain Viollon were looking into Xapian integration, and found the time to start splitting <a href="http://plone.org/products/collective.indexing/">collective.indexing</a> into <em>z3c.indexing.dispatcher</em>, which will in turn dispatch to components like <em>z3c.indexing.zcatalog</em>, external indexers or asynchronous queues similar to the one in <em>ore.xapian</em>. For extracting data from content in Plone, there will be a <em>plone</em> namespace packages with adapters for data extraction.</p> <p>Because <a href="http://plone.org/products/collective.indexing/">collective.indexing</a> is about adapters and utilities and not data structures, it is simple to switch parts of it as it becomes available in <em>z3c</em> packages, and when that's done, move it into the <em>plone</em> namespace and hopefully also towards inclusion in Plone core...</p> <h2>Sponsored by</h2> <p><a href="http://www.elkjop.no/"><img class="image-inline" src="resolveuid/4e61c35d8a124ab01cbf776feffb5ae6" alt="Elkjøp logo" /></a></p> <p>--<br />Andreas Zeidler and Helge Tesdal</p> <p>&nbsp;</p> No publisher Helge Tesdal - Jarn 2008-04-01T07:33:33Z Blog Posting http://www.jarn.com/blog/explore-portlet <img class="image-right" src="resolveuid/a495a27a76fcfab7da90bd858b8b33bd/image_mini" alt="" /> <h2>Navigating the seas of Plone</h2> <p>One of the most important things you need to do when using a content management system such as Plone is to find your way to your data. Typically this involves navigating through a site using the navigation portlet: for every step your browser needs to load another page so you can find the next step on your path. This can be a slow procedure since you need a full page load at each step.</p> <h2>A faster way to travel</h2> <p>To help you get to your destination faster we wrote the <a href="http://plone.org/products/collective.portlet.explore">Explore portlet</a>. This is an alternative to the standard Plone 3 navigation portlet which features AJAX-based navigation: simply click on the triangle next to a folder in the navigation tree and its contents will be loaded into the tree. Click again and the tree collapses again. This gives you a very fast and intuitive method to explore the contents of your site.</p> <h2>Sponsored by</h2> <p><a href="http://www.elkjop.no/"><img class="image-inline" src="resolveuid/4e61c35d8a124ab01cbf776feffb5ae6" alt="Elkjøp logo" /></a></p> <p>&nbsp;</p> <p>--<br />Wichert Akkerman</p> No publisher Wichert Akkerman - Jarn 2008-03-07T10:51:15Z Blog Posting http://www.jarn.com/blog/getting-rid-of-redundant-reindexing experimental.contentcreation cuts amount of reindexing. <h3>Redundant reindexing</h3> <p>While creating a new object using the factory tool, the new object would be reindexed four times.</p> <ol><li>ObjectAddedEvent event handler</li><li>Finish construction in factory method</li><li>processForm in Archetypes.BaseObject</li><li>Rename after creation</li></ol> <p>This is 3 times more than necessary. The first two are called by <em>factory_tool.doCreate</em>, and the two last are in <em>processForm</em>.</p> <h3>Avoiding reindexing</h3> <p>All the reindexing was happening in methods called by <em>content_edit_impl</em>, and all reindexing went through <em>CatalogMultiplex</em>. In <em>experimental.contentcreation</em> we already take advantage of the fact that <em>CatalogMultiplex</em> gets a list of all catalogs to index in, and by returning an empty list, the object is not indexed. The strategy would be to let <em>content_edit_impl</em> change the result of the catalog listing while the reindexing method was called, then finally do 1 reindex afterwards.</p> <p>First of all we needed to override the <em>content_edit_impl</em> script. In Plone, it is implemented as a skin script, which enables easy customization. As we needed trusted code, we added it to the patch in <em>experimental.contentcreation</em> and set it as an attribute to the <em>Portal</em> class, which prevents the use of the script in <em>portal_skins</em>.</p> <p>Before calling <em>doCreate</em> and <em>processForm</em> from <em>content_edit_impl</em>, we create a lambda function that returns an empty list and set the <em>getCatalogsByType</em> method of the Archetypes tool object to the newly created lambda function. In addition, we set <em>_p_changed</em> to <em>False</em> on the Archetypes tool to avoid the transaction machinery trying to save the function during savepoint.</p> <p>After <em>doCreate</em> and <em>processForm</em>, we delete the function object and reset <em>_p_changed</em> again, before doing a single and final <em>reindexObject</em>.</p> <h3><img class="image-right" src="resolveuid/413da58e760aa535a016908c678d47c6" alt="Redundant reindexing" />The result</h3> <p>No performance blog posting without a graph. We tested the setup with JMeter on a laptop, 10 loops for each scenario. The blue bar is content creation with <em>experimental.contentcreation</em> with 1 reindex.</p> <p>Our tests show that time went down from 567ms in regular Plone to 203ms with <em>experimental.contentcreation</em> and the redundant reindexing removal. <em>experimental.contentcreation</em> without reindexing removal was 284ms.</p> <p>If you have converters to handle uploaded files and more fields and indexes in the catalog, the time difference will be even bigger.</p> <h3>Future</h3> <p>We also created a <em>plone.app.content</em> based content type with formlib forms to compare response times compared to regular Plone. While not directly comparable and not feature complete, the numbers give an indication of what to expect in the future, and why people are looking into Zope 3 technology.</p> <h3>Try it out for yourself</h3> <p>To try it out for yourself, you can check out <a href="http://svn.plone.org/svn/collective/experimental.contentcreation/trunk">experimental.contentcreation</a> from SVN, add it to your buildout (don't forget the ZCML slug) and restart Zope.</p> <p> -- Helge Tesdal</p> No publisher Helge Tesdal - Jarn 2008-02-17T22:11:31Z Blog Posting http://www.jarn.com/blog/membrane-performance Optimization of the object_implements method in Membrane <h3>The challenge</h3> <p>While looking at content creation in a Membrane scenario, we noticed that <em>object_implements</em> took a lot of time. <em>object_implements</em> is a catalog index wrapper that lists both the directly implemented interfaces <strong>and</strong> any interfaces that can be adapted into. This is used when looking up user objects, as you normally associate adapters with your member types to make them behave like users.</p> <p>There was a note in <em>object_implements</em> about it being slow. Timing the method call while running tests indicated that the method typically took <strong>1.2 seconds</strong> on a MBP laptop. It also did not help that objects are reindexed 4 times when creating content, but that's <a title="Getting rid of redundant reindexing" href="resolveuid/36e1ea51b7421a9435fdf93a40d3483d">another story</a>.</p> <h3>The problem</h3> <p>Previously, <em>object_implements</em> was using the <em>getRequiredAdapters</em>&nbsp; method from <em>zope.app.apidoc.component</em> to list all required adapters for an interface. <em>getRequiredAdapters</em> would check all adapters and instantiate a registration info object for each matching adapter. It would be called once for each interface implemented by the object currently being examined. Judging by the package name, this info object was originally intended to be used for API documentation, and contains more info than we need for this use case.</p> <h3>Poking the adapter registry</h3> <p>As we were not aware of any current methods in the adapter registries that would give the result we were looking for, we started <a href="http://dev.plone.org/collective/changeset/58797">poking the internals of the registry</a> from membrane. The registry has a dictionary of interfaces and their corresponding adapters, and we used that to look up the adapters. If an exception occurs in the new code, we have a fallback to the old code.</p> <p>Poking the internals of the adapter registry is not future proof, as we are circumventing interfaces and depending on the internal data structures of the adapter registry to stay the same. While not very likely to change, the data structure might change between Zope versions - requiring us to update Membrane.</p> <p><img class="image-right" src="resolveuid/d848b317be8bf81ad27bc9992ed34921" alt="object_implements optimization" /></p> <h3>Measuring improvement</h3> <p>After testing again, the time spent in <em>object_implements</em> was <strong>0.0006</strong> seconds, and the set of interfaces returned were exactly the same with both methods.</p> <p>The difference was difficult to express in our regular go-faster-graphs, so we used an area plot instead. The four pixels on the right indicate the time after optimization relative to before the optimization.</p> <p>This change is checked in on <a href="http://dev.plone.org/collective/changeset/58797">membrane trunk</a>, and will be part of the next release unless there are any unforeseen problems.</p> <p> --<br /> Helge Tesdal</p> No publisher Helge Tesdal - Jarn 2008-02-17T21:32:19Z Blog Posting http://www.jarn.com/blog/plone-2-5-performance Profiling of content creation and teamspace scenario in Plone 2.5, and publishing contentmenu and factoryhack. <p>We have blogged about performance before, after doing <a title="Improving Plone performance" href="resolveuid/a8edf29a40dc8dd8dc4cf64f6ac04e79">profiling for Plone 2.5</a> and during the <a title="Plone performance sprint" href="resolveuid/a72bdfd5ff78f1636ad53d5cb3125260">Copenhagen Performance Sprint where we made experimental.contentcreation</a>. Prior to experimental.contentcreation we made two Plone 2.5 products named factoryhack and contentmenu that are now available in the <a href="http://dev.plone.org/collective/browser/experimental.contentcreation/plone2.5">collective</a>.</p> <h3>Performance improvements</h3> <p>First of all we'll take a look at what you'll get from these products by setting up some tests with jmeter. Each test is run 10 times after warming up 2 times. Neither of the tests were done on plain Plone, they have several add on products installed, and other factors that affect performance. The results can not be used as an absolute benchmark on Plone performance, but are useful for showing relative results with and without the optimizations.</p> <p>The first test shows results for content creation. Listing is listing folder contents, Addform is showing the addform to add content in the folder, Create is pressing the save button and creating the content, and show is displaying the newly created content.</p> <img class="image-inline" src="resolveuid/ddf3744637278c126448c3dfb670d05a" alt="Plone 2.5 optimizations when creating content" /> <p>&nbsp;</p> <p>The second test is for a teamspace scenario. Teamspaces is viewing a listing of teamspaces, Teamspace is viewing one teamspace, Subfolder 1 is listing contents in a subfolder of Teamspace, and Subfolder 2 is another folder, parallell to Subfolder 1.</p> <img class="image-inline" src="resolveuid/8546508399128cff2d75bb5d5d38d6cd" alt="Plone 2.5 optimizations in a teamspace scenario" /> <h3>Install</h3> <p>If you want to try this for yourself, you can check out the products into your Products directory using command line svn or a client like TortoiseSVN:</p> <pre>svn co https://svn.plone.org/svn/collective/experimental.contentcreation/plone2.5/factoryhack factoryhack</pre> <pre>svn co https://svn.plone.org/svn/collective/experimental.contentcreation/plone2.5/contentmenu contentmenu</pre> <p>Content menu then has to be installed in the install products control panel.</p> <h3>Technical</h3> <p>The technical details are covered pretty well in the two previous blog postings [<a title="Improving Plone performance" href="resolveuid/a8edf29a40dc8dd8dc4cf64f6ac04e79">1</a>][<a title="Plone performance sprint" href="resolveuid/a72bdfd5ff78f1636ad53d5cb3125260">2</a>].</p> <p>Factoryhack is mainly about optimizing the portal factory tool, but also the type listings. It is implemented as a monkey patch, meaning that it it automatically applies itself when Zope starts, and is completely gone as soon as you remove it from the Products directory.</p> <p>Contentmenu is about improving the type listings further. It is installed in the control panel and provides a skin override. It does not alter the data in any way, and will be gone as soon as you uninstall in control panel and remove from Products directory.</p> <h3>Disclaimer</h3> <p>Note that the type listings are optimized by bypassing the regular security machinery and trying to emulate it in a more efficient way by gathering all local roles first, then check against roles required for adding content. This is working in production, but if you experience unexpected results in the add content dropdown after installing these products, you will have to uninstall them again. It does not affect actual create permissions, only the drop down menu that lists the types available. The worst that can happen is that you don't see all content types you are allowed to create, or get an error message because you are trying to add a content type without really being allowed to.<br /><br />-- Helge</p> No publisher Helge Tesdal - Jarn 2008-02-08T14:57:14Z Blog Posting http://www.jarn.com/blog/announcing-plone-enterprise-forum <p><br />We are proud and happy to announce a new interest group and mailing list: "<em>Plone in the Enterprise</em>".<br />There is a growing interest in features for using Plone in Enterprise-level deployments like:</p> <ul><li>scaling to handle hundreds or thousands of parallel authenticated users</li><li>Enterprise search (being able to quickly search through millions of objects)</li><li>Integration </li><li>Clustering and hosting solutions</li><li>High availability architcture</li><li>more…</li></ul> <p>We are now launching a separate forum for the discussion of Plone in the Enterprise.<br />There are already many interesting projects happening, and the sooner we can join our efforts the better it will be for the world of large-scale Plone. Let's share experiences, needs, war stories and help each other on development of large-scale features:<br />The mailing list is available at<br /><a href="http://lists.plone.org/mailman/listinfo/enterprise">http://lists.plone.org/mailman/listinfo/enterprise</a><br /><br />It is available in Gmane for those of us that prefer a newsgroup interface:<br /><a href="news:gmane.comp.web.zope.plone.enterprise">gmane.comp.web.zope.plone.enterprise</a><br /><br />and as a web-forum at Nabble:<br /><a href="http://www.nabble.com/Enterprise-f30548.html">http://www.nabble.com/Enterprise-f30548.html</a><br /><br />If you care about enterprise level Plone, Please join us!</p> <p>--</p> <p>Geir Bækholt</p> No publisher Geir Bækholt - Jarn 2008-01-28T22:43:04Z Blog Posting