Skip to main content

Problems with Plone version pinning

Since we splitted our one huge Plone site to several smaller ones we've been looking for the correct procedure to keep things running smoothly and to make updates as painless as possible. So far we haven't found the right way and every update has had some problems. Some of the problems are related to the fact that we have about 24 sites to maintain and adding a new content type to all of them means lots of work. Below are all the steps needed for update (and solutions we've founded so far).

1. Add your new product to buildout.cfg eggs and zcml list.

Do this 24 times and you'll find yourself thinking there has to be better way. Luckily there is; you can extend your buildout.cfg with some general config file (eg. deployement.cfg). Most tutorials describe a way where you have those two files living side by side in filesystem and you run buildout with -c deployement.cfg option, but this doesn't remove the actual problem.

We have our deployement.cfg living in different server on the web, but this brings us some new challenges. Actual extending of buildout.cfg:s parts won't work, or at least I haven't found the way for that). So if your buildout.cfg includes something like this

eggs =

and you want to extend that eggs list from deployement.cfg you'd think following would work:

eggs +=

Well - for our disappointment - it won't. Instead of extending you'll need to remove eggs attribute listing from original buildout.cfg and add it to your deployement.cfg with all the eggs the original buildout.cfg had. That'll do what you want and now you can share that deployement.cfg by adding url to it in your buildout.cfg:s extends attribute.

2. Run the actual buildout command

Once again - repeat this 24 times and if you didn't swear editing all those buildout.cfg:s I mentioned in step I'm sure you'll do that by now. We didn't find any ready solution for this so we made our own called jyu.bomanager. Our tool has buildout.cfg where we can configure all the sites we have. By convention we've named our buildout directorys with the same name our sites are. What jyu.bomanager does is that it runs buildouts automatically for all our sites and restarts them. All seems to be good here, but wait - there are problems too. I'll come to that later.

3. Update/install the new addon.

Now when we have our 24 sites up and running we just need to update or install our new product. I guess you already see what's the problem so I won't repeat myself anymore. To solve this problem we developed a tool similar with jyu.bomanager called jyu.sitemanager. What jyu.sitemanager does is that - depending our mission - it can log in to all of our sites as admin and either install specified new product(s) or upgrade those which needs updating. If everything has gone ok this far we don't have any problems with this step either.. unless we have failed testing our new addon and there are bugs in it, but that'll never happen to us :)

Weakest link... step 2. So far running buildout is usually the part where things fall apart - even when on testing server all went well few minutes ago. It won't help much if we've automated all the steps so that you'd need only few commands to watch the process just happen if buildout fails. There are many points where we can fail - could be down, PyPI could be down or there could be some nasty surprise where we have some unpinned version updated - maybe brought to our buildout by some addon requirements - or just forgotten to pin.

Today we had scheduled 2 hours timeframe to update all our sites and we hoped that this time everything would go smoothly. Maybe next time... it seems that even though we've mirrored to our own server and pinned versions we still had some loose version out there. Plone 3.2.2 pins zc.buildout to version 1.1.1, but running buildout got us new version of plone.recipe.zope2zeoserver which expected to find 'relative_paths' attribute from zc.buildout. That attribute has appeared on zc.buildout since 1.2 versions, so you see the problem. We modified versions.cfg so that we'll get zc.buildout 1.4.1 version and things went smoothly. Another problem came from LDAPUserFolder. We have python-ldap 2.3.8 installed in our production environment, but LDAPUserFolder had version requirement for anything bigger that 2.3.0 (I'm not sure about exact version number and I'm too lazy to find out) so our buildout happily fetched 2.3.10 which needs libraries our production server doesn't have. Crash. Another version pinning and we were good to go.

Although buildout is blessing sometimes it feels like wild wild west. It isn't funny when we're supposed to update our site and our buildout hangs when it's supposed to fetch data from pypi or Now we're thinking setting up eggproxy and/or own pypi mirrors so maybe next time we have smooth experience.

Popular posts from this blog

Plone 4, ZEO and supervisor

This post belongs also to the "lessons learned" category.

With Plone 3, ZEO and supervisor combination you've probably configured your supervisor to start plone instances by running $BUILDOUT/parts/client1/bin/runzope.

Problem is that with Plone 4 your $BUILDOUT/parts/client folder doesn't contain anything else than etc folder. You know starting instances by targeting supervisor to use $BUILDOUT/bin/client1 fg doesn't work like you'd expect (supervisor would control the client1 script - not the actual plone process).

My colleague Jussi Talaskivi figured that using 'console' argument instead of 'fg' for bin/client1 script should do the trick. With 'console' argument stopping, starting and restarting Plone 4 instances with supervisor works like a charm.

Below is full example of working supervisor configuration.

[buildout] parts = supervisor [supervisor] recipe = collective.recipe.supervisor port = 8200 user = xxxx password = xxxx pr…

Plone RPM deployment

It's been ages since I wrote my last post. I guess now it's time to make that up with post about Plone deployment. In a last few months I've had chance to invest part of my time at work to develop our Plone deployment model and since there hasn't been that many posts about the subject I thought I'd share my experiences for the community.

Before I get to the details here's some background information about the environment I've been dealing with. I'm working at the University of Jyväskylä, Finland. Our university is a heavy weight Plone adopter here in Finland. Plone is in use by every faculty and department. We have about 500 - 700 content managers and about 50 - 60 separate Plone instances. Our front page gets about 2 - 3 million hits page views / month. The important information in previous data is the amount of instances. 50 - 60 is just my estimation about our current status and that amount is increasing every year. I'm sure you got the idea so I&…

Speed comparsions between Plone and Wordpress

Jon Stahl wrote recently a blogpost about Plone being three times faster than Dropal, Joomla and Wordpress. We had a small discussion about this in my workplace and as my colleague pointed out this wasn't a really that comprehensive test that you could state Plone being 3 times faster than it's competitors. This seemed a bit unfair test considering how fast this has been spread in tweet/blogosphere, so I decided to repeat the test with a bit more critical viewpoint.

What's wrong in the original test? No one would consider opinion poll with 10 answers nowhere near trustworthy - it's all the same with requests. I didn't want to put up all the cms so I just set up second best (Wordpress 2.9.1) and compared that to my Plone development site (4.0a3). As a comparsion I did the first test with same ab command Jon used and my iMac gave following results:

Wordpress: 7.13 requests / second
Plone: 17.10 requests / second

So far we have clear winner and Jons data holds up... almos…