Jon Stahl wrote recently a blogpost about Plone being three times faster than Dropal, Joomla and Wordpress. We had a small discussion about this in my workplace and as my colleague pointed out this wasn't a really that comprehensive test that you could state Plone being 3 times faster than it's competitors. This seemed a bit unfair test considering how fast this has been spread in tweet/blogosphere, so I decided to repeat the test with a bit more critical viewpoint.
What's wrong in the original test? No one would consider opinion poll with 10 answers nowhere near trustworthy - it's all the same with requests. I didn't want to put up all the cms so I just set up second best (Wordpress 2.9.1) and compared that to my Plone development site (4.0a3). As a comparsion I did the first test with same ab command Jon used and my iMac gave following results:
Wordpress: 7.13 requests / second
Plone: 17.10 requests / second
So far we have clear winner and Jons data holds up... almost. Anyways.. to do a bit more realistic test I used 100 requests - first with no concurrency and then concurrency value set to 3 (ab -n 100 -c 3 http://...) Results are a bit different:
No concurrency, 100 requests
Wordpress: 7.18
Plone: 16.46
Concurrency 3, 100 requests
Wordpress: 13.04 requests/second
Plone: 16.50 request/second
To add some more requests I got just small differences anymore. With ab -n 200 -c 3:
Concurrency 3, 200 requests
Wordpress 13.09 req/s
Plone 16.37 req/s
Concurrency 4, 200 requests
Wordpress: 13.04 req/s
Plone 16.39 req/s
So when I just added some more requests situation was quite the same, but when we have (in the real world) that much requests in shortest time possible there are bound to be some concurrency. Adding concurrency to my test situation doesn't look that bad for Wordpress anymore.
Even though Wordpress did better job with more concurrency it still got beaten by Plone in my rough testing. Yet it's not that spectacular to be 1.2 times faster than being three times faster. And don't get me wrong - as a Plone developer and administrator I'm really thankful for all the people who have contributed to Plone tuning in one way or another. It's still good to be a bit critical before we claim that good results :)
To be clear, I'm not really familiar with Wordpress or anything which has something to do with PHP. As Jon I also put up MAMP environment and installed Wordpress there using PHP 5. There were no other customizations so I don't know how tuned it was for production use. My Plone on the other hand is my development instance and it was running in debug mode on the foreground so it was in worst possible situation for Plone to run... or well.. I could have enabled very specific logging, but that would have been overkill for this test.
What's wrong in the original test? No one would consider opinion poll with 10 answers nowhere near trustworthy - it's all the same with requests. I didn't want to put up all the cms so I just set up second best (Wordpress 2.9.1) and compared that to my Plone development site (4.0a3). As a comparsion I did the first test with same ab command Jon used and my iMac gave following results:
Wordpress: 7.13 requests / second
Plone: 17.10 requests / second
So far we have clear winner and Jons data holds up... almost. Anyways.. to do a bit more realistic test I used 100 requests - first with no concurrency and then concurrency value set to 3 (ab -n 100 -c 3 http://...) Results are a bit different:
No concurrency, 100 requests
Wordpress: 7.18
Plone: 16.46
Concurrency 3, 100 requests
Wordpress: 13.04 requests/second
Plone: 16.50 request/second
To add some more requests I got just small differences anymore. With ab -n 200 -c 3:
Concurrency 3, 200 requests
Wordpress 13.09 req/s
Plone 16.37 req/s
Concurrency 4, 200 requests
Wordpress: 13.04 req/s
Plone 16.39 req/s
So when I just added some more requests situation was quite the same, but when we have (in the real world) that much requests in shortest time possible there are bound to be some concurrency. Adding concurrency to my test situation doesn't look that bad for Wordpress anymore.
Even though Wordpress did better job with more concurrency it still got beaten by Plone in my rough testing. Yet it's not that spectacular to be 1.2 times faster than being three times faster. And don't get me wrong - as a Plone developer and administrator I'm really thankful for all the people who have contributed to Plone tuning in one way or another. It's still good to be a bit critical before we claim that good results :)
To be clear, I'm not really familiar with Wordpress or anything which has something to do with PHP. As Jon I also put up MAMP environment and installed Wordpress there using PHP 5. There were no other customizations so I don't know how tuned it was for production use. My Plone on the other hand is my development instance and it was running in debug mode on the foreground so it was in worst possible situation for Plone to run... or well.. I could have enabled very specific logging, but that would have been overkill for this test.
Jukka-
ReplyDeleteI agree, my benchmark was pretty much the crudest, simplest one possible, and certainly not representative of any real-world scenario. I'm the first one to admit that I don't have the skills to design a really rigorous one. ;-)
Thanks for trying out some other combinations!
For what it's worth, I do recommend running & testing Plone in non-debug mode. The PHP apps don't have a debug mode, so I think it's more of a fair comparison.
In my testing, Plone 4 is significantly faster when not in debug mode. I get 11 pages/sec in debug mode, and 14.9 pages/sec in normal mode.
Upping the concurrency (from ab -n 10 to ab -n 20 -c 3) only drops me from 14.9 to 14.4, not a very big hit.
Can't say that my test was actually representative either, but I just felt I needed to give PHP alternative a fair chance as concurrency is something what happens in the real world scenario :) It seems that Plone handles concurrency well but it's something which makes Wordpress to start actually doing something. I think this has something to do with the fact that Zope is already running when the requests come in. Probably displaying navigation portlet in the front page would also change the results a bit. I think results could have been much more better for Plone with Chameleon templates, but I'm happy to see it perform this good already with worst sane settings.
ReplyDeleteThis was also en excuse to check how Wordpress has developed since I've used it many years ago. UI looked really good compared what it was back then. Too bad the language is the same ugly spaghetti mess :)
So now we need to see this test repeated on a linux environment, since in real life, deployments are done on Linux not OS X.
ReplyDeleteThere's full blown Unix ticking under OS X, so I really don't believe that would matter that much in the end results. Of course some Linux distros might have more recent versions of the program stack behind this test, but that's the case in every setup depending what kind of server you have.
ReplyDeleteAnd if we're talking about the real life deployments, this test has very little to do with them, unless we're measuring how those systems handle the situations where we'd get surprising amount of requests to the pages which are not cached. If you're willing to put up such a test I'd still be interested of the results.
Jukka,
ReplyDeleteThe speed benchmarks you post with concurrency are relevant too, but most production instances of Plone run with 2-N ZEO clients, which will help this a lot. What you're testing now is a 2-thread (maybe 3-4?) Plone against a PHP app that creates/destroys the thread on every request, which does scale "wider" out of the box, if that makes sense.
Even our installer sets up ZEO with clients for you, so this is "out-of-the-box", but we don't make decisions on load balancing for you. But correct, we start getting into more advanced setups then.
Jon's point was simply to counter the (incorrect) perception out there that "Plone is slow", where in reality it's actually as fast as or faster out of the box than Joomla/Drupal/Wordpress, which are by some considered to be pretty lightweight and fast.
And I think he succeeded with that task.
(also, I suspect you're testing in debug mode, the drops Jon are talking about seem more realistic for a 3-4 concurrency setup)
ReplyDeleteLimi, you're correct - I was running my Plone site debug mode switched on as I was unsure what kind of settings does Wordpress have when it's pulled out of the box. I thought I'll make it worst case scenario for Plone and it still managed to hold the Jons data quite well.
ReplyDeleteThe different approach how web pages are served makes fair comparison difficult. After all it's the difference how threads are handled what makes putting up equal test for both systems hard. Still the fact that Jon used only 10 requests with no concurrency makes the results arguable - at least when the difference was that big, someones bound to have pea in his nose about the results. If not for the results, then because of the results and the fact that they were tested by a Plone developer. Some might say that ab doesn't take sides, but the way we run the command can do that.
Also the fact that the navigation portlet is missing from the default front page makes this a bit unfair as all Plone developers know it's one of the heaviest thing in every page load. Of course it's arguable too wether or not it should be counted as it still is the default front page, but in practice people do use navigation portlet when they put up a Plone site and same kind of portlet exists in Wordpress too (are they called portlets in Wp?). Anyways the data about that test is in Jons posts comments - Plone did better also with navigation portlet displayed so that rules one reason out.
There are and will be so many chances to argue about the results, so I wanted to rule out few. That's why I decided to repeat the tests with more request and added concurrency (that's what I'd do with my own sites too).
I'm glad that Jon posted his test and that this subject has caused this much discussion - people should know todays Plone is speedy beast!