A few days ago I read an article that compared some frameworks. Amongst others, both Zend Framework 1 and Zend Framework 2 were included. What struck me was that ZF2 was 5 times slower then ZF1. I know, it’s in beta, not yet optimized for production use. But I couldn’t believe the difference was so great. So I’ve tested it myself and want to share my results in this blog.

Setup specs

I’m running the benchmark tests in a VirtualBox machine on my Thinkpad with the following specs;

Host

  • Thinkpad T520
  • Intel(R) Core(TM) i7-2620M CPU @ 2.70GHz
  • 8GB RAM
  • 500GB 7200 HDD
  • Ubuntu 11.10

VirtualBox machine

  • 1 CPU
  • 1GB RAM
  • 3GB HDD
  • Clean Debian 6 install
  • PHP 5.3.3-7+squeeze8 with Suhosin-Patch
  • Apache/2.2.16 with KeepAlive disabled
  • APC enabled

How I benchmark

To correctly interpret the benchmark results, it’s important to know how the results have been achieved.

Zend Framework 1

I’ve downloaded the latest Zend Framework 1 from the website, this is version 1.11.11. With the ZF.sh tool I’ve created a standard skeleton project.

Zend Framework 2

For Zend Framework 2 I’ve used the Zend Skeleton Application from GitHub. This includes Zend Framework 2, to be precise; version b3d970ee42.

ApacheBench

I’ve created a simple script that uses ApacheBench 2.3 and gnuplot to create the test results.
The scripts loops through the following steps;

  1. Restart Apache on the VirtualBox machine and wait 5 seconds.
  2. Run ab on ZF1, write the results to file.
  3. Restart Apache on the VirtualBox machine and wait 5 seconds.
  4. Run ab on ZF2, write the results to file.
  5. Generate a plot of the results and wait 5 seconds.
  6. Increase the number of concurrent connections and run again.

Benchmark results

I did the following benchmarks:

  1. 2000 requests, 5 concurrent connections
  2. 2000 requests, 10 concurrent connections
  3. 2000 requests, 20 concurrent connections
  4. 2000 requests, 40 concurrent connections
  5. 2000 requests, 80 concurrent connections

I tried this with 512MB RAM in the VirtualBox machine, but this wasn’t enough for 80 concurrent connections (the OOM-killer had some fun) so I had to upgrade it to 1GB. That worked better, the swap file wasn’t used in any of the tests.

Graph results

The lower the response time, the better.

5 concurrent connections

20 concurrent connections

40 concurrent connections

80 concurrent connections

Conclusion

  • ZF2’s response times are about 4 times slower than ZF1 in all tests.
  • ZF1 is able to handle 51 requests/second, ZF2 12 requests/second.

So, can we conclude that ZF2’s performance sucks? No, definitely not.
Don’t forget it’s still in beta and I’m using a standard skeleton without any optimizations like caching and class map autoloading. And in this respect ZF2 has much more to offer than ZF1.

Before publishing I’ve discussed the results of this benchmark on freenode.net/#zftalk.2 to find out what the problem could be. Matthew Weier O’Phinney pointed out that the bulk of program execution right now is in the DI container. They know how to address this, but have lowered its priority in favor of finishing features needed.

Matthew said; “I’d also argue it’s premature to report benchmarks at this point. We’re well aware of issues currently, but plan to address them pre-stable release” And he’s completely right. This benchmark was meant purely to see if the article from piprime.fr made any sense. And it’s a starting point for me to see what you can do with the different optimizations and what their performance impact is.

There will be a follow-up to this article, hopefully with the conclusion that, with all possible optimizations correctly configured, ZF2 will kick ZF1′s ass!

What are your thoughts about this article? Please let me know in the comments!

Leave a Reply

Your email address will not be published. Required fields are marked *

34 thoughts on “Zend Framework 2 performance

  1. Great work Tim de Pater!

    But as for the “So, can we conclude that ZF2’s performance sucks? No, definitely not.” part, as a ZF fan, I can only say it SUCKS at the time being and is far too slow in their progress as compared to Symfony! Don’t forget that ZF have dedicated team on it, and it’s a pitty to see such slow progress since last year and also keeping optimization for when “everything is ready” is kinda lame!

    //Wasseem

    • “keeping optimization for when “everything is ready” is kinda lame!”

      Remember the age-old programming adage; “premature optimization is the root of all evil”.

      I personally get stuff working and then look at shaving off the ms.

    • Absolutely, one has to compare apples with apples.

      I’m currently using Zend Framework 2.1.4 on a full fledged web application, many database queries and about 12 active modules.

      I’m very impressed by the performance even during high traffic.
      What developers must remember is, the best framework is not always the fastes one, we need to also consider coding standards, conventions, team collab, code quality, longlivety etc.

      Zend Framework provides all the above and now with ZF2 we have performance as well, even if you have to implement some caching strategy, which most large platforms do anyway regardless if its ZF, PHP, Java etc.

  2. So am I reading this correctly.. on your testrig ZF2 takes on average 1,5s to generate a skeleton application with 5 concurrent users?

    And you didn’t use a 486DX2 for the benchmark? That’s pretty bad..

    I’m not sure if this is the case for ZF2, but with ZF1 I always felt that the biggest problem was that it wasn’t optimized for the ‘most likely path’. But instead a ton of stuff gets a chance to register plugins, bootstrap and what not. It’s applying the java “load & configure everything first” mentality, but that doesn’t work for an application that’s supposed to be fully setup, run and destroyed again within the course of a few hundred milliseconds tops.

    • “So am I reading this correctly.. on your testrig ZF2 takes on average 1,5s to generate a skeleton application with 5 concurrent users?”

      Well, I accidentally switched the 5 and the 20 concurrent users graphs. So for 5 concurrent users it’s not 1,5s but about 0.4s.
      But you can’t really rely on the absolute numbers, only the relative differences between ZF1 and ZF2 have value.

    • No, ZF2 is all lazy loaded. If a controller doesn’t ask for a database adapter from the service manager, a database connection is never even created. I’m seeing 30ms page load times with no caching, on a low end VPS. I threw 100 concurrent users with apache bench, couldn’t get page load times over 1/2 a second. Its very performant.

  3. Hi, very interesting post. Almost wanted to conclude it sucks, but just when I thought about it.. I read your warning, spot on :).

  4. Pingback: Aus den Blogs - Zend Framework Magazin

  5. Today I’ve some preliminary benchmarks with ZF2 skeleton application, and I have serious doubts that the final product will be faster than the existing ZF1 (that already is a hog). I noticed that the skeleton app was about 3x slower than a lightweight but db-dependant app I’ve done, so I fired up the profiler to see what was going on. Well, about 50% of execution time is being spent with DI-related stuff, and less than 15% with actual autoloading, so even if they manage to somehow cut the execution time in half, it would still be a lot slower than ZF1. Don’t forget, the ZF2 skeleton has no db connection, no locale and no complex routes to parse. But hey, I’m no expert, so I may be wrong.

  6. WOW, this is pretty weird. ZF1 out of the box is quite slow and i thought ZF2 would be a big improvement. I have tuned ZF1 alout to perform nicely (but you still have to avoid things like Zend_Date or Zend_Local if you are not caching).
    They need to at least beat ZF1, and as it looks now they have a long way to go.

    I have heard nice things about Symfony and this news makes me more curious about it even more. ZF2 will be a major rewrite, so I guess the switch to ZF2 or Symfony wouldn’t be a big difference.

    • Beta 4 provided a huge performance boost over beta 3 in our app. We have 2 modules, dozens of controllers, hundreds of actions and a few dozen helper classes.

      We’re using the standards put forth by the Akrabat tutorials, so nothing special. Fully auto-loaded.

      I think the biggest difference between b3 and b4 was the autoloading and base MVC performance.

      Beta 5 was just released. Most likely the last before a few RCs.

  7. Have you tried benchmarking any recent releases? At the time of writing 2.0.0rc7 is the latest version and likely the last before a stable release is announced. Thanks

    • I’ve done some quick tests with rc6, and while it seems an improvement when compared to older releases, it is slower than a stock ZF1 app by at least a factor of 2.
      It is curious how it is difficult to find online relevant information regarding ZF2 performance. Even when they refer performance, what you read is vague and dubious. That alone isn’t a very good sign. It may take some time until ZF2 gains some traction but – after 2 years and serveral API changes – I’m not counting on it. ZF2 is strange enough to drive ZF1 developers to other choices, like Symphony, or even to maintain a fork of ZF1.

  8. Great Blog. Thanks for the information.
    I see you have used the ZF2 in your test performance. I think zend has release the stable version now. I currently see ZF2 2.0.2. So depending on this, do you have any test performance on this.
    I am a newbie and had only tested a couple site in ZF1 but i wondering how could i use ZF2 and hows its performance with ZF 2.0.2

    thank you.

  9. Being a fan of ZF , I did some of your tests on a VM (ubuntu 12.04 , basic LAMP setup, ZF 2.0.2 ) and I’m suprised to see that I only get response times of 6000 ms / 2000 ms (there are 2 types of response times that are listed after we run AB ). ZF1 gets me around 300 ms for the 6000ms equivalent and Symfony 2 around 380ms (for the equivalent of 6000ms in ZF2). My point in the end is that I also tested the same command (ab -c 3 -n 10 http://url) with a ZF2 project hosted on zend’s phpcloud and there i got between 300-400 ms ! I wondered how come thats happening, until I saw they’re using NGINX… So i installed nginx under ubuntu 12.04, zf2 skeleton app, ran the same ab test and voila , around 300 ms. I am still wondering though whats happening that it takes so much on Php5/Apache …
    Maybe someone from the ZF2 team will cover this subject I’m really looking forward since I find the difference being HUGE (10x or more)

  10. We’re considering using ZF2 for a new project because ZF1 won’t be supported anymore in the future.. the skeleton app (using zf 2.1.0) is like 235ms to run on my PC (16GB RAM, 3.4ghz i7, 256GB SSD). That’s about 80ms slower than our beefed up zf1 app! How the crap can Zend say its better performance?

  11. I’m wondering, if there’s any benchmark which compares others frameworks.

    Is there too a way to run a batch script to test others frameworks / test my own

    + which are the 2000 requests made ? simple html / content, sql request or just php loops ?

  12. Pingback: Zend Framework 1 vs Zend Framework 2 performance | Ask Programming & Technology

  13. Pingback: ZF1 vs ZF2 … | Mohit Choudhary

  14. Hi Tim,

    Since your test was run almost 2 years ago I and probably more readers would be eager to see how the latest version of the Zend Framework improved over the last 24 month.

    Do you think you could have some spare time to test the same benchmarks again? Or do you have the VirtualBox image at hand?

    Regards,
    chriskapeller

Volg ons

Twitter

Bedankt @dara_amersfoort, voor een geslaagde eerste #CodebrewClub! http://t.co/10TUybkS4m
- Saturday Jul 26 - 10:06am

RT @mvandepolder: Vanuit de trein zie ik het #CodebrewClub-terras van @dara_amersfoort/@enrise al.
- Friday Jul 25 - 1:54pm

Hey #Codebrewer, kom op vrijdag wat eerder achter je scherm vandaan voor een vrijdagmiddagborrel in Amersfoort! http://t.co/Mbq3RkoH8l
- Tuesday Jul 22 - 9:54am

Make yourself at home, take look around. Welcome at Enrise. http://t.co/qnI3qjpgvn http://t.co/j6PEtdw2o5
- Monday Jul 21 - 11:31am

Caution: Consuming energy drink can have some strange side effects on you. http://t.co/eRhCHm7kHh http://t.co/OYERpI2Qqa
- Wednesday Jul 16 - 10:47am