Thursday, March 28, 2013

Web Framework Performance - Startup Founders Need to See These Numbers

In Web Framework Benchmarks, there are some very interesting and surprising numbers around the performance of various web frameworks.   Startup Founders really need to see these numbers.  And I hope you are not running on Cake PHP when you see these numbers.

Many of us are putting more into the front-end and having the application logic and back-end exposed through JavaScript (JSON) APIs.  In some ways, this frees us from worrying as much about the specific framework that's being used.   I've found myself looking mostly at talent and time to market.  But these numbers are causing me to pause a bit and really think about the choice of framework in terms of performance.

In looking at these numbers, seeing Cake PHP at 500x slower, Ruby-Rails and Django at 50x slower really surprised me.

I was also surprised by the performance improvement on dedicated hardware as compared to EC2 instances of roughly 10x. 

Important Implications

Well I'm currently working with startup founders on their systems in JRuby, Django, PHP and Java. 

Several of these are B2B applications with relatively smaller audiences.  I'm feeling okay about our choices of frameworks that are slower and will cost more in terms of hosting and managing growth.  The availability of talent was an important factor.

However, startup founders who are building applications that have:

  • Large Audiences - consumer facing
  • Complex Processing - examples: Matching, Social Network Analysis, Compatibility Scoring, etc.
  • Database Intensive

need to consider going with a higher performance solution.  Most startups do not get a chance to move from one framework to another.   It takes a lot of time and effort and the result is that you get to go through a whole new set of bugs only to get back to where you started but with a faster, more scalable application.  Think about twitter - but they had lots of money.

Often we justify building an MVP in whatever framework is fastest to build or where we have resources that know that framework.   You may get into the market, but just know that you are going to pay the price when you start to get traction.


Edward said...

Hi Tony,

I read the full results and I am not sure I fully understand. The results for the database access test (single query) were very different than the first graph, and the multiple queries even more so.

So which test is the most relevant?

Tony Karrer said...

I only showed the first graph, but you are right that there are several different tests being shown. Different frameworks perform better or worse on different tests, but consistently several frameworks were finding their way to the bottom.

Any application is going to be a complex blend of lots of different things. These tests are crude - but they suggest you are giving up a lot of performance going with several of the more common frameworks.

Edward said...

My developer and I read the tests and concluded the php for a social network was mroe than adequate.

We started in Drupal but are now looking at codeignitor. Do you have any experience or thoughts on that framework?

Tony Karrer said...

I've had experience with both for smaller scale projects. Depending on what you are doing, these will both reach a point where scale can be a major issue.