Wednesday, February 10, 2016

What are the Technical Performance Requirements for your Startup?

By far the most popular post on this blog is 32 Questions Developers May Have Forgot to Ask a Startup Founder.  It was originally written in 2011 and has had amazing staying power.  While I’ve updated it a few times, it continues to get at important questions that startup founders need to be asking.  I find myself sending it to startup founders all the time – maybe just slightly less than Free Startup CTO Consulting.

One notable gap in the 32 Questions post is Performance.  Luckily, some of the folks at TechEmpower just posted Think about Performance Before Building a Web Application.  It does a good job of laying out different aspects of performance that should be thought about prior to creating a system.

I want to take a slightly different cut at the topic of performance.  While it’s a messy topic, I’m going to try to lay out some of the additional questions that developers should be asking a Startup Founder around the performance requirements of the application.

To get us started and to grossly oversimplify performance, conceptually we can think about the system as consisting of the following elements that I’ll refer to throughout the post.
  • Requests.  We get a set of requests for our system to do something – generally from users or external systems.
  • Compute.  Our system must access our data, possibly 3rd party services, do some calculation and then get back to the user or the other system with our response.
  • Response. The pages or API response we provide back.

Application Characteristics

Any discussion of performance starts by understanding what the software does, how it is used, and what it interacts with.  A developer should find out:
  • What are the different types of users?  What do they do?
  • Is SEO important? – this is really another type of user – a crawler
  • Are we providing an API to other systems?  What are the characteristics of how these are used?  - again, this is like a user type.
  • Are there any time-based operations?  Overnight calculations?
  • Are 3rd party services used?  What are the characteristics of how they are used?  What are their performance characteristics?
  • How many of each type of user are there?  How many might be using the application at the same time (concurrent users)?  Will there be spikes of concurrent usage?
  • What data is used in the application?  How big is the data set?  Are there complex aspects to the data?
  • What computations / algorithms are part of the application?  Are any of the calculations done often?  Are any of the calculations complex?
  • Are there any aspects that have specific performance needs?  For example, are you providing a stats service that needs frequent, fast updates?

Response Time

Once we understand the overall characteristics of the application, then we want to drill down on some specific performance characteristics.  We generally start with response time needs because, in many ways, this is ultimately the measure of performance.  If you think about our system picture above, response time is roughly the time it takes to get our page or API call back from the system. 

It’s well documented that response time has significant business impact:
The impact is quite real.  But as with most things in tech, the picture is far more complicated than that.  Consider two different types of systems:
  • eCommerce or Content web site.  These will have many individual web pages, with specific URLs, optimized for SEO.  Each page needs fast response time (both time to first byte and total load time).  Pages may not have much dynamic content on the page. There may be lots of pages.
  • Web Application such as Web Mail or a gated Social network.  The content is not used for SEO so response time characteristics may be quite different.  If the initial load time of the web application was 10 seconds but bringing up an individual email took less than 1s that’s likely an okay characteristic.  Technically, this may open the door to a single-page application (SPA).  These often often have a relatively longer time to load and then has really good performance once you are “in the application.”
Of course, response time is quite a bit more complex than this.  You will be looking at aspects like:
  • Time to first byte (TTFB) vs. Load time
  • API calls
  • Global delivery?
  • Mobile delivery possibly with slow connections?
As a startup founder, you need to think about the characteristics of your solution and what you need from a response time standpoint.

Request Volume

Assuming we know what our system needs to produce (the right side of the picture) and how fast (response time), then the next big question is really how much?  We want to find out what requests the application gets (left side of picture) and how often these come in.  This is generally turned into a Requests per Second number.

Most of the time we will start by asking about Concurrent Users – and this is generally the number that startup founders are thinking about when they talk about scalability.  Concurrent users are the number that are on your web site or web application at the same time.   Of course we need to combine number of concurrent users with what the users are doing in order to have more of a picture of what this means. 

For example, let’s assume this is a content site.  For human users, they request a page with content, likely the content page is relatively simple, the user reads/scans the page for a little bit, they decide to click something else which requests a new page.  This may take 10 seconds.  So some quick math:
  • Each user generates 0.1 requests per second
  • 1,000 concurrent users generate 100 requests per second
Those are really interesting numbers for a technical person.  Of course, this gets much more complicated.  A developer will want to drill down on:
  • Different types of users?
  • Different use cases?
  • Traffic spikes?  TV Coverage?  Real-time events?
  • API Usage?
  • Growth rates?
This will give us a clearer picture of Request Volume for different kinds of requests that our system needs to process.


Now we know the volume we need to satisfy coming in on the left and the response time required on the right.  The middle is what the system needs to do in order to respond to that volume of requests within that timeframe. 

Developers will want to explore with a startup founder where there may be complexity in the system.  We want to do this for two reasons: (1) how complex is the software we need to build – complexity generally means more time/cost to build, and (2) how long will it take for the system to calculate responses.  I’m only going to focus on the second aspect – understanding complexity as it relates to performance.  And really I’m only going to scratch the surface here as complexity is something that a startup founder and a Technical Advisor would need to explore together.
  • Computation Complexity – What do we need to compute?  What are some of the more complex aspects of system calculation?  Natural language processing?  Matching algorithms?  Complex reports?  Are there widely varied use cases with different performance characteristics?  Any blocking operations?
  • Data Complexity – What data are we dealing with? How big is the data set?  What are the largest number of a single type of entity?  Are there aspects that need to be pre-computed?  Any time series data?  Any logging/auditing data?
  • 3rd Party System Complexity – What are the characteristics of the 3rd party systems?  What will happen when they are slow or non-responsive?  What happens when they return poor quality results? 

Last Thoughts

Yikes, that turned into a lot more than I was originally thinking as I started this post.  Hopefully the core model makes sense.  As a startup founder you need to think about the characteristics of your application and then think about the Volume, Complexity and Response Time requirements.  For some applications, it will be relatively straight forward to think about the technical performance requirements for your startup.  However, in many cases, this is a place where you really should be talking with a technical advisor or reaching out to get Free Startup CTO Consulting in order to understand what you need.