Home / Blogs

The Ultimate Differentiator: Reliability!

Every company that monitors their site or application performance focuses on two key metrics Availability and Speed. However, there is a third metric, Reliability, which is often misunderstood or in some cases ignored by companies. Reliability measures availability, accuracy, and delivery of a service within a time threshold.

Reliability is difficult to define and measure as it is different for each company and service. To simplify it, you can think of Reliability as how consistent are you in delivering the "service". The question it is trying to answer is: Can the "service" deliver the same experience, every time, all the time to all users? The service, could be a site, an application, a process, or a series of processes.

For example for Amazon, reliability might answer the question; can a user browse the site, add an item to the cart, checkout and receive the right item on time? For an ad-serving company like DoubleClick (Google), reliability might mean: can the adserver serve the right ad all the time for the right user on the right website, within 200ms.

Reliability is key to every business, but is even more important to vendors that sell services impacting the performance of other products. The service of a CDN, or Adserver, or Widget company can have a huge impact on the performance of a publisher or ecommerce site. Reliability is a key differentiator for such vendors.

To illustrate the Reliability concept, let's compare DoubleClick's Ad-serving system with an unnamed ad-serving company. We are defining reliability as how consistent is the adserver in delivering an ad to an end user. (Because we do not have access to the two products and cannot control the testing, we cannot go more in depth and measure certain things like getting the right ad on the right site.)

Let's start by looking at the overall performance (Response measured in ms):

DoubleClick:

ABC Company:

DoubleClick's Response is consistently below 190 ms as it can be seen by the flat line. The response of the other adserver is fluctuating a lot (between 250 and 850ms). In other words, they are not consistent in delivering their ads.

Using Catchpoint, we were able to easily rule out DNS and Network connectivity issues (they were flat for both) and identified the culprit, the Wait time. (The time from the request was sent to the server to the first byte of response — in other words how long it took for the server to process and respond).

DoubleClick Wait and Response time:

ABC Wait and Response Time:

The other company's adserver response time fluctuates during critical business hours due to the wait time.

DoubleClick's Performance by Hour:

ABC's Performance by Hour:

ABC vs DoubleClick:

Which leads me to believe their servers are overloaded during business hours and cannot handle the requests based on how their application performs. It seems like the company need to take a look at their capacity planning and also optimize their code.

Another way to look at consistency is by using Standard Deviations, as we know by now averages can lie!

Standard Deviation provides a simple way to define consistency: a large standard deviation value indicates a high degree of inconsistency within the measurement population, whereas a low small standard deviation value indicates a higher degree of consistency.

To summarize, Reliability is about how consistent your service is performing. It looks at Speed, Availability, Integrity (delivering the right ad) over time to ensure service is within acceptable thresholds. Reliable services will have flat response lines, or low volatility.

By Mehdi Daoudi, CEO at Catchpoint Systems Inc
Follow CircleID on
Related topics: Cloud Computing, Data Center, Web
SHARE THIS POST

If you are pressed for time ...

... this is for you. More and more professionals are choosing to publish critical posts on CircleID from all corners of the Internet industry. If you find it hard to keep up daily, consider subscribing to our weekly digest. We will provide you a convenient summary report once a week sent directly to your inbox. It's a quick and easy read.

I make a point of reading CircleID. There is no getting around the utility of knowing what thoughtful people are thinking and saying about our industry.

Vinton Cerf, Co-designer of the TCP/IP Protocols & the Architecture of the Internet

Share your comments

Great Post David A. Ulevitch  –  Dec 02, 2010 7:33 AM PDT

This is great.  I think most people underestimate the importance of reliability and consistency.

To post comments, please login or create an account.

Related

Topics

Domain Names

Sponsored byVerisign

DNS Security

Sponsored byAfilias

New TLDs

Sponsored byAfilias

Cybersecurity

Sponsored byVerisign

IP Addressing

Sponsored byAvenue4 LLC