Home / Blogs

How to Handle an Outage Like a Pro

Blair McKee

In just the last two weeks, there were three major DNS outages between Google, Microsoft Azure, and Fonality.

But only one of these companies was able to make even bigger waves with the way they handled their blunder. Fonality, who sells VoIP services and business phone systems, offered a very rare and transparent analysis of their outage. In a detailed statement from Chief Marketing Officer Jeff Valentine, readers were given crucial insight on how to prevent the same mistakes from happening to other companies.

The four-hour outage crippled nearly every aspect of their online business, from their website, status site, voice services, and even video chat. All thanks to a simple DNS error. After hours of investigating, the problem was rooted back to a single mistake by a network engineer who had accidentally connected a test network to the company's production network. This mistake shut down their DNS system, which in turn knocked all of their services and websites offline.

"DNS is one of those things that gets overlooked… You make your voice servers super-redundant, but you take it for granted that DNS will always work."

Lessons Learned

The sad reality is that often times things have to go horribly wrong for one person, in order for other people to learn how to prevent these issues from happening to them. Fonality's commitment to transparency will hopefully save many businesses from suffering the same fate. Here are just a few of the key lessons Fonality had to learn the hard way (but are easy fixes if you are proactive).

Lesson #1: Backup Your DNS

While there is no way to 100% be assured that your DNS will always be available, you can never have too many backup systems in place. Secondary DNS is easy to set up and requires little to no maintenance. Simply signup for services with another DNS provider (who assumable has Secondary DNS services) and configure your records to be delegated to your secondary provider in the event your primary provider goes down. This is extremely useful for clients who host their own DNS services and are reluctant to move over to a cloud-based DNS provider. This way you can keep your primary systems where you want, but just in case, you have a backup that is able to handle your traffic load.

"DNS is too important. We cannot let it go down in the future. We already had a backup, but that didn't help in this case. So we need a backup for the backup. We are putting in place an offsite system with DNS and name servers at a different location." —Valentine

Lesson #2: Keep Your Status Site Separate

It's common sense, you don't use your corporate website to host your status page. But what most people forget is you should also keep your DNS records with separate providers. Fonality learned the hard way that even though they were hosting their status page on an entirely different site, they were using a subdomain "trust.fonality.com" who's records were dependent on their DNS system.

Even if your status site is on a separate subdomain, web host, etc… it will still be unavailable if your DNS goes down. That's because DNS is the first point of contact between a user and your website, and if it is unavailable, so is everything else. This is a simple mistake even larger companies still forget, as seen with Fonality.

Lesson #3: Connect with Clients

The first that usually happens where there is an outage is your social media effectively "blows up". It is a true test of patience to be able to go through and talk to each client individually and address the complaints, especially when they are mad. However, Fonality did an A+ job at reaching out to clients via social media channels and addressing client concerns one by one during the days following the outage.

Outages tend to test client trust, however, this can be mitigated with transparency and continued support for affected clients. Some companies use SLA's, which give clients the amount of lost services (sometimes more) back to them. Others use discounts or waive fees to regain trust. However, the best way to reestablish a healthy relationship with clients is maintaining a connection. Sometimes it's hearing the voice of an executive reaching out and offering their apology. In Fonality's case, their CMO's statement was one of the best reaction's we have seen to an outage and should be a standard for all service providers.

This article was originally posted on DNS Made Easy's company blog: social.dnsmadeeasy.com

By Blair McKee, Digital Marketing Manager for DNS Made Easy. More blog posts from Blair McKee can also be read here.

Related topics: Cloud Computing, DNS, VoIP

 
   

Don't miss a thing – get the Weekly Wrap delivered to your inbox.

Comments

To post comments, please login or create an account.

Related Blogs

Related News

Explore Topics

Dig Deeper

Cybersecurity

Sponsored by Verisign

DNS Security

Sponsored by Afilias

IP Addressing

Sponsored by Avenue4 LLC

Mobile Internet

Sponsored by Afilias Mobile & Web Services

Promoted Posts

Buying or Selling IPv4 Addresses?

ACCELR/8 is a transformative IPv4 market solution developed by industry veterans Marc Lindsey and Janine Goodman that enables organizations buying or selling blocks as small as /20s to keep pace with the evolving demands of the market by applying processes that have delivered value for many of the largest market participants. more»

Industry Updates – Sponsored Posts

Global Domain Name Registrations Reach 329.3 Million, 2.3 Million Growth in Last Quarter of 2016

Neustar to be Acquired by Private Investment Group Led by Golden Gate Capital

Don't Gamble With Your DNS

Verisign Releases Q2 2016 DDoS Trends Report - Layer 7 DDoS Attacks a Growing Trend

How Savvy DDoS Attackers Are Using DNSSEC Against Us

Radix Adds Dyn as a DNS Service Provider

Dyn Partners with the Internet Systems Consortium to Host Global F-Root Nameservers

Is Your TLD Threat Mitigation Strategy up to Scratch?

Domain Management Handbook from MarkMonitor

i2Coalition to Host First Ever Smarter Internet Forum

What Holds Firms Back from Choosing Cloud-Based External DNS?

Computerworld Names Afilias' Ram Mohan a Premier 100 Technology Leader

Protect Your Privacy - Opt Out of Public DNS Data Collection

Verisign & Forrester Webinar: Defending Against Cyber Threats in Complex Hybrid-Cloud Environments

Measuring DNS Performance for the User Experience

Dyn Evolves Internet Performance Space with Launch of Internet Intelligence

Introducing Verisign Public DNS: A Free Recursive DNS Service That Respects Your Privacy

Internet Grows to 296 Million Domain Names in Q2 2015

Protect Your Network From BYOD Malware Threats With The Verisign DNS Firewall

Influential Law Firms Have Become Early Adopters of '.law' TLD