Webinar: Your Business Continuity Plan Could Be an Epic ...docs.media.bitpipe.com/io_13x/io_133845/item... · Your Business Continuity Plan Could Be an Epic Failure, Webinar, October

_____________________________________________________________________________________ Your Business Continuity Plan Could Be an Epic Failure, Webinar, October 4, 2016 ©2016 Charter Communications. All Rights Reserved. Time Warner Cable Business Class and the TWCBC logo are trademarks of Time Warner Inc. Used under license. All other trademarks remain property of their respective owners.

Webinar: Your Business Continuity Plan Could Be an Epic Failure

October 4, 2016

Moderator:

Steve Shepard, Author, Educator, Speaker

Shepard Communications Group

Featured Speaker:

Andre Kindness, Principal Analyst, Serving Infrastructure and Operations Professionals

Forrester

Panel Discussion:

Chris Hinch, Senior Network Engineer

Gregory Technologies & NaviSite Client

Dan Toomey, Director, Sales Engineering

Time Warner Cable Business Class

Introduction

Steve Shepard

Hello, everyone. Welcome to our webcast today. I’m your host, Steve Shepard. While the title of the program is

“Your Business Continuity Plan Could Be an Epic Failure,” we’re going to do everything we can today to

ensure that it isn’t. You’re going to hear from three authorities on business continuity today. We’re going to

share their knowledge and experience with you. They are Andre Kindness from Forrester, Chris Hinch from

Gregory Technologies, and Dan Toomey with Time Warner Cable Business Class. I’d tell you a little bit more

about the panelists later in the webcast when we hear from them.


2

Now, before we get started, here’s what we’d like to do. We’d do a little bit of level setting with a couple of

polling questions. So here is our first polling question if we could. What we’d like you to do, please, is to select

three network outage types that you believe your company is least prepared to recover from. Yes, these are

anonymous. So if you would just select three. What we’re going to do is we’re going to go through the

questions and then we’re going to review the answers a little bit later on in the program. So take a minute or

so. Select your three and then, in just a little bit, we’ll go on to our second question. Hopefully you don’t have a

hard time selecting from these six options.

Very good. Let’s move on to polling question number two if we could. Select the three most important features

or capabilities that you’d like to add or expand to your business continuity or disaster recovery plan. Select the

three most important capabilities you’d like to add. By the way, while you’re doing that, let me just note that we

will sometimes refer to business continuity and disaster recovery as BCDR. So I’ll get that acronym out there

right away. Go ahead and take a minute. Select your three most important features, then we’ll get started.

Business continuity and disaster recovery: How prepared are you?

Let’s get going here. Now we’ve collected all this information. We’ll come back to the responses a little bit later

on. What I’d like to do is just a little bit of introductory comment, if you will, before we actually jump in to our

presenters. It’s no secret to anyone out there in the audience that business continuity and disaster recovery

are really critical success elements in the increasingly IT-dependent enterprise. Like so many things that form

that operational foundation of business, they often are ignored or at least denied the attention that they

deserve. Like the network they tend to be invincible—until they’re not.

So to make this point I’d like you to travel with me back in time to 2014, which was a spectacularly bad year for

data centers and, more importantly, for the people who depend on them. Just as an example, in February of

that year the data center for the State of Iowa caught fire and it put them out of service for 16 hours. In May of

that same year, a power outage in New York City brought down 2 internet data centers, which caused severe

impact for more than 20 companies. In September a badly executed infrastructure change at Facebook caused

a series of major outages for the application.

Those are important and obviously painful, but here’s the real issue. It’s a lot more than just the interrupted

service or the inconvenienced customers. On average, in today’s IT environment, a data center failure costs

customers more than $8,000 a minute. The average recovery time is 119 minutes. That’s two hours. If you do


3

the math, that’s just shy of a million dollars, and that’s a relatively brief two-hour outage. As you’re going to

hear today, there are a number of factors that cause these outages. The bad news is that hardware does

occasionally fail and software does go wonky and power does fluctuate, and cyber criminals aren’t going

anywhere. The good news: there are ways to reduce the impact. So let’s get started and have a look at some

of these.

Our first guest today is Andre Kindness. In his work as a principal analyst with Forrester, Andre serves

infrastructure and operations professionals. He lends his expertise to organizations that have network

operations and large architectures. He helps them look at their attempts to develop a more comprehensive

technology strategy for their digital businesses. So this should be a very, very interesting presentation. Andre,

to you my friend.

Glitches can cost millions

Andre Kindness

Thank you very much, Steve. I want to thank everyone on the call today, taking time out of their busy

schedules to tune in this webinar put on by the Business Class team from Time Warner Communications. As

Steve was pointing out with respect of outages, I’d say we’re in another inflection point within networking. In

much ways as Steve has brought up, during the first one the expectation of what the infrastructure needed to

do was really reset by the mobile mindset, cloud services, and a data center virtualization. This isn’t really

anything new to you and you’ve heard this on many different presentations—steak and potatoes.

I would say those technologies, and technology itself, doesn’t really reflect the digitalization that’s occurring

within a business. As a new inflection point that I want to highlight with you in this next slide, if you’re looking at

the screen in front of you and getting distracted with Twitter or Facebook at work, you might be realizing that,

“Hey, I don’t think these companies are that digitalized.” Probably because you’ve had some bad experiences

on the aircraft and then sitting on the tarmac like myself for hours on end. But they really are. These

companies have the ability to track luggage in foreign airports and now being able to update your mobile app if

your luggage had made it to the flight that you’re on and what carousel it’s going to. A lot of things that they’re

doing is really changing the customer experience for their customers themselves going from airport to airport.

Like notifications on when the gate has changed even before they see them on or you will see it on the

overhead at the airport themselves.


4

As we brought up a little bit ago, as Steve brought up about data centers going down, I would say it’s even

worse than the numbers he had brought up because a lot of that is just about that moment in time. This new

inflection point that I want to bring up in itself is a slight hiccup that could be disastrous to the business itself.

It’s not just about the fact that someone can’t get email or get on a shared drive, but these glitches will cause

millions of dollars of damage if not tens of millions.

Look at Delta and what happened and the fact that there was a failed electrical switch. Not even failed

networking equipment, not a data center going down. Just the amount of sales in that moment in time was $8.2

million. Here’s the thing: it took them almost two to three days to catch up on everything they need to do. So

we have to think about the cost of overtime, the cost of hotels, the cost of trying to re-route people in other

flights and payoff for overbooking and situations like that, that actually was closer to $30 million.

Even JetBlue, they had a power issue and they had 200 flights canceled. That’s a huge amount, and nowhere

near what Southwest had to deal with, and that was over 1,000 flights and that’s because of a router going

down. Does anybody want to guess how much that cost them over an eight-month period trying to catch up on

everything? That router went down for an hour. It took them four days to recover. How much do you think they

spent? I could tell you it was close to $82 million that they’re claiming on their financials when they reported

that in—$82 million for a two-hour outage. That’s humongous. I can draw about a ton of other scenarios. There

was something at Google and what happened in that environment, but I really want to highlight with you the

fact that in this new world, a glitch will cause companies millions and millions of dollars.

Winning and retaining customers depends on digital transformation

As Steve pointed out, I’m Andre Kindness, Principal Analyst, Infrastructure and Operations. I spend a lot of

time working with companies like yourselves combing through enterprise data, respondents, and information to

really help companies implement best practices. That being said, I want to spend the next 15 minutes with you

sharing with you a survey report that Forrester put together for TWC around backup and disaster recovery. We

looked at how common are network outages for businesses today, what are the causes and operational

impacts from that. Some of them are highlighted already. If we did network cloud and resource redundancy

and diversity, would that mitigate a lot of down time and the cost that these companies are incurring with this?


5

Let’s get started. I’ll be honest. Normally we’d like to start off an IT presentation from a technology standpoint

or growth numbers or things in that perspective, but I want to do something a little different. I want to start off

from the customer side. Why? That’s where the conversations are at, it’s where the money is coming from a lot

of your IT projects themselves, and in many ways we’ll help you help them make the right choices on where

things are going. If you’re looking at the screen in front of you, this is a different perspective in the airline

industry. What we’re seeing in digitalization of changing what customers are expecting from the business itself

is starting to occur not in front of the screen itself or on a mobile device but what’s happening in these facilities.

You look on the left-hand side for retail store. You look on the right-hand side for hotel. These companies are

incorporating a lot of technology to change what’s occurring in these facilities, to change that customer

experience, and to win and serve and retain customers in a completely different way.

Let me give you an example of what’s happening in the retail industry. You hear stories about Burberry, Tesco,

Carrefour trying to digitalize. The whole industry is in a battle of existence of online retailers. Nordstrom is

spending millions of dollars creating a mobile app to differentiate themselves, to create deeper levels of

intimacy. So combine with that what’s happening in a store using RFID, shelf weight, heat sensors, light

sensors, LEDs, digital screens, you name it. These companies in the retail industry are transforming that

environment inside their store, customizing that feel when someone walks in there. So what we think about is,

is the diverse set of technology really to change that experience, to change that business model within these

facilities themselves? It’s not even happening within just a retail store or a hotel, but this is occurring on oil

platforms, inside buses themselves or trains. There’s a huge amount of technology that we’re trying to leverage

to really change what’s occurring in there.

A diverse business environment is key

This diversity, the technology itself, is not just the fact of diversity of technology, but what I want you to

highlight and pick up on this is diversity. This is why I want to start off on the business side or the customer

side, because a lot of the infrastructure and operations should really align to the business strategy in a

direction that the business is taking things and what they’re doing from a strategic standpoint. Diversity is a key

element of that. If you’re looking at the screen and we’re looking at a customer within a store, you can almost

also equate this customer within a hospital.

Many of these businesses are not doing this alone, but they’re doing it from a diversity of technology and a

diversity of business partners out there. “Companies can’t do it themselves,” as I’ve mentioned. These


6

companies are turning to other ones to serve the customer requirements to multiple companies and services.

At hospitals you have doctors and they’re not technically part of the staff. You have the nurses. You have the

staff. You have outside companies like GE, insurance companies, pharma, contractors, all focusing on the

customer and trying to enhance the customer experience and deliver what the customers want. Hospitals are

fighting for patients. They’re calling them customers but they’re fighting over them to make sure they come to

their facilities, and they’re doing this with multiple types of resources, as I mentioned.

In the retail industry, you take a look at a company like Best Buy sharing your space with other technology

companies and service vendors out there, even the fact that they have Geek Squad in there. That physical

environment is being shared by multiple partners to really create a different feeling when you go into that store

and to be able to cater and deliver the products and services the customer walking into that store wants to get.

That diversity is really about business diversity, the common business element we see today.

So a diverse set of vendors, partners, and resources is occurring across there. It does not even happen in the

business, as we start to see—we move away from the business aspect and we look at the first team. And the

first technology team working with the business itself is application developers. To serve this business

environment the application developers need to deliver a faster way of doing things, which is giving birth to

new style development, stateless, loosely coupled services over a diverse set of platforms all knitted together

to deliver applications much more quickly. Let me rephrase that again: a diverse set of applications and

services and infrastructure to deliver that.

Is your infrastructure prepared?

So if you’re looking at this question: “Is this infrastructure prepared to do this? Is your team prepared to do

this?” We’d probably say no. Why? If you’re looking at the response to a question that we have sent out, “Has

your company experienced a technology outage on the last five years?” A 100 percent of them said yes. A 100

percent said yes. Maybe a few of you are saying, “Well, it might not be a big deal, the outage.” Let that sink in

and think about Southwest, United, Delta, or JetBlue. Was that not significant to them? Bringing the point home

is, if it’s not a big deal on the business side, 59 percent of companies struggle with reducing employee

productivity and operations due to the outage. Almost 70 percent said that their employees were affected by

this. That’s an incredible amount. Think about that. We’ve had technology being around for a long time and the

fact that customers will be ultimately affected by employees not being able to do their job. Forty percent, or


7

close to 40 percent, said they experienced lower customer service and product, and 30 percent were affected

by lower customer experience itself.

In today’s world, we’re making choices in nanoseconds by the speed of which a page gets loaded on our

phones or the fact is when we walk around and make choices about, “Hey, we’re not getting the service we

want,” and just a click away we can find another store having that product. The fact is, 30 percent is not an

okay number to have a bad experience within that facility itself. This should be a red alarm for anybody working

for a business. I don’t care if you’re a retail store, a hospital dealing with customers directly. Even in the

business-to-business world, having a 30 percent issue with your manufacturing lines, being able to deliver

products, getting them into the stores, that is a significant amount and really not acceptable in a world that we

have an expectation of getting what we want where we’re at, wherever we’re at, whenever we want it, that

mobile mindset. Harking back on this, 30 percent is transformational for companies to start making changes,

and it upsets a lot of business folks themselves.

How to prepare for an outage

The thing is, if you’re looking at the data on the screen in front of you, it’s not that there’s some complexity or

there’s some advance or technology behind that. A lot of these issues are, the fact is what power outages

within this area are causing within these environments. Seventy-three percent of these companies have had

power outages. Another 65 percent are claiming system hardware. Another 42 percent are claiming that failed

software caused these outages. In that, 51 percent of them weren’t even prepared for a failed software or a

firmware update or dealing with patches within these environments.

If you didn’t know, a month ago, Google actually had that same issue with updating two routers when one of

their facilities in the Midwest went down for a while and they went to 50 percent performance. That was their

EC2 platform, where they have app developers building things out there or putting things out there just like

Amazon and other areas. So you need to think about that. A lot of companies are putting their web services or

putting their business applications out there, and to have something like that go down in this world, totally

unacceptable when we think about the bigger scheme of what’s occurring.

So now that we think, is there a way to do things better? Can we have the ability to change the way of what’s

happening with this environment itself? We start to look at some of actually what’s happening to be successful

and to change it. You start to see companies out there maybe taking a different approach. Seventy-six percent


8

of IT pros believe some of the factors that we need to do is provide employees other ways to connect in and

promote access, as they start to think about geography and diversity around spreading out your data centers

and spreading out your applications and storage and backup centers, and not the fact of making multiple

redundant ones but leveraging cloud resources, having that diversity going in, and the fact is where your

resources are at. Also the fact that geography-wise, because we think about Sandy. We think about hurricanes

coming in the Southeast, the West Coast having earthquakes themselves. You don’t want everything in one

location.

Even separate entrance facilities: 70 percent believe it’s important to use BCDR solutions that give them a

separate entrance facility to guarantee network diversity in an event of network disruptions. Even 68 percent of

IT pros believe that having managed solutions will constantly monitor the network and help the IT team out. It

will help them in this way as well.

What are your peers doing?

So we think about this overall thing that your peers are really looking at: “How do we take the concept of

business diversity, technology diversity, and apply that to the network and apply that to the clouds

themselves?” We start to see in this world itself that already, within retail stores and within manufacturing sites

and hospitals, within those areas we see that applications, data, employees, and customers are connected

together over a diverse set of resources over various networks.

You’re like, “What do you mean by that, Andre?” In these facilities we’re using WiFi, wired, Bluetooth, RFID,

and NFC to connect all these things together. That’s why I want you to think about the bigger scheme of your

overall business if you’re looking at this. The key to creating a digital business, to maximize use of available

resources, internal and external—this is why you’re starting to see tech management professionals like

yourself build out their disaster recovery plans and back up plans over a diverse set of resources. Improving

infrastructure resiliency and building out network diversity and redundancy, connecting to different cloud

resources out there, using different platforms, MPLS, internet as a combination of active/active scenarios

always maintaining that on state for the business itself.

So the technology community has seen similar trends when it comes to cloud. Twenty-one percent have

already adopted the cloud capabilities for their BCDR. Let my sync that in. Already 30 percent have their

backup and disaster recovery, 46 percent of IT pros are currently adding or enhancing to these capabilities


9

themselves. Thirty percent have them and another 46 percent are adding on to that. The hybrid cloud, cloud

diversity as a platform, these are enabling their businesses to always remain on and to mitigate some of the

aspects of having some point solutions, some point aspects, some point parts of the infrastructure having an

outage themselves, being able to spread your risk out across multiple platforms.

The benefits of diversity

So you think about this. What is this actually delivering for the business? In doing so, 70 percent of IT

professionals believe they will keep the business running. If you’re looking at the screen, 58 percent increased

network ability and performance, followed by the enhanced customer experience. Customer experience, as I

rotate back to that concept, is you’re delivering to the customer who always expects to get what they want,

wherever they’re at, whenever they want to have that product or service, so that experience itself. That in itself

should be a motivator to change what you want to do. In this age of the customer, business success requires

the ability to leverage digital assets both inside and outside the business, the boundaries in the organization,

for the benefit of the customers. This idea is driving new strategy and requirements around BCDR.

A new direction

Let me rephrase that again. This is the new direction. Diversity is the new direction and fundamentally what

needs to occur within our infrastructure today. We can’t rely on point aspects, single links. We can’t rely on

single platforms anymore. Things have to be spread out so we can minimize the risk. This is why in many

ways, if you look at the applications being built, it’s about micro services. The idea behind that is, in the cloud,

you’re not responsible for the server going down, and so you want to break X cables down into micro services

because of different hardware components. To offer changes with what’s going on, you need the application to

be resilient. So the idea behind micro services is to spread things out and have redundancy across different

infrastructure and platforms to allow that application to exist, to serve the business and ultimately to serve the

customer.

That old method of duplicating links, for example on the network side, but having one on active state and one

on the backup state is dead. You can’t have that in the world because the time of switching one to another is

much too long. You need to really have them in active/active state. Network diversity is always using one type

of it having MPLS, but the reality is, what you want to do is have maybe MPLS use cell and use satellite,

multiple things, and using them in active/active/active state across all of them and continuously choosing the


10

best one based on cost, based on performance, and what it’s delivering to the business itself. So we think

about network diversity, cloud diversity, operational diversity making up the core elements of where this new

digital business is going. It really is the backbone to the data we’ve collected, and a lot of organizations are

leading this charge on their BCDR. That infrastructure and operations are embraced across multiple industries

out there.

Hopefully, I’ve covered quite a bit in the last 20 minutes. The fact is, how important is it for this backup and

disaster recovery that diversity is the key element across it? What I’m going to do is hand this back to Steve

because I want to see what you, as the attendees, chose on the earlier polls on some of the questions that you

had. Steve, let me bring this back to you.

Luck favors the well prepared

Steve Shepard:

Thank you, sir. Andre, it’s funny. As I was listening to you, I remembered an experience that I hadn’t thought of

in many years. Back in the 80’s, when I worked for a phone company out in California, we had a major disaster

that occurred. The disaster was that the USS Enterprise, the aircraft carrier, that used the home port in the San

Francisco Bay, came home. Came under the Golden Gate, entered the bay and dropped anchor to stop. In the

process of dropping anchor, accidentally clipped the carrier cable that ran across the bottom of the bay. This is

when we learned very quickly and very painfully about the concept of route diversity. We had two repair

centers: one in southern California and one in northern California. They were on separate carrier systems for

obvious reasons. These are the primary places you call when your phone doesn’t work. That was when we

learned that, while they were diversely routed, the carriers were actually inside the same cable at least for that

part of the transport. So as a consequence we knocked both of the centers offline, and obviously it was a very,

very expensive and very long and nasty recovery.

What I found really interesting about your presentation was that one slide that says, “100 percent of the

respondents had an IT outage that resulted in reduced productivity for employees as well as business

operations.” What that really says is, there is no such thing as a business or an industry that is immune from

this. This is going to happen. So the real message here is that we often say, “Luck favors the well prepared.”

So we want to make sure that we’re actually doing that, and in a few minutes our panelists will speak to that

just a little bit. With that, thank you, sir.


11

The most vulnerable recovery-failure areas of your current BCDR plan

Let’s go to our poll results and see what folks said. Andre, I want your comments on this.

Andre Kindness:

Yes. No surprise there on power outage. Well, number three are natural disasters, right? These things have

been basic for a long time that we still struggle with. It almost makes you want to stop and think about what you

should be doing in these, power outage or natural disasters, these are things that are common, we’ve been

dealing with technology for the last hundred years when we’ve had electricity. We really need to think about the

way we’ve done things in the past working out. We need to really think about doing things in a different way.

Look at the top three ones, cyber security or just security attacks are nothing. It has been common. Natural

disaster is common. Power outage is a common thing. It’s not the cutting-edge types of things that are causing

the most angst for these companies. Its things that we’re most comfortable with, maybe to the point as we

probably should start to approach this in a different way.

Steve Shepard:

Yes. Absolutely. It’s also interesting how they break down in terms of external versus internal, right? I mean the

power outage, cyber security attack, and natural disaster—not much we can do about that.

Andre Kindness:

No.

Steve Shepard:

Right? Being more careful with upgrade failures, making sure we have adequate networking in place, adequate

support in place and so on, that stuff that we can in fact do something about. So it’s really interesting that some

of these issues are things we can actually do things proactively to prevent. Here are some that we simply have

to respond to and be prepared for when they do in fact happen. Very, very interesting.

Andre Kindness:

This data—and I forgot to mention, I even noticed that it doesn’t add up to 100 percent. So people aren’t just

having one either. I think that my data shows that as well. People aren’t just having one issue. They’re having

multiple issues, which is even more concerning across it. We can’t just prepare for one, but we really have to

prepare for a multitude of things because people are experiencing things across multiple areas.


12

Steve Shepard:

Absolutely. Any self-respecting IT person who believe their network is 100 percent foolproof has a fool for a

client, right? [Laughter] That’s serious. That’s a serious issue.

Three most important features to add to your BCDR plan

Let’s look at question number two and see what we found here. To you, sir. Very interesting.

Andre Kindness:

I’m just reading through what the—yes, very close to the data that we have presented on what other than

maybe the top three, just maybe in a different order. The fact is to see remote access, spreading geographic

diversity, cloud services, very similar to the data that we had collected as well; and if you take a look at it, the

geographic diversity, getting around the natural disasters, how to deal with that; the fact is about electricity, the

same idea, how do you spread across different platforms, use different cloud services out there, connecting to

them. And then network diversity, the fact is offering employees remote access again. So in many ways, those

things that maybe we can’t deal with, we don’t own specifically like natural disasters, I know professionals out

there are looking and being smart about the ways they’re starting an approach, and looking for ways to get

around it to make sure that they have business continuity in this new digital world.

Steve Shepard:

Yes, absolutely. Frankly, some of these things, looking at them and thinking about the current state of the IT

world, some of these would be considered to be table stakes, right? The fact that we’ve got a number of

companies out there that don’t have these yet bares evidence by the fact that people are saying, “I’d like to

have this”; speaks a lot to the importance of perhaps modifying the BCDR plan along the way. So very, very

interesting.

Andre Kindness:

Well, yes. You remember 51 percent were not prepared for firmware or software updates on their

infrastructure. I mean, to your point, that’s been around as long as technology ECs have been around inside

the business. So the fact is, 51 percent should be disturbing to everybody on the phone.


13

The ever-more important role of the network in BCDR

Steve Shepard:

Absolutely. Well, very good, sir. Thank you. Okay, stick around, Andre. We’re going to bring our panelists in

now. So let me introduce these folks once again, Chris Hinch and Dan Toomey. So Chris Hinch is a senior

network engineer with Gregory Technologies and a NaviSite client. He has got a lot of experience here both

frontline and leadership in networking, and he has had the great pleasure of serving as a network analyst, an

IT manager, and a director of technology. Welcome, Chris.

Dan Toomey is a director of mid-market sales engineering with Time Warner Cable Business Class. Now, Dan

directs the presale design of complex solutions for mid-market clients and has a lot of experience with things

like unified communications, MPLS, IP-VPN security, and managed service. Dan, I want to thank both of you

for joining us today. You’ve both been knee deep in this whole business continuity thing for quite some time.

So I’m hoping you’ll share with us some of your stories as we go through this.

Let me get to the first question. I’d like to ask both of you this question, if I could. We’ve reached a place in

business today where the concept of “the network” has largely become invisible except, of course, when it

breaks. Now, in terms of business continuity and disaster recovery, how does the network role shift and

change as some of the things that Andre was just talking about and things like virtualization, distributed

business operations, and a mobile workforce, and so on, become more the reality rather than the exception?

Chris, why don’t I start with you, if I could.

On premise versus cloud based

Chris Hinch:

Yes. I mean, everything is going to be based on a business use case and there’s never a one-size-fits-all for

anything. What we typically do is, we figure out what the business plan is and how they would like to see their

company move forward, and we tailor a solution for them, because some things need to be on premise for

speed base, and some things need to be cloud based just for access, like your Office 365. Most people don’t

realize that email is business critical nowadays, because that’s how people communicate. So why not move

your email to the cloud so it’s always up? Because here in the States, Office 365 has one of the best SLAs you

can get. I know that’s not the case over the pond a little bit, but here in the States, it’s great. I mean, everything

that we do, if there’s not a one-size-fits-all, you have to really dig in to what’s core to your vertical, your


14

business, and those are the things that you may want to look seriously into: “Maybe this does need to be in the

cloud in case our building burns, catches on fire.” It’s no longer there, no longer accessible, we can still

function as a business.

Steve Shepard:

Very, very good, excellent. Thank you, sir. Dan, how about you, any thoughts?

Dan Toomey:

Yes, Steve. We see similar comments from the folks that we talked with when we’re out talking with customers

and prospects. Office 365 has piqued the interest of a lot of folks in the mid-market, and it really does depend

upon the applications that are most mission-critical for businesses, but we are certainly seeing an increased

interest from folks in the mid-market space around virtualizing, getting their core applications accessed out of

the cloud for the purpose of keeping them survivable in the event of a disaster, because they are so critical to

our customers in terms of keeping their revenue flow going in the event that there would be a disaster. So

we’re seeing increasing interest in putting critical applications such as email in the cloud.

Steve Shepard:

Very interesting. It’s funny, I’m sitting here thinking that there was a time not all that terribly long ago when the

concept of the application service provider, the ASPs, first hit the marketplace and, as we all probably

remember, and in some cases would like to forget, that whole concept went down in flames, and it went down

in flames for a couple of reasons. It was really the first rudimentary attempt at a cloud kind of a solution, but the

market wasn’t really ready for it yet. This idea that said, “I’m not going to put my really business-critical stuff in

somebody else’s hands. That just scares me to death,” but it was also the issue that said we really didn’t have

adequate, universal, truly ubiquitous broadband connectivity yet, and therefore, the whole thing went away.

How fascinating it is now that people are changing that 180 degrees and saying that survivability is going to be

most based on my ability to put my business-critical stuff into the cloud because of survivability capabilities and

redundancy, and least redundant routing, and all that kind of thing. Just very, very interesting shift in focus,

which I think is cool.


15

Copper-based last-mile access

So, here’s another question for you guys. We occasionally hear that the infrastructure has reached a level of

sophistication and stability such that it’s just not a problem anymore. Dan, what are your thoughts on this?

Dan Toomey:

I think we are getting there. I think we will get there. My background is in MPLS, and I can tell you that over the

last 10 to 15 years, a lot of networks—and really it doesn’t even just have to relate to MPLS. It could relate to

internet services, but whether it’s a private network or internet access, there are a lot of those types of services

out there that are provided with last-mile access that is copper based. There are a ton of T1s out there. There

is quite a bit of Ethernet over copper-type technologies, whether it be Ethernet over two-wire copper or whether

it be Ethernet over T1, which is basically just Ethernet over four-wire copper.

There is a lot of copper last-mile access out there and we see that. So when we’re talking with folks out in the

field, we see the pain points associated with that. We see the outages associated with folks who have copper-

based network assets that have been in the ground for 100-plus years in many cases, and we see the lack of

ability to have a diversified service, and they have separate entrance facilities in many cases where there’s that

copper-based access prevalent.

So what we’re seeing is very much an increased interest in folks moving off of copper-based last-mile access

to solutions where there’s fiber-based last-mile access, where you can’t have multiple entrance facilities and

you can’t have diverse paths through the network, and you just have an overall better reliability associated with

fiber-based assets than you do with copper-based assets. I think, Steve, we’re getting there and we will get

there, but there is still a tremendous amount of copper-based last-mile access out there, and I see folks really

eager to get off of that, and into more reliable networking solutions such as fiber-based access.

Steve Shepard:

Yes, but I have to say that for those of us that started our careers in copper, you cut me to the quick. [Laughter]

I bleed copper sulfate here because of [laughter]. No question about it. In fact, very interesting, I recently was

doing some work over in Africa and I got an opportunity to go visit a landing point for one of the major optical

fibers that provides a redundant ring all the way around the continent of Africa. And it was fascinating because

one of the biggest issues they have, of course, is copper has a lot of value, right? It’s expensive, you can sell it

for a lot, and when you don’t have a lot of money, this is a great way to make money, and people were going in


16

and lighting fires under the optical fiber cable that comes up from out of the ocean only to find that the whole

thing just melted and went away, because it was not copper at all. [Laughter] It was, in fact, fiber. So they had

to put up signs that said, “There’s no copper inside here. Trust us. This is actually optical fiber.” Luckily, the

word spread fairly quickly. So, funny how even the market of thieves learns and goes to other sources of

revenue.

Diversity: an ongoing process

So, guys, diversity comes in a lot of different flavors. We talk about power diversity, network diversity, going to

the cloud, et cetera. Chris, I’m just curious here for just a second. How should a company approach this

consideration as they start to develop and perfect, and sharpen their strategy for business continuity? What

kinds of things come into the discussion?

Chris Hinch:

I mean, once again, there’s not a one-size-fits-all. You really need to figure out what the company can survive

on if your internet or if your infrastructure was down. Could you survive in a one-day outage, in a two-day

outage? Could you survive in a week outage? I mean, everything needs to come down to, how long can you

operate without your infrastructure, meaning just cell phones or if you’re on Office 365, can you operate on just

email? Is your business going to survive? If not, then you really need to see what is core. I mean, if BCDR

comes up to be a budget-conscious thing, which for most clients nowadays that is, then you need to look at

really what the customer, what the client needs in order to survive. Not at 100 percent, but just so they can

continue to at least float by and make money, and once they figure out what that is, they need to know it, they

need to use it, the need to update it, and they need to test it frequently.

Steve Shepard:

So it has to be a conscious ongoing process. This is not a once-and-done kind of thing?

Chris Hinch:

No, never, no.

Steve Shepard:

Very, very good.


17

Chris Hinch:

As technology evolves, so should your BCDR.

Steve Shepard:

Yes, of course. Dan, anything you want to add there?

Dan Toomey:

Yes. I’d go back to the results of one of the poll questions and the comment that Andre made that they didn’t

add up to 100 percent whether it was power or network issues, or fiber cuts. You have to focus on all aspects

of the plan, and I don’t really put one potential risk point as higher an importance than another. I think we’ve

seen here through the result of that poll question that folks have experienced more than one of those issues,

and I think they all have to be contained within a comprehensive plan like Chris just alluded to. So those would

be my comments.

BCDR: Where do you start?

Steve Shepard:

Let me ask both of you. This is interesting, because it’s one thing to have great technology and well-trained

people and so on, but sometimes companies have to start somewhere. I mean, I assume there are resources

out there that people can get to. Are there online resources? Where would they go if they’re saying, “I don’t

think our BCDR strategy is up to snuff?” Where do they start? What would you recommend? Dan, I’ll start with

you, because you were just talking.

Dan Toomey:

Sure. From my perspective, you could go to NaviSite.com, and there are quite a few resources out there that

you could draw from. There are also all kinds of resources out on the internet just doing simple Google

searches, but there are some very great materials that you could look at NaviSite.com.

Steve Shepard:

Chris, any place in particular that you’d go to?

Chris Hinch:


18

To echo Dan a little bit, you can get on Google until you’re blue in the face, until you fall asleep from reading

articles because you’re up so late. That doesn’t mean that it’s all good. For every five, six you read you’re

probably going to have one that’s relevant. I would tend to lean towards your peers. Go ask out what other

people are doing. What are they finding that’s working? We got a lot of clients here in the area that I’m in, we

had the floods really bad last year and so they came running to us. It’s like, we need a better disaster recovery.

So we’re actually in the process of revamping our website, which is GregoryTechnologies.com. They’ve put a

lot of helpful hints out there and how they can reach directly out to us just for a consultative thing, not

necessarily to use us. I mean, we’re more than happy for you to use us, but my main goal is just to make sure

that everybody can function. The new website should be launching here in the next couple weeks or so. So just

keep looking out there and we’re going to put a lot of helpful hints out there for you.

Steve Shepard:

Now, I do want to take exception to one thing you said and I’m going to argue to the death with you on this

one, and that was you made the comment that you can read articles about disaster recovery until you fall

asleep. I’m pretty sure that’s the kind of stuff that’s supposed to keep you awake at night. [Laughter] Let’s be

real here for just a second.

Top three best practices

So, a question for both of you and I think, Dan, I’ll start with you on this one. What are your top three, for lack

of a better term, best-practices recommendations with regard to BCDR for our listeners that are on the phone?

Dan Toomey:

Sure. I guess the first one for me would be have a documented plan, a well-documented plan. Have something

written down whether you do that internally, organically, or whether you have some consultancy in helping you

do that, have a plan.

And second of all, once you have a plan, make sure that that plan is updated as you do things like introduce

new apps into your environment. Whatever changes you do in your production environment, make sure you

have that plan updated with revision control and that it’s pretty constantly updated to reflect where you are at

any given time.


19

And the third thing that I would recommend is test that plan. Make sure that your plan is well known by folks

who may need to use it in the event of a disaster and make sure that you test it periodically. It could be every

six months, it could be every quarter, it could be once a year depending upon your comfort. The frequency in

how you test it and when you test it will be based upon your comfort, but test it. Make sure that whatever is in

your plan, your folks who are responsible for implementing that plan in the event of a disaster are

knowledgeable on the plan, and they’re prepared to take action in the event of something happening.

Steve Shepard:

So, have a well-documented plan that is truly aligned with the business and the business operations. Number

two, keep it updated. It’s not a once-and-done thing. And number three, make sure that it’s periodically tested,

and that the right people know how to use it, how to access, how to respond, and so on. Very, very good.

Chris, same question to you really, but let’s be realistic here. I mean, you got to see this firsthand, right? You

recently, as I recall, found yourself trying to recover from a major failure. In fact, as I recall, you celebrated your

recovery with the birth of your daughter the day after the customer was back online, correct?

Chris Hinch:

Yes, that’s correct. [Laughter]

Steve Shepard:

So your gift for recovery was your daughter. Very, very good, all right. [Laughter] Let me ask you the same

question. First of all, any specific best practices beyond what our colleague just shared with us?

Chris Hinch:

We preach the same thing that he said, just not as in depth. Know it, test it, update it. When we get into our

BCPDR roles around here, it’s know it, test it, update it. Exactly what Dan said: have a plan, know what the

plan is, but also know your infrastructure. Know what’s in the rack. Know what’s there, because if you have

somebody that’s operating it and they don’t know what it is, maybe they’re not the ones that need to be on the

plan for you. Test your plan. Make sure that your plan works. Make sure that your backups are working. Just

because you get an email saying that it backed up successfully doesn’t mean that it’s really there. That just

means that there were no errors. It doesn’t mean the files are there.


20

Update it. We’re in the phase still going through where some of the clients are still in this physical server and

they’re not in the virtual servers yet, but they’re going through the conversion of P to V’ing. If you’re P to V’ing,

make sure that you update your backup and your retention policies, because backing up a physical box is

completely different than backing up a virtual box; and if you ever have to restore, a bare metal is completely

different from a snapshot, and they just don’t work well together. Make sure that, as your technology evolves in

your infrastructure, whether it’s on premise or on cloud, that as the technology grows, make sure that your

BCP and your DR grows.

Know your equipment

Steve Shepard:

Let me follow up with this with you and that is, are there any specific skills or areas of expertise beyond the

obvious that people should have, ideally would have, et cetera, in this space?

Chris Hinch:

The skill set can be all around. It’s going to be really dependent on your industry. IT for a homebuilder is going

to be completely different from an IT for a lawyer office, because it’s just completely different verticals. Make

sure that you know your equipment. Just because you see a green light doesn’t always mean that’s the good

thing. An amber light sometimes means that it’s good where a green light can mean it’s bad, because every

equipment is different.

Every provider, whether it’s Cisco, whether it’s HP, whether it’s Dell, they all do things slightly different and a

blue light to Dell means one thing when a blue light to HP means another thing. So don’t just start pulling

cables because you think you see a red light. Don’t just start pulling drives. Make sure that you really know

your equipment, because the one that you just alluded to, that was a fact of the client didn’t exactly know what

they had in the rack and they’re down for 17 days. My time alone to get them up was close to $50,000. That

was just my time. That’s not a lost revenue, that’s not any harbor that we had to put in. That was just my time.

Steve Shepard:

For 17 days, that’s bad, right?

Chris Hinch:

Very. [Laughter]


21

Steve Shepard:

That’s a joke. [Laughter] Okay. All right, gentlemen. Thank you very much. Well done.

Downtime is more expensive than a BCDR plan

What I want to do now in our remaining 10 minutes is to transition over to the Q&A portion of the program.

We’ve got a number of questions that have come in and I’d like to direct them to our panelists. I’m actually

going to direct this first one, Andre, to you. It’s really more of a point made by the person, but I like it. He says,

“I like the point made by the presenter, Mr. Kindness, about the cost of the BCDR solution being less than the

cost of not having the BCDR solution.” That is, the business costs and losses than it can prevent. I mean,

clearly this is an important reality. Andre, in your experience, have you seen situations develop like this where

somebody found out the wrong way about the cost that they were going to encounter?

Andre Kindness:

Yes, it has come up multiple times. I have one client, a big food manufacturer. Nearly in the presentation, top

out inflection point, and the second one was understanding how much technology affects the overall business

in that respect. This food manufacturer, they offload cereal off the manufacturing line right into trucks so they

don’t have any holding cost. And they’re always updating the forklift drivers on what palette goes on which

truck, but it was an older application and they needed the network in that environment, especially the wireless

ones so that the latency was pretty low between the roaming of APs, but they didn’t do it as good as they

should have done. But what that causes is, for the manufacturing line or the process of extruder to actually

stop, and for it to stop what that means for the business itself is actually a six-hour downtime. Not that they

can’t get it up and going which they can turn it back on, but they had the super heat, the extruder, and

everything, parts of the plant to get the bacteria out of it.

So the relationship between—it might be the network, might not be something within a millisecond or within a

minute or 10 seconds, but we need to understand a fully extrapolated out of what to other things coming back

online is, and for the business to completely come back online just like with Southwest even. Well, it took them

four days, for two hours.

That’s the thing I wanted to drive home was, we have to understand the connection between what happens on

the network and the infrastructure and what it means ultimately for the entire business itself. I think that’s the


22

next step in our evolution as networking professionals out there, or technology professionals, is understanding

that business magnification of what things cost and what actually the infrastructure is doing for the business or

helping the business.

Distributed databases

Steve Shepard:

Excellent. Thank you, sir. Dan, let me throw this one to you. One of our audience members says, “I agree with

the point about the role and the value of diversity of assets, and data.” Any thoughts about persistent data

stores and a very diverse placement approach where you have lots of databases and so on, where they tend

to be more challenging, because they’re more distributed? Dan, I throw that to you cold, but any of you, if you

have any thoughts on that, that would be great.

Dan Toomey:

I would say that you can use the cloud to assist in some of that. The importance of the cloud is becoming very

important as we move forward, especially where I sit in the mid-market. There are a lot of folks with distributed

databases who are looking at various solutions to aid them.

Datacenters in the cloud

Steve Shepard:

Very good. So interesting. A follow-up question that came in right after that one says—and I really like this

one—this is a good question because this is a pervasive problem. “How do you convince an IT person that

these changes need to be made if this is someone who is skeptical of things like cloud, remote access, and

even laptops on the network?” This is the famous “BYOD is bad” kind of situation. [Laughter] Chris, why don’t I

throw this one to you to start? What would you do?

Chris Hinch:

In the beginning, in the cloud phase, I was very against it. When I was the director of technology, the word

“cloud” scared me to death more than anything, but as I moved more into the role that I’m at now and actually

see it in use, the use cases are way more beneficial than they are detrimental. You don’t have to go full cloud.

You can go hybrid.


23

Office 365 is the biggest example of that. I don’t know how many people out there love managing Exchange.

It’s probably not that many. [Laughter] So put it up in the cloud, let it run. [Laughter] People that run Office 365,

those are the people that love Exchange. Let them do it and they do it well, and they do it good. Ever since

we’ve moved all our clients to 365, we have not had any issues at all with them. Just because the word “cloud”

is there, technically it’s a datacenter. It’s like the one that the Gregory Technology web services that we use.

Our cloud is housed in Andover, Massachusetts, and it was replicated to Santa Clara, California. So once

people realize that the cloud is actually a datacenter somewhere, it makes them feel a little bit better.

Steve Shepard:

Yes, it’s always funny to me when people point at the sky when they talk about the cloud today. Even IT

people, they point up. “It’s not up there. It’s a datacenter. Come on.” [Laughter] It also points out the reality.

Chris Hinch:

You’re right, their perception is they just think that it’s in the cloud, we can’t see it. Well, I mean, if you really

want to see it, you can hop in a plane and go there, and see it if you want to.

Steve Shepard:

You bet.

Andre Kindness:

Yes, but you know what? I think if you drive home the points, right, education, and the second thing is for our

clients, we tell them to, “You got to change that IT person’s metrics too.” So you educate them and then you

motivate them, right? I think it behooves the business as well as managers to invest the time, but also to align

their metrics and how they measure the person to what they want to get out of the person.

Steve Shepard:

Let’s face it, there’s a certain amount of old-fashioned fear, because quite often the role of IT is that they’re the

ones that do this stuff, and when they’re faced with this reality that says, “We’re moving everything to the

cloud,” they feel as if their job is in jeopardy—and frankly, in some cases it is. So a lot of companies have to go

through that process of saying, “Here’s how we’re going to repurpose you to make sure that your skills don’t go

to waste, but that requires an active decision on the part of leadership.” One of the things this comes down to

is this whole argument that says, “Look, if you’re not in the business of running a datacenter, why are you


24

running a datacenter?” I mean, in some cases I get it, but in other cases, it’s because this is what we’ve always

done. So sometimes a simple shift in velocity, in philosophy rather, might be the right decision in a situation like

this. Interesting.

Dan Toomey:

Steve, I think some of the hang-ups some folks have with putting apps in the cloud is around security as well,

and what I would say is, there are private networking arrangements in the cloud platforms, and so if the issue

around security is the big issue, there are some secure ways to access cloud, and certainly one to bring that

up.

Steve Shepard:

That’s an excellent point.

Power redundancy

Chris Hinch:

I was just going to say one other thing. Since everything has gone away from being physical service and it’s all

mostly virtual now, ever since we virtualize things here, we rarely touch our servers. So why not put it

somewhere else that has power redundancy with NPlus1 that has network redundancy with NPlus1? I basically

NPlus1 everything. I don’t touch it on a regular day. So it doesn’t need to be here. Why should we have to pay

for it?

Building a business case for BCDR

Steve Shepard:

Yes. That, of course, is one of the key arguments and, of course, that means that somebody has to do the hard

work, which means you got to build a business case that says this is the right thing to do. And in some cases it

may not be the right thing to do. There may be other alternatives, other options, right? Very, very interesting.

Chris Hinch:

Yes. If you need help building business cases, that’s part of my job. So definitely, if this is something you’re

really interested in, reach out.


25

Network diversity

Steve Shepard:

Yes, absolutely. I think we got time for maybe one more question here. Andre, let me throw this one to you if I

could. This is an interesting question. “The main issue I’ve had with cloud services, especially with housing

servers, is that if the internet goes down at our facility then nobody can work, as opposed to having servers in

house still allowing the workers to continue to do their job.” So the argument is, by having local infrastructure,

you can protect continuity or protect business operations. Thoughts on that?

Andre Kindness:

I don’t think the internet is going to go down, but it might be a link. [Laughter] It reminds me of a South Park

episode where everybody’s running around about the internet going down. [Laughter] That being said,

probably a link goes down and that digitalization is happening at an edge, and you always have to have it on,

that’s where you start to see this rise of hybrid cloud and the acronym of SDLAN, always having active/active

links, which drives us back to my points during the presentation, network diversity.

If you’re relying on maybe a DSL or even MPLS—and I know their comment about MPLS not going down. I just

had a food manufacturer actually deal with this. MPLS went down, but they had active/active systems, were

using 3G, they were using satellite and MPLS, but they were running across all three. So maybe the

performance went down, but they never were completely disconnected from the internet, and that’s the whole

idea behind that network diversity or hybrid LAN. If that’s the case, those areas should have that type of setup,

and the idea that having servers localized to have the applications means you have to update them, and things

have to consistent, they always have to be connected somewhere else to make sure that the data is relevant.

In the end, we’ve all realized that isolated systems like that don’t really work out. I mean, everything is going to

be connected together and always has to be on, and having things just in isolated pools, because if something

goes down, when it comes back up, how do you know what state it is in to replicate with something else, and

what data is current, and what data is current somewhere else. And that just invites a whole bunch of

headaches as opposed to, “Hey, let’s put the resources, instead of having localized servers, let’s put the

resources around making sure that we’re always on in this new world.”

Steve Shepard:

Very good.


26

Andre Kindness:

I don’t think anybody else wants to add into my little diatribe. [Laughter]

Conclusion

Steve Shepard:

I actually think we’re right at the top of the hour. As much fun as we’re having, we’re going to have to stop

here, but I do want to thank everyone for the questions and we will be responding to them. I want to thank you

all for sticking with us throughout this entire webinar. Now, remember that you can download a lot of content at

the resource list widget from Forrester, the technology assessment profile study, as well as a very informative

infographic that accompanies it. The story of Gregory Technologies is another asset there that’s very, very

interesting and that features our panelist, Chris Hinch.

Now, if you want to learn more about these solutions from our various sponsors, you can click on the URL link

widget or you can call the number that you’ll find there, which is 866-379-1449. Now, in the meantime, Andre,

Dan, Chris, I want to thank you very much for lending your experience to the webinar, to our whole event here

and once again, to our audience, I want to thank you so very much for attending and sticking with us

throughout the hour.

For Time Warner Cable Business Class and NaviSite, I’m Steve Shepard. All the best.

Documents

Webinar: Your Business Continuity Plan Could Be an Epic ...docs.media.bitpipe.com/io_13x/io_133845/item... · Your Business Continuity Plan Could Be an Epic Failure, Webinar, October