Upload
others
View
11
Download
0
Embed Size (px)
Citation preview
An in-depth look atapplication bench-marking for QoEIn this exclusive E-Guide, enterprise management analyst Dennis Drogseth
offers an in-depth look into application benchmarking for quality of experience
(QoE). This guide offers insight into the difference between QoE benchmarks
and diagnostics for application performance and examines some touchstones
for calculating true QoE—explaining why technical benchmarking alone is not
enough and offers a look into the technologies available for enterprise use today.
Sponsored By:
E-Guide
An in-depth look at application benchmarking for QoE
Table of Contents
Page 2 of 12
Table of Contents:
QoE benchmarks or diagnostics for application performance: What’s the difference?
Quality of experience: Why technical benchmarking is not enough
QoE benchmarking unique approaches and unique environments
Related Resources from Fluke Networks
An in-depth look atapplication bench-marking for QoE
E-Guide
Sponsored by:
QoE benchmarks or diagnostics for application performance:What’s the difference?
By Dennis Drogseth
The importance of managing (monitoring as well as actively managing) networked applications for QoE should be all
but self evident. EMA data from 2007 (Figure 1) showed that a healthy 72% of respondents from a wide range of
enterprises and service providers had more than 20 remote branch offices, and in a parallel EMA survey, 34.1% had
more than 100 remote locations. Not only are IT applications becoming the heart of many businesses and central to
their competitiveness, networked applications in all their flavors are reshaping how and where enterprises across virtu-
ally all verticals, government agencies, and even some service providers are doing business. Networked applications
are in fact actually enabling new business models across verticals—true today with Web2, and even truer tomorrow
with unified communications, and in particular the advent of globally dispersed service-oriented architectures (SOAs).
Figure 1: How Many Branch Offices Does Your Company Have?
Given this challenging situation—one that nevertheless is placing more not less attention on network operations—
many networking planners, engineers, managers and architects are looking to leverage more holistic and cohesive
views of network-to-application performance, including QoE. But beyond this almost constant drumbeat for “cohe-
siveness,” there are generally two types of approaches, two schools of thought, and often two types of personalities
most associated with the challenge of managing networked applications. Group A includes those within the network
team who prefer to see their role as absolutely bounded by the networked infrastructure without much (or any)
interest in dialog with other constituencies. These constituencies might include application managers, data center
managers, and even application developers, as well as service management teams or those groups that are directly
customer-facing. To the degree that network operations has a defined set of customers—and in some cases this
group is often first call for networked application problems—the constituencies can directly include the customers
themselves. The second group, Group B, includes those within network operations who either grudgingly, or more
proactively, recognize the need for these dialogs as an intelligent extension of their broader responsibilities.
An in-depth look at application benchmarking for QoEQoE benchmarks or diagnostics for application performance:
What’s the difference?
Sponsored by: Page 3 of 12
As you may have assumed by now, this analyst, along with EMA’s IT consulting team, favors Group B. And so to
answer the question of QoE, this e-Book will reflect a wholehearted affirmation of the idea that application QoE is
a collective, collaborative endeavor that cannot be done with stovepiped metrics and stovepiped thinking. In this
section, we’ll look at the differences between QoE metrics and diagnostic metrics. While the two may overlap, they
are fundamentally different.
Figure 2: How Would You Rate the Following in Terms of Diagnosing the Root Cause of a Problem?
As you can see in Figure 2—taken from early 2007 EMA research—network configuration, systems configuration
and application configuration top the charts, followed by event-based information, when it comes to diagnosing
problems with application performance over the network. Other technologies—such as dynamic packet analysis,
session reconstruction, and flow-based volume information—follow. This data is striking in that it shows the growing
awareness of configuration information—changes made to the infrastructure, in actually diagnosing problems
reflective of application performance over the network. The population involved with this research was about 60%
network operations and 40% application management and data center (respondents had to feel that they “owned”
or at least “shared in owning” responsibility for managing application performance over the network). This data
alone reflects the need for collaboration, and the report also showed that more than 60% of our respondents actu-
ally had organizational changes made to support better cooperation between network and application managers.
All this is well and good, and useful in its way. In other research, the importance of flow-based information as a
continuum (application flows, packet analysis, route analytics, transaction analysis) showed more strongly, which
is worth pointing out here. But the even more important point is that none of this is QoE.
In this same research in Figure 3, we see some rather basic priorities relevant to QoE, with availability edging
out response time 71% to 59%, followed by mean time to repair (MTTR), mean time between failure (MTBF), and
round-trip latency across the network. In following sections, we will look more closely at what QoE metrics can
and should be, and how you can best instrument to gauge them. Suffice it to say here, though, that the first step
An in-depth look at application benchmarking for QoEQoE benchmarks or diagnostics for application performance:
What’s the difference?
Sponsored by: Page 4 of 12
in planning for effective QoE for networked applications is to recognize that it is a collaborative effort. The second
step is to realize that QoE and diagnostics are typically two separate categories of metrics, even if there is some
overlap. One set of metrics, QoE, begins to capture that unfathomable complexity of actual user “experience” in
dimensions, ideally, most relevant to the user.
Figure 3: How Would You Prioritize the Below for Assessing Quality of Experience?
Putting yourself in their shoes is the way to start. One hint: If you think that the network is a self-contained entity
and that you should care only about the demarcations of your WAN, you’re missing the point, even if that’s how
you’ve been trained organizationally. The network is a part of the larger delivery system for QoE in which the only
true metrics that matter ultimately reside within the flesh-and-blood experience of your consumers. Even if you’re
a WAN service provider, you should begin to think twice about the complacency of ignoring this core requirement.
As many as eight years ago, one CIO threw out his brand-name telco for one that would partner more smartly on
applications versus plumbing. And in the enterprise, network management will forever be relegated to blue-collar
status, without even a seat at the table to defend itself, if it ignores this increasingly relevant lesson.
The good news is that technologies are available today that are well tuned to network operations that can address
this very requirement. We’ll address those in section three. In the next one, we’ll examine some touchstones for
calculating true QoE.
About the Author: Dennis Drogseth is the vice president of Enterprise Management Associates (EMA), an IT
management research, analysis and consulting firm. Having joined EMA in 1998, Dennis currently manages the
New Hampshire office. He has been a driving force in establishing EMA’s New England presence. Dennis brings
24 years of experience in various aspects of marketing and business planning for systems and network solutions.
He directs a team of analysts who focus on the development of the Networked Services Management practice areas
that span performance availability and service management across enterprise and telecommunication markets.
His team also addresses accounting, billing, QoS, outsourcing and other disciplines related to these markets.
An in-depth look at application benchmarking for QoEQoE benchmarks or diagnostics for application performance:
What’s the difference?
Sponsored by: Page 5 of 12
N E T W O R K S U P E R V I S I O N ©2008 Fluke Corporation. All rights reserved.
As a certified Cisco Technology
Developer Partner, Fluke Networks
gives you valuable insight into
Cisco infrastructure technologies
and data sources.
Fluke Networks helps Cisco customers
discover the power of Cisco IOS® by
utilizing embedded technologies such
as NetFlow, IP SLA, and Performance
Routing (PfR), for in-depth forensic
analysis, VoIP and application
pre-assessment testing, proactive
performance management and more.
www.flukenetworks.com/cisco
Discover the power of Cisco IOS technologies
s a certified Cisco TA
,artnerr, Flukeloper PvDe
aluable insight into ou ves ygiv
echnology o TTechnology
sorke NetwFluk
insight into aluable insight into ou ves ygiv
eastructurCisco infr
.cesand data sour
elps Cisco custometworks he NFluk
e power odiscover th
ed techndg embedutilizin
insight into
e technologies
ers ps Cisco custom
OS® by f Cisco Io
ch es suologitechned techndg embedutilizin
, IP SLA, anetFlowas N
or i), fRfg (PRoutin
d aoIP an, Valysisan
t testinene-assessmpr
eagance manormperf
ch es suologitechn
ce anormerfod Pan
csienorepth fforn-d
oncatiappli
oactive g, prtin
.eord mt anenem
sorkenetw.flukwwww.fluk .com/ciscos
N E T W O R KW O R K S U P E R V I S I O N ©2008 Fluk .eservedts rghon. All riatie Corpork
Quality of experience: Why technical benchmarking is not enough
By Dennis Drogseth
In the last section, we looked at some metrics prioritized by respondents in EMA research from last year. The
rankings topped with availability, then response time, then MTTR, MTBF and networking round-trip latency. We
also separated this class of metric from diagnostic metrics such as configuration information and forensic session
reconstructing.
In this section, we’re going to take the bull by the proverbial horns and look in more depth at metrics truly targeted
at QoE. The first thing to say is that in terms of customer priorities, our respondents got it wrong. Any number of
studies have shown that end users care more about degraded response time than intermittent availability issues.
This has more to do with human psychology than network engineering. End users typically believe that a complete
failure in availability will soon be remedied, whereas they feel isolated and unsure that any action will be taken if
their response time is degraded. Moreover, degraded response time tends to persist far longer than most availability
issues, especially in 2008. So, in this case, their perception really is reality.
And by the way, end-to-end network latency is of course only an approximation of response time. In the next
column, we’ll dive more deeply into unique technologies—most notably synthetic transactions and observed
transaction response. But for now, suffice it to say that while both apply, for true QoE it’s important to capture
observed response time from the end-station out.
Yet response time can also be problematic in other ways. Averaged response time over a day or a week or a month
may not be very meaningful in itself. Inconsistent response time metrics, even with faster overall averages, can be
far more troublesome to rhythms in working and communicating than somewhat slower but more consistent service
delivery. And those terrible spikes that alienate users can occur within a single minute—spikes that may not only
help to catch alienated users but also help to provide insight on where the problems lie.
And response times, as central as they are, are just one metric for QoE. For instance, setting the appropriate
response-time goal should be predicated not on pure-play speed but on appropriateness and cost. This, of course,
is where SLAs can come in—and they should be based on what you know to be true about the needs of your
customers, not what you presume to be true. For some applications, such as email, your customers may care
more about flexibility in accessing their mail between wireless and tethered environments than about absolute
response time.
Even availability can be a challenge. The availability of the “network” is in itself a far from obvious discussion. In
Figure 1, you can see a number of components ranging from servers, hubs, routers and database transactions that
can all affect availability and, of course, performance. The math is easier in availability—as availability tends to
aggregate across components as Figure 1 demonstrates. Performance metrics can be more complex, and—depend-
ing on the specifics in timing and parallel activities—they may or may not aggregate.
An in-depth look at application benchmarking for QoEQuality of experience: Why technical benchmarking is not enough
Sponsored by: Page 7 of 12
Figure 1: End-to-End Availability
And yes, MTTR and MTBF do affect end-user experience—statistics that you need to understand whole-cloth through
your service organization if not directly through your own internal metrics. In other words, if you really care about
QoE, you should understand MTTR and MTBF as they affect the service consumer, not simply as they are relevant
to one of the components in the network.
But other metrics will come into play with various degrees of relevance. These include flexibility and choice of
service—something in which network planning plays a role. Data security is another core value that people may
not think of in QoE, but for certain applications and certain information, it can be a prime customer concern—one
for which, when cost is a factor, your customers may be willing to pay more, for more absolute guarantees. And
speaking of cost—well, cost effectiveness and even visibility into usage and cost justification are of increasing
interest to business clients who may themselves be expected to contribute to the value of their service. Mobility is
another QoE attribute, more important for some applications than others, as I’ve already indicated. And frankly, the
list goes on.
The main point to remember is that each application and each customer set may suggest different QoE parameters.
This means dialog (either direct or indirect through your service organization), and that dialog should be iterative,
as business demands and requirements change. You can save yourselves a lot of time, money and grief just by
making sure up front that you’ve invested in listening to your customers’ top requirements, and then instrument to
support those, versus scattering your efforts in an introverted and uninformed manner. In this way, QoE is a little
bit like being a good partner in a marriage—doing what’s right for the two of you, not just doing what you believe
to be the right thing without asking.
An in-depth look at application benchmarking for QoEQuality of experience: Why technical benchmarking is not enough
Sponsored by: Page 8 of 12
Service Level Objective: 99.9% availability;downtime <= 50 minutes
LAN 99.97% 13.14
Local Server 99.95% 21.9
Building Hub 99.96% 17.52
Intranet Router 99.88% 52.56
Remote Host 99.93% 30.66
Order Entry Applic. 99.90% 43.8
Customer Data Base 99.92% 35.04
Inventory Data Base 99.91% 39.42
Total 99.42% 254.04
Minutes ofComponent % Availability Downtime
QoE benchmarking unique approaches and unique environments
By Dennis Drogseth
Benchmarking for QoE immediately presents a challenge from any number of perspectives. There are many network
technologies that, in most environments, function together. These support a variety of protocol types and in the end
enable the delivery of multiple application types. I don’t think EMA has worked with a single IT or service provider
that had a simple, monolithic set of requirements for delivering application services over the network.
Figure 1: Which of the Following Technologies Does Your Network Currently Incorporate?
Let’s look first at the network transport technologies present in most IT organizations today, as in Figure 1. Old
technologies such as frame relay and ATM cohabit with wireless, VPN and MPLS networks, many of which function
as virtual overlays to core Ethernet transports. WAN, LAN and VLAN technologies exist together, along with the
many flavors of WiMAX for last-mile wireless complements. And while the answers for QoE benchmarking reside
beyond all of these transport challenges, they are far from immune from them. If you care about QoE, you need to
have real visibility into the design and performance of your network, ideally with technologies that can begin to
effectively link the user “experience” with real network transport issues.
More to the point for QoE itself, there is a wide array of application technologies and types, as in Figure 2. As you
can see from this data from last year—and EMA has revisited this discussion on several occasions with virtually
identical results—Web applications clearly dominate when it comes to managing the performance of applications
over the network. Many of these application types come with unique protocols—such as AJAX, Java or ActiveX, and
SOAP—and are increasingly becoming componentized so that more pieces of different applications are residing in
different locations, escalating the complexity of the user’s end station (laptop, desktop, mobile device) with virtually
the rest of the outside world. This complexity will only become greater as service-oriented architectures (SOAs)
An in-depth look at application benchmarking for QoEQoE benchmarking unique approaches and unique environments
Sponsored by: Page 9 of 12
become more and more pervasive. Data from mid-2007 shows that SOA is already beginning to challenge VoIP as
a protocol of concern to operations managers.
Figure 2: Which Applications Are You Delivering or Planning to Deliver Over a Network?
Given the limits of space, I have chosen to focus on benchmarks targeted at this increasingly mainstream, Web-
based application world, as opposed to VoIP or video, which do, of course, present unique challenges of their own
and which deserve their own, separate discussion. The first thing to stress is that QoE telescopes the problem
beyond the network itself into the true end-user experience. You can’t benchmark the network if you’re blind to
these transaction-driven details, which often require complementary and more application- oriented monitoring than
most network management solutions offer. As a result, I am going to focus primarily on those technologies and
solutions targeted at capturing transactional levels of awareness.
One of the more hotly debated topics over the last few years has been the difference between synthetic transac-
tions and observed transactions in testing application response. Another part of this discussion involves where the
transactions are monitored by location—e.g., in the data center, at the edge of the data center, or from end stations
themselves. And if the end station, is it sufficient to have instrumentation on a per-branch office basis, or do you
need to strive for blanket coverage across a wide variety of end-user points?
As for synthetic versus observed, the truth is that both are valuable. Synthetic tests are proactive, can give you
more consistent data suitable for SLA requirements, and can let you know if availability is lost, which observed
transactions typically cannot. Many synthetic tests also offer diagnostic value, especially when the scripts are
optimized to look at certain types of transactional behaviors that occur on an ad-hoc basis in the real world. On
the other hand, synthetic tests occur at specified intervals and therefore may fail to capture any number of real
problems that occur in finite time frames. They may create overly simplified pictures of real-world experiences by
a wide variety of the customers you may care about most. Moreover, many observed capabilities have become
increasingly rich in function and are beginning to offer much of the granularity of insights once available only in
synthetic tests. So the truth is that both synthetic and observed should be in place—if you really care about QoE.
An in-depth look at application benchmarking for QoEQoE benchmarking unique approaches and unique environments
Sponsored by: Page 10 of 12
Placement is also a choice that is more both/and than either/or, but one where you should apply common sense
based on business needs and cost versus a monomaniacal urge to achieve technical perfection. Data-centric trans-
actional monitoring can provide back-office detail that is quite useful in diagnostics, but it can also provide rich
insights into certain issues surrounding QoE—in some cases, playing back actual transactions in cinematic fashion.
These solutions can catch Web-application design issues, and even problems with layout and graphics, to which—
from a network perspective—you may feel immune. But your immunity can be challenged when you’re blamed for
degraded network performance that’s really caused by poor application design. However, there are also solutions
that provide full-service playback from the user end station and which capture a different set of dynamics more
thoroughly—fully aligned with how the end user sees the world. And while many of these, especially those opti-
mized for rich transactional detail, are optimized for branch office rather than blanket deployments, both are
available and, to be frank, offer largely complementary values.
And then many of the more network-centric solutions for QoE benchmarking sit at the edge of the data center and
calculate end-user experience, in some cases in conjunction with insights into the back office transaction as well.
Some solutions also offer probes, or are probe-based, with a focus on network segments and are optimized to
assess bandwidth/per application usage. While many of these function at Layers 3 and 4, almost all at least support
HTTP monitoring, or detailed capture of Session Layer (Layer 5) transactions, and some offer more granular trans-
action analysis through Layer 7. Though most of these are not the most “heavy hitting” in the true QoE sense,
having their insights can allow you to execute far more quickly on actually diagnosing the cause of the problem,
and some will suffice for proactively anticipating most performance degradations in remote locations.
I’m going to close this e-Book with a partial list of vendors that offer strong insights into QoE from one or more of
these perspectives. These include services such as Keynote and Gomez, which can offer valuable insights across
the board, but which have come out of the tradition of synthetic test suites. Data-center transaction-oriented
monitoring tools such as Empirix (which combines Web and VoIP call center interaction) and TeaLeaf (for granular
transaction replay and analysis) can be complemented by strong end-station visibility through solutions such as
AlertSite, Coradiant, Symphoniq and Xangati. Multi-purpose capabilities for QoE are available from larger vendors
such as Compuware, Fluke Networks, OPNET and Quest. And finally, these complement some of the more network-
centric approaches for QoE such as those from Apparent Networks, NetScout, NetQoS and Shunra.
How you choose to approach this thorny problem is, once again, a matter of pragmatism, in which cost and overall
resources should be balanced with the need to know all the gory details. But do realize that benchmarking for QoE
is an operations-wide endeavor and should ideally even touch application development and QA/Test as well. You
can’t optimize the network in a vacuum when the experience you’re trying to support lies beyond it.
An in-depth look at application benchmarking for QoEQoE benchmarking unique approaches and unique environments
Sponsored by: Page 11 of 12
An in-depth look at application benchmarking for QoERelated Resources from Fluke Networks
Sponsored by: Page 12 of 12
Related Resources from Fluke Networks
Monitoring & Improving Application Performance across your Company (WebEx player required—download at
www.webex.com/downloadplayer)
Correlating Granular Network and Application Visibility for Improved Performance
Leveraging Capabilities in Your Existing Cisco Network for Optimized Performance
VoIP and MPLS—Making it work for your company
About Fluke Networks
Our history of innovation, product quality, and customer service began in 1948. Today, Fluke Networks is part of
Danaher Corporation, a growing Fortune 500 company and leading manufacturer of professional instrumentation,
industrial technologies, tools and components with revenues of more than $11 billion (USD) annually.
Our technology offerings are used by major carriers including AT&T, Global Crossing, Sprint, Verizon Business and
others to run their managed services. Our global reach of sales offices, laboratories, factories, and home and retail
environments spans six continents and more than 50 countries and gives customers the peace of mind that they
made the right choice in partnering with Fluke Networks for all of their Performance Management needs.
www.flukenetworks.com