Identify 3 Case Studies Whitepaper

8/8/2019 Identify 3 Case Studies Whitepaper

1/14

Best Practicesfor ResolvingBusiness-CriticalApplication Problems

3 Case Studies from the

IT Support Battlefield


2/14

Best practices for resolvingbusiness-critical application problems

2006 Identify Software, Ltd. 2

Abstract

With the cost of application downtime ranging from nearly $100,000 per hour

in the retail industry, to literally millions per hour in the finance industry its not

surprising that companies devote significant resources to resolving application

problems. What is perhaps surprising (and, fortunately, correctable) is how much

money and effort the process demands. Why is application support so costly?

Firstly, the laborious process of problem resolution is still performed manually by

staff from the application development, IT operations, and help desk teams,

which ties up company resources.

Secondly, root cause analysis is still an iterative process of information gathering,debugging, and attempting to recreate the problem in a lab, leading to finger-

pointing among departments, long resolution times, exorbitant downtime

costs, and frustrated users.

Accelerate the problem resolution process, and you can eliminate some of the

costs and improve customer service.

This paper examines the challenges of application support and presents three

(3) case studies that demonstrate how leading global companies in the banking,

telecommunications, and healthcare industries are using best practices to automate

their support processes, eliminate the need to recreate problems in the lab, andreduce their IT support costs significantly all with a rapid ROI.


3/14



Table of Contents

Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2

Applications demand a new kind of support . . . . . . . . . . . . . . . . . . . . . . . . . .4

Downtime costs big time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5

Finance Industry Case Study: Danske Bank . . . . . . . . . . . . . . . . . . . . . . . . . . .6

Telecom Industry Case Study: Cox Communications . . . . . . . . . . . . . . . . . . . .8

Healthcare Industry Case Study: Cerner Corporation . . . . . . . . . . . . . . . . . . . .9

Identifys AppSight Application Support System . . . . . . . . . . . . . . . . . . . . . .11

Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13

About Identify Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .14


4/14



Applications demand a new kind of support

We live in an online, wired world. The digital and physical are so intertwined

that almost nothing can happen unless the relevant applications are up and

running. You would think that as enterprise applications became indispensable,

they would likewise become more and more dependable. Instead, the rising

tide of systems complexity and interdependency are making enterprise application

problems inevitable, intractable, and more elusive.

Business interruptus reigns

At one major insurer, for example, a payables application mysteriously kept

crashing on startup, but only for one unlucky group of users, and only on two

of six servers. After investing over a month and $50,000 in a multi-engineer

analysis, the organization still had not discovered a cause or cure.

At an upscale retail specialty store, cashier queries for customer records were

hanging up the registers during the wait for overdue server responses. Then,

every time an impatient cashier rebooted an individual register, this locked

up all other registers, requiring a 30-minute server reboot that suspended all

transactions storewide. Nine technicians worked the problem for three months

to no avail, costing over $150,000 not counting lost sales and goodwill.

A leading hospital management organization faced recurring, intermittent

processing delays in a Web-based application. When initial resolution efforts

proved fruitless, the organization deployed a 15-person crisis team that cut

across all application and infrastructure domains. This team spent some 3,000

hours $300,000 in fully loaded costs and then, still stymied, looked outside

for further help.

These are not isolated examples. They represent business as usual (or the

interruption of same) throughout the Fortune 2000 today.

More power, more players, more problems

The characteristics that make todays enterprise applications so capable and

productive when all goes well, make them nearly inscrutable when things go

wrong. Contemporary enterprise applications encompass millions of lines of

code, never fault-free in such quantities. They serve myriad users who, being

human, predictably do unpredictable things with them. They span multiple

platforms and networks, run by multiple groups who wield fragmentary

* See p.11 to see how these problems were resolved.


5/14



expertise, authority, and views of the environment.

The net results are mission-critical organisms with millions of moving parts.

At this scale of complexity, processes inevitably collide and conflict, with elusive

root causes and costly results.

Downtime costs big time

With the cost of application downtime in major organizations ranging from

nearly $100,000 per hour in retail, to literally millions per hour in banking and

brokerage, its not surprising that companies devote the necessary resources to

resolving application problems. What is perhaps surprising (and, fortunately,

correctable) is how much money and effort the process demands.

Throughout the 7.2-year average lifespan of major enterprise applications,

software maintenance and support costs represent between 60-80% of total

software lifecycle costs. This limits developers ability to advance new initiatives,

and goes far to explain why corporations tend to view IT operations as cost centers

rather than the strategic enablers and centers of value they can and should be.

Software maintenance and support is so costly because the largely manual, trial-

and-error-driven approaches that prevail cannot cope with todays complex

enterprise applications. Consider the course of a typical application malfunction.

An end user reports a problem to the help desk, initiating a mutually frustrating

exchange. The responder tries to learn everything possible, but the user is hard

pressed to help because until problem intruded he or she was focused on

business objectives, not keystrokes and system states.

Brokerage $4.5M Healthcare $0.5M

Telecommunications $3.0M Hospitality/Travel $0.5M

Energy $3.0M E-Commerce $0.1M

Banking $2.6M Retail $0.09M

IT-dependent Manufacturing $1.5M Transportation $0.09M

Business impact of downtime, per hour

Source: META Group, Oct. 2000 and The Yankee Group, March 2004


6/14



Problem replication is the roadblock

Even when end users trouble reports are impeccable, the problems that absorb

the bulk of support resources present themselves as mysteries. In addressing

these, fully 80% of the effort goes not into resolving the problems per se, but

into replicating them so their root cause can be identified. Its painful to spend

resources re-creating problems no one wanted in the first place. But when the

root cause is unknown, the problem must be replicated, just as your car has to

reproduce that indescribable, intermittent noise when the mechanic is listening

or you cant get the help you need.

On average, problem replication efforts involve 6.5 experts, who must replicate

the problem 4.8 times (with fingerpointing, headscratching, and further grilling

of exasperated end users) before they establish its root cause.

Following are examples of how three top organizations in different industries

are using an innovative technology approach that enables them to eliminate the

cumbersome problem replication process, so they can meet the application support

challenge head-on accelerating the problem resolution process and reducing

application support costs significantly.

Finance Industry Case Study: Danske Bank

Founded in 1871, Danske Bank is the largest bank in Denmark and a leading

player in the Scandinavian financial markets, offering a wide range of financialservices, including insurance, mortgage finance, asset management, brokerage,

credit card, real estate, and leasing services. In 2000, Danske Banks IT organization

designed Business PC to provide its customers throughout Europe with a

reliable, easy way to perform financial transactions securely and cost-effectively.

While Business PC would

provide new levels of

service and cost efficiency,

application problems

constituted a major risk.

A single lost or delayed

transaction could proveboth expensive, and

damaging to the

banks reputation.

A lifecycle approach to application problem resolution

Ebbe Skak Larsen, Chief Architect at Danske Bank, recognized the need for a

better infrastructure for application problem resolution throughout Business PCs

Return on investment for the first yearof operation alone amounted to more

than $1,000,000. - Ebbe Skak Larsen, Chief Architect


7/14



lifecycle. After initiating an in-house project to build a comprehensive system

support solution, Larsen found Identify Softwares AppSight Black Box systemand Danske Bank has been using it ever since.

With AppSight, the support team can easily pinpoint the root cause of application

failures, such as performance bottlenecks, configuration problems, and functional

errors, without having to recreate the problems, saving them significant time

and costs. Today, Business PC is offered in 11 localized versions and serves over

15,000 companies and 50,000 users in across Europe and the US. AppSight

Black Box is an integral part of the Business PC package, enabling Danske Banks

support team in Aarhus, Denmark to provide users with technical support as

problems emerge.

Eliminating the need to recreate problems in a lab

According to Larsen, even with good QA, errors will occur in any complex

application. "Production errors are typically difficult to recreate in our test and

customer support labs and might result in our customers experiencing costly

application downtime," Larsen explained. AppSight helps Danske Banks support

operation work more effectively, streamlining the process of capturing and

communicating problems, and providing rapid root cause analysis for problem

resolution. "It can cost a company tens of thousands of dollars to pinpoint a

single application error, recreate the problem, and resolve the issue, in addition

to lost employee productivity and customer confidence," said Larsen. "AppSight

enabled us to identify the cause of problems in a few hours, rather than a few

weeks. We have solved some very elusive problems we may never have found

without AppSight."

Bottom-line benefits

Danske Bank adopted AppSight Black Box technology because it ensured the

highest levels of customer support for Business PC. Prior to implementation,

Larsen estimated that using AppSight would save the bank more than $900,000

over a period of five years. A year after implementation, Larsen calculated that

return on investment for the first year of operation alone amounted to more

than $1,000,000.


8/14


Telecom Industry Case Study: Cox Communications

Cox Communications is among the largest broadband communications companies

in the U.S., serving over six million customers in more than 20 states and

Cox.com is the primary interface between Coxs 16,000 employees and its

customers. Through Cox.com, Cox Communications provides the latest service

information, sells new services, and provides state-of-the-art electronic bill

payment. The Cox technical support team supports 30 internal applications and

40 intranet applications for each of the companys remote customer-facing

sitesthe complete system comprises over 25,000 nodes.

Maintaining responsiveness, while containing costs

Rapid application error resolution is crucial to generating revenues through

Cox.com. "We need to provide our customers with real-time responses to their

requests, regardless of any errors that a transaction has generated," notes James

Hampton, Coxs Manager of Configuration Management and Systems. Every

time a customer signs on to Cox.com, we want to be able to offer him promotions

and special discounts according to his profile, as well as update his account

history. As a result, a single transaction touches between 2 and 6 servers.

The Cox.com system

ensures that the

customer gets a

payment receipt, even

for a failed transaction.

The order is then

reconciled offline by

the back-office group

in a time-consuming,

manual process.

To reduce application

support costs, Cox

selected AppSight for

its state-of-the-art

problem resolutiontechnology. Given the complexity of our system, resolving application errors is

very costly and time-consuming, explains Hampton. Recently, for example,

we had a problem with a specific transaction failing to complete its update

sequence through our system. It took us three days, involving two people each

from three functional groups to determine that the error was the result of an

Oracle database being taken offline for regular maintenance.

Getting to the root cause of that problem took us over 140 man-hours. Now


Getting to the root cause of that problemtook us over 140 man-hours. Now AppSightenables us to record the exact movement ofa transaction and the associated activity.

We now solve this kind of problem in less

than an hour. - James Hampton,

Manager of Configuration Management Systems


9/14


AppSight enables us to record the exact movement of a transaction and the

associated activity. We now solve this kind of problem in less than an hour,says Hampton.

Comparing configurations to pinpoint problems

Cox is finding that AppSight is effective in finding the root cause of transaction

errors in their web applications, and in pinpointing more complex issues, such

as periodic performance degradation on specific servers. For example, Hampton

recently applied AppSight to understand why one database server slowed down

periodically in the morning. AppSight provides us with the ability to compare

the configurations of a server that is working properly with that of a problematic

server, all from the applications perspective, says Hampton. We combinedthis comparison with log samples during these slow periods and discovered

scheduling issues with a group of COM objects. We adjusted schedules and

achieved significant performance improvement. Typically, the technical support

team spent over 50% of its time recreating a problem scenarioAppSight

eliminates this step. You want to minimize the fingerpointing and get to actual

problem solving as fast as possible, Hampton notes.

Providing value throughout the application lifecycle

Using AppSight in production provided Cox with huge savings, but Hampton

sees even greater value in applying the technology throughout the applicationlifecyclefrom development and testing to production. AppSight provides a

comprehensive log of how an application actually behaves in the production

system. I would like the development team to review those logs and see what

kinds of errors and conflicts emerge from a classic set of sessions. This final

check would make our application rollouts to production much smoother and

could dramatically reduce the number of undetected problems in our applications.

Healthcare Industry Case Study: Cerner

As a world leader in healthcare software solutions, Cerner is replacing paper

charts with intelligent, interactive electronic forms designed to improve patient

care and business management by eliminating error, variance, waste, delay,

and friction. Because the lifeblood of modern medicine is information, and

healthcare institutions increasingly depend on software solutions to manage and

deliver it, downtime and application problems can bring patient care to a costly

and hazardous halt.

Minimizing downtime requires rigorous quality assurance and its typically a

manual, laborious, and costly process. Given the labor it takes to assure the

software quality demanded in healthcare, where lives are literally at stake, we



10/14


were immediately intrigued by Identifys AppSight Black Box technology,

recalls Owen Straub, Cerners VP of Engineering.

Cerner first used AppSight to accelerate its Quality Assurance process, and

found that the more than 40 engineers in its QA group gained two hours

per person, per day. Because Cerners QA engineers also provide level-three

customer support, they soon saw the difference between solving problems in

the lab with AppSight and the challenge of troubleshooting problems in the

field without it.

Zooming in on root causes

Cerner now embeds Identifys Black Box technology in its Cerner Millenniumhealthcare solutions platform. The Black Box is transparent to users and has no

effect on system performance. But its effect on customer support is dramatic.

Either running proactively and feeding log data into cyclical, FIFO buffers, or

running on-demand when problems emerge, the Black Box captures field

problems when, where, and

how they happen. This

enables Cerners engineers to

analyze customer Black Box

logs with AppSight, swiftly

pinpoint the root cause of a

problem, and proceeddirectly to a solutionsaving

the hours, days, or even

weeks involved with problem

replication. We found that

employing AppSight in production virtually eliminates the need to recreate

problems in the lab, saving us significant time and costs, says Straub.

For example, a Cerner client in radiology reported a baffling problem:

a transcriptionist found that random paragraphs kept mysteriously disappearing

from her transcriptions. The problem escalated to the point that the client needed

it fixed immediately, since avoidably incomplete information in healthcarerecords poses great potential danger. Cerner sent its level-3 SWAT team on-site,

as stipulated in the clients service agreement. When the transcriptionist stepped

out for a moment, the problem happened again, so the Cerner engineer activated

the Black Box. Then, when the problem quickly recurred, the engineer was

able to study the log and discover that the transcriptionist, in cleaning up raw

transcription files, kept hitting a combination of keys that the relevant program

interprets as delete paragraph. With the root cause revealed, the fix was

simple, and nearly instantaneous. Had AppSight Black Box technology not been


We found that employing AppSight inproduction virtually eliminates the needto recreate problems in the lab, saving

us significant time and costs. - Owen Straub, VP of Engineering


11/14


used to trap this problem in the wild, Straub estimates that replicating such

an unforeseen and idiosyncratic problem could have taken weeks.

Along with reducing the duration of client site visits and dramatically shortening

downtimes, AppSight has reduced the number of site visits needed because

Cerner engineers can now solve more problems remotely. Before AppSight,

says Straub, tough customer problems pulled our developers away from new

product work for intolerably long stretches. Now that AppSight reveals the root

causes, they can be instant heroes and get right back on task.

AppSight creates a win/win situation: Cerner delivers superior product performance

and better, timelier customer support while spending less money to do so.

Cerner clients deliver superior patient care and gain business efficiencies whileenjoying increased confidence in their information solutions and the company

standing behind them.

Identifys AppSight Application Support System

Identifys AppSight accelerates problem resolution and reduces application

support costs by capturing system operation in real time, at every level of

granularity. This lets support personnel analyze root causes and resolve

problems without first facing the costs, delays, and uncertainties of replicating

application problems.

AppSight uses Identifys patented Black Box software technology to capture acomplete, real-time record of enterprise application system events, conditions,

and configuration. Just as the flight recorder or black box in an airplane captures

real-time data that provides invaluable diagnostic information if theres a problem,

AppSights Black Box logs provide diagnostic information that helps get to the

root cause of problems that arise with an enterprise application and/or the

system on which it runs.

AppSight allows actual events to be replayed like a powerful DVD that shows

the end-user activity, synchronized with the system-level activity, and even the

associated coderather than inexactly replicated through laborious guesswork.

With full problem information captured in one compact Black Box log, problemresolution cycles that once took weeks and included endless fingerpointing and

multiple blind alleys, can now be compressed to just a few hours, or sometimes

even minutes.

The Black Box is an extremely compact, software-only module that requires NO

changes to the source code, executables, or system. The performance impact is

negligible. It works either on-demand or in continuous mode, recording events

at levels of detail that you specify, and retaining the record in a cyclical buffer

for the duration you choose.



12/14


AppSight works seamlessly across your entire application landscape J2EE,

Microsoft .NET, Windows DNA; on clients and servers, locally and remotely. Ithelps Help Desk, IT Operations, and Application Development teams collaborate

by providing each with Black Box log analysis views at appropriate levels of

detail. With AppSight, each recording layer is just one click away, making root

cause identification quicker and more efficient, and because Black Box logs can

be distributed through standard trouble ticket systems, it ensures that all teams

share an accurate, consistent, and congruent view of the application problem.

AppSight complements and synergizes the existing, more narrowly-focused

tools that organizations use to support their enterprise systems, such as systems

management products like Unicenter, Patrol, Tivoli,

and MOM, which provide monitoring and

troubleshooting for the system infrastructure,

and software development tools that enable

developers to test software functions and

hardware/software interactions under lab

conditions. In contrast, AppSight focuses on

the real-time, production behavior of the

application, generating Black Box logs that

fill a huge informational void. By placing

these logs at the heart of related system

information and, optionally, generating

ideal logs that serve as templates

for smooth system functioning, support

organizations can readily pinpoint anomalies

and root causes like never before. And once support

team members transfer their energy from replicating problems to resolving

them, the payoff is profound. By supporting, streamlining, and automating the

formerly manual processes that make up application support, AppSight reduces

overall application support costs by some 40%-60%.

To appreciate the difference that AppSight makes, lets revisit the three problems

we mentioned earlier:

When the healthcare insurer installed AppSight, they had already invested$50,000 into a stalled application problem analysis. With AppSight, they were

able to solve the problem in less than two hours. Analysis of the Black Box log

pinpointed the root cause of the problem to be access rights to the required

Windows registry key, due to an inadvertent and undetected change.

AppSight enabled the specialty retailer to solve a cash register problem that had

eluded them for more than three months at a cost of $150,000 and counting

in just one day. Resolving this problem not only stopped a $50,000-per-

month cash burn, but also enabled the retailer to resume scheduled new store



13/14



openings that had been placed on indefinite hold for the duration of the crashing

cash register nightmare.

Finally, the hospital administration organization that had fruitlessly deployed a

15-person, ad-hoc crisis resolution team to troubleshoot its web-based application

found that AppSight helped them narrow the problem down for the first time

since it emerged in just a few hours. By isolating the root cause to specific

components in the application server tier, the company was able to return most

crisis team members to their normal responsibilities. Within the next several

days, the remaining engineers pinpointed the problem to threading issues with

their COM object, and solved it.

Had the company deployed AppSight from the beginning, they could have

reduced their investment in engineering hours from 3,000 to less than 100 saving over $290,000 in this one incident alone!

Summary

The characteristics that make todays enterprise applications so powerful and

productive when all goes well, make them nearly inscrutable when things go

wrong. Contemporary enterprise applications encompass millions of lines of

code and serve myriad users who interact with them in ways that cannot be

adequately tested in a lab.

Throughout the 7.2-year average lifespan of major enterprise applications,software maintenance and support costs represent between 60-80% of total

software lifecycle costs. It is so costly because the largely manual, trial-and-error-

driven approaches that prevail cannot cope with todays complex enterprise

applications.

New technology approaches, such as Identifys AppSight application problem

resolution software, are helping top organizations in the finance, retail, healthcare,

manufacturing, insurance, communications, and many other industries accelerate

the problem resolution process by eliminating the cumbersome process of problem

replication along with the fingerpointing and frustration that go with it to

improve application performance, reduce application support costs, and

enhance business service.


14/14


About Identify Software

Identify Software, the pioneer and leader in application support software is

driven to help enterprises and software vendors accelerate the application problem

resolution process to optimize business critical application support processes.

Hundreds of customers rely on Identifys application support solutions to reduce

application support costs and improve business service. Founded in 1996,

Identify is a global organization, with Operational Headquarters in Raleigh,

North Carolina, and offices throughout the U.S., Europe, and Asia Pacific.

For more information about Identify, please visit www.identify.com or

call 800.364.5467.

Identify is a registered trademark and AppSight is a trademark of Identify Software Ltd. All other names contained herein are the

marks of their respective holders.

Documents

Identify 3 Case Studies Whitepaper