Toxic Applications - An Actionable Primer

8/14/2019 Toxic Applications - An Actionable Primer

http://slidepdf.com/reader/full/toxic-applications-an-actionable-primer 1/13

White Paper

February 2009

Toxic Applications:

Defusing the Ticking Timebombs

in Your Mission-Critical Systems

An Actionable Primer

for Business Executives

Some toxic applications can get the attention of CNN,

but the more damaging and insidious impact is that which

drags down day-to-day business performance. This paper

explains how core business applications can become toxic

over time and what IT executives can do about it.

A CH IE VE I NS IGH T . D E L I V ER E XCE

Dr. Bill CurtisSenior Vice President and Chief Scientist, CAST

Co-author of the original CMM standard at SEI

and a preeminent authority on IT software quality



Toxic Applications: Defusing the Ticking Timebombs


An Actionable Primer for Business Executives

Page 2

Contents

I. Introduction

II. Why is Application Quality

a Business Issue?

III. Flying Blind

IV. Pay Now, and Pay a Lot

More Later

V. It’s About Time for Business

Executives to Own

Software Quality

Risk o IT ailure can seriouly

damage a company’s reputation

and proftability.

Executive Summary

A recent problem in the AirTran ight dispatch system caused massive ight

cancellation and delays – over 100 ights were delayed and more than 10,00

passengers ound their plans suddenly disrupted.

On November 8, 2008, Fox News reported a worldwide ATM scam that swind

$9 million and possibly jeopardized sensitive inormation rom people around

the world. The computer system or a company called RBS WorldPay washacked. ATMs rom 49 cities were hit simultaneously – including Atlanta,

Chicago, New York, Montreal, Moscow, and Hong Kong.

These are just two o a number o recent, highly-publicized ailures o busine

critical systems – ailures that can wipe out your hard-earned credibility in th

blink o an eye.

The cause o such disasters? Toxic applications. Every business is laced with

toxic applications – time bombs hidden deep inside mission-critical applicat

that are just waiting to explode, oten when least expected.

Toxic applications should be every C-level executive’s worst nightmare, but

are even aware o them or the dangers they pose. Yet, C-level executives are

ones held accountable to stockholders, regulators, and customers when these

time bombs explode and destroy business value.

This paper introduces the concept o toxic applications and explains why C-l

executives need to pay attention to them now. It provides actionable guidelin

or engaging in productive dialogue with IT management to make the right ris

cost-beneft tradeos or signifcantly improving the business value o missio

critical applications.

CIOs can (and will) spend buckets o your business dollars improving the

quality o mission-critical applications. But how much is quality worth, and

how much is enough? You will never know i you don’t unearth and quantiy t

business impact o your toxic applications and the business risks that lurk de

within them.



Highlights




Page 3

1. Coming to grips with IT risks - A report rom Economic Intelligence Unit, 2007

I. Introduction

While there is no precise defnition o ‘toxic applications’, every business

and IT executive has seen them. They are the applications that crash the

corporate website, suer outages during peak business hours, produce corrup

data in fnancial reports, yield confdential customer data to hackers, and res

the very enhancements that are required to compete in ast-moving markets.

In short, they are the applications that make ront page news and expose

business executives to unriendly questions rom the press and the Boardo Directors. These are the applications that get IT executives invited to the

CEO’s ofce or heart-to-heart conversations about the risks to which IT

has exposed the business.

With IT at the heart o every modern enterprise, is it any wonder that most

business executives believe IT ailures are their greatest risk – ahead

o terrorism, natural disaster, fnancial risk, or regulatory constraints (1)?

Many business executives see IT applications as big expensive black boxes

– inexible, unpredictable, complicated, and too oten, deective. These

applications orce managers to place huge bets into which they have no visiband over which they have no control. Without measuring the internal quality

a business application, its risks to the business remain hidden behind innocu

status reports that ail to reveal the dangers that lie within.

Historically, the risks o toxic applications were difcult to quantiy because

their origins were shrouded in the arcane languages o programmers. Those d

are over. Not only can the risks in an IT application be precisely identifed an

quantifed, but it is incumbent or IT and business executives to take proactiv

steps to mitigate these risks. Tackling the hidden but lie-threatening risks to

business is the next rontier in management maturity.

This paper is intended or all C-level executives, not just CIOs and CTOs wh

get paid or managing the ugly tangle o IT plumbing that supports the busine

and generates business risk. It introduces the concept o toxic applications a

explains why C-level executives need to pay attention to them now.

It goes on to provide actionable guidelines or engaging in productive dialogu

with IT management to make the right risk-cost-beneft tradeos or signifca

improving the business value o mission-critical applications.

Without measuring the internal

quality o a business application,

its risks to the business remain

hidden behind innocuous status

reports that ail to reveal the

dangers that lie within.



Highlights




Page 4

II. Why is Application Quality a Business Issue?

Certainly, CIOs and business executives should not have to deal with source

code. Yet, the challenge o managing risk and total cost o ownership (TCO) i

business issue that must be managed top down and driven by the business.

Application quality is not just a technical concern. Ensuring the quality o

work delivered by the development teams, the service providers, the architec

and even the CTO’s ofce itsel, is not merely a technology challenge. Takingownership o team perormance, with respect to key perormance indicators

(KPIs) such as risk and quality, and all the related personnel and political

issues, is not something that IT executives can delegate to anyone.

What gets measured gets done. In IT, application managers ocus on

requirements, schedules, and budgets. They don’t have the means to manage

the quality o the sotware produced by their teams. Consequently application

quality depends on the knowledge, skill, and experience o developers –

attributes that have been proven to vary by more than 20-1 across developers

Business managers need greater control over risks than the chance assignme

o talented developers to their critical initiatives. Unless application quality i

measured and managed, it will not receive consistent priority.

It’s better to fx problems at the root. It is much more eective to identiy

and attack the root causes o problems that put your business at risk. There i

a direct causal connection between the quality o source code at the heart o

mission-critical applications and the number and size o the business outage

they create.

Reactive measures do not reduce the damage. Managing application quali

proactively helps avoid the scramble and the cost o handling it reactively.Firing application managers, unleashing an army o lawyers on suppliers,

or outsourcing development do nothing to reduce the damage done by toxic

applications. These actions are the all-too-requent results o ailing to

proactively manage application quality.

There is a direct causal

connection between the quality

o source code at the heart o

mission-critical applications

and the number and size o the

business outages they create.

Architectural and code

deects in sotware

development are the root

causes o most o the damage

done to the business.



Highlights




Page 5

III. Flying Blind

Toxic applications are like toxic fnancial derivatives. These derivatives were

concocted rom large batches o loans, many o which were so risky they coul

never be repaid. Once hidden inside certain fnancial derivatives, the risks o

these atal loans become invisible. Without visibility into the risks hidden in

derivatives, there is no way to evaluate the impact o these risks, and hence,

no way to price these fnancial instruments. These risks were ticking

time bombs hidden deep within the system; we now know the damage theycaused when they blew up. It’s the same with the risks that lie hidden within

toxic applications.

Shockingly, ew IT executives have any more visibility than their business

partners into the great risks hidden in these toxic applications. Why?

For three reasons:

1. Most mission-critical applications are a moving target. They are constan

being modifed and enhanced to serve new business needs, growing

unstoppably larger and more complex by the day.

2. Most mission-critical applications are multi-language, multi-tier, and

multi-platorm. It is impossible or anyone to have an end-to-end view o

such applications.

3. Most mission-critical applications are built by geographically-dispersed

teams, oten working or dierent companies. Most executives have no

objective measure o how these multiple pieces will perorm when they

come together in the production environment.

Why Measuring Quality is Hard. Measuring sotware quality to identiy toxic

applications is difcult because modern business systems are made up o

millions o instructions, written in multiple programming languages, using a

complex data model that is controlled by hundreds o business rules. Even

worse, these applications are oten enhanced over many years by dierent

teams working or several dierent suppliers and with signifcant turnover.

There is no single mind that can understand it all, and ew design decisions

are ever recorded – even the brightest and highly motivated struggle to maste

these giant puzzles.

Toxic applications are like toxic

fnancial derivatives.

Shockingly, ew IT executives

have any more visibility than

their business partners into

the great risks hidden in these

toxic applications.



Highlights




Page 6

Figure: Mission-Critical Applications: Multi-Tier, Multi-Platform, and Multi-Language

The picture above illustrates the complex web o interactions that characteriz

typical mission-critical applications. These interactions thread between syste

components that manage the user interace, the business logic that controls

transactions, the database that houses complex orms o inormation, and the

sotware that manages how these components interact with other enterprise o

legacy applications that have their own interace, business logic, and databas

components. Is it any surprise that 50% o the eort spent trying to change a

business application is spent trying to fgure out what is going on in the syste

and how it is connected2? Failure to manage all these interactions produces

nightmarish side eects such as outages and degraded perormance, the caus

o which lie hidden across a vast tangle o computer instructions.

The most serious application risks are those hidden in awed interactions

among dierent components and sub-modules that developers cannot see eve

ater they result in outages, degraded perormance, corrupt data, or all victimto hacker intrusions. Most Quality Assurance teams3 ocus exclusively on the

external quality o an application; that is whether the unctional aspects o th

application have been programmed correctly. While it’s clearly essential to

deliver what the business needs, it is equally important that the application

perorm with minimal business risk and be easy to change to meet pressing

business needs.

Most mission-critical

applications are a moving

target, growing unstoppably

larger and more complex by

the day.

Enterprise Applications Middleware Web/Client Server Applications ASP/JSP/VB/.NET

Batch

Shell Scripts

CICS

Connector

Web

Services

FilesDatabases

COBOL

Database

Application LogicJava, C++, …

Frameworks Struts MVC, Spring

Data Management LayerEJB - Hibernate - Ibatis

Legacy Applications

CICS Monitor (Cobol)Tuxedo Monitor (C)

!

!

!

2. Pro. Mordechai Ben-Manachem - Sotware Quality, Producing Practical and Consistent Sotware

3. IT teams in charge o unctional, system, and load testing. System testing is the process o executing a program

or application with the intent o fnding sotware bugs. Load testing being the process o putting demand on

a system or device and measuring its response.



Highlights




Page 7

Functional evaluation is like ensuring that the rooms and urnishings in a new

house are consistent with the architectural drawings. However, it does not en

that the house was ramed correctly, that the wiring was properly insulated, o

that the plumbing joints were properly sealed – hidden internal risks that wil

bring a house to ruin.

IV. Pay Now, and Pay a Lot More Later

Most business executives frst become interested in application qualitywhile struggling to answer questions rom the media such as, “What caused

this business disaster?”, “How long will it take to get your business back to

normal?”, and “Can you ensure this will never happen again?” Here is

a sampling o incidents where poor application quality translated into large

business losses and lots o unwanted press.

•In1999,a$112MERPsystemimplementationfailureatHersheyFoodsl

to massive distribution problems and a 27% loss o market share.

•InMarch2003,asoftwaredefectcaused4,700KaiserPermanentepatien

to get the wrong medications.

•InJuly2003,OrbitzLLC,aleadinginternettravelreservationsite,suffe

a24houroutagebecauseofadatabaseproblem.

•InJune2004,RoyalBankofCanadafellbehindinprocessingsalary

depositsforthousandsofCanadianworkersasmillionsoftransactionsw

delayed by a computer glitch.

•InDecember2004,aMizuhoemployeeaccidentlytraded610,000shares

ofastockat¥1ratherthan1shareat¥610becauseofafaulty‘CANCEL

commandinsoftwareattheTokyoStockExchange.Lossestotaled$331M

andtheresignationsoftopofcialsatbothMizuhoandtheTokyo

StockExchange.

• In2005,theUKInlandRevenueproducedtaxpaymentoverpayments

of$3.45Bbecauseofsoftwareerrors.

• InApril2007,ResearchInMotionexperiencedamassiveoutageinits

Blackberrye-mailserviceduetoqualityproblemsinasoftwareroutine

that was not believed to be capable o impacting the service.

•InNovember2007,passengersformedhalf-milequeuesatterminals

afteraglitchinAirCanada’scomputerreservationsystemdelayed

fights worldwide.

The problems that cause

outages usually do not show

themselves during testing since

it takes the growing load o

business transactions to push

the application over the edge.



Highlights




Page 8

In order to understand and control such risks managers need to understand th

source o the risks. IT risks can be classifed broadly into two categories (a)

operational risks – those that have an immediate impact on business operatio

and (b) project risks – those that have an impact on the IT organization, and

adversely aect the business in the long term.

a) Operational risks. There are fve types o risks that the business aces rom

poorly engineered business applications. Each risk has its unique consequenthat can result in a toxic application i not addressed in time.

1. Outages - system outages are requently caused by the system

becoming overloaded with repetitive tasks and shutting down. The

problems that cause outages usually do not show themselves during

testing since it takes the growing load o business transactions to p

the application over the edge. The business loss rom outages begin

with the lost revenue rom dropped or incomplete transactions, and

oten includes the cost o lost customers.

2. Degraded perormance - While degraded perormance may be

indication o an impending outage, the system may continue to trud

along growing slower and slower with each increase in the volume

o data it must process. Some perormance issues will be uncovered

during load testing, but many others will appear only in the comple

production environment, as data volumes and system usage sudden

peak. Degraded perormance escalates maintenance costs, drains t

productivity o IT teams, reduces business productivity, and rustra

customers.

3. Erratic behavior - Although the unctionality in the interace mabe correct, there may be inconsistencies, mistakes, and unintended

side eects in the way dierent developers construct Graphical Us

Interace (GUI) screens. These problems may only become visible

when users begin interacting with the application in ways that were

never anticipated during development and test cycles. When input

are lost or compromised, customers come to distrust the system and

ultimately the company behind it; internal business users suer los

productivity.

Degraded perormance

escalates maintenance costs,

drains the productivity o

IT teams, reduces business

productivity, and rustrates

customers.



Highlights




Page 9

4. Data corruption - The frst person to detect data corruption is ot

a business customer spotting inconsistencies in sensitive business

documents. Data corruption oten occurs because developers do no

adhere to the rules that speciy how their components should intera

with the database. As a result, database records are updated withou

the appropriate coordination or control, leading to weeks o lost

transactions data and countless more weeks spent on re-entering

lost transactions.

5. Security breaches - Nothing damages a company’s reputation as

than security exploits that enable hackers to access critical busines

inormation. Vulnerabilities to such attacks typically do not show u

in testing.

b) Project Risks. Risk rom bad construction can cause damage even before t

application goes into production. There are scores o statistics and examples

showing huge wastes o money and time during project development.

•In2005,after5yearsand$104Mspent,theU.S.DepartmentofJustice

InspectorGeneralreportedthe$170MFBIVirtualCaseFileprojecttobe

failure.Overone18-monthperiodduringthecourseoftheproject,theF

gaveitscontractornearly400requirementschanges!

• In2005,BritishfoodretailerJSainsburyPlchadtowriteoff$526Minve

in an automated supply chain management system that never worked.

•InOctober2004,AvisEuropewrotedown€45Mduetoproblemswiththe

newERPsystem.Developmentwashaltedbecauseofdelaysandhigher

due to implementation and design problems.

•In1992AmericanAirlineswrotedown$165MwhenitcancelleditsAfrm

projectduetopoorlyconstructedsoftwarethatcouldnothandlethe

anticipated load o reservation transactions.

Such ailures are rooted in IT management’s inability to get control over prod

quality, especially when the IT system is very complex.

Risks rom bad construction can

cause damage even before an

application goes into production.



Highlights




Page 10

Consider the ollowing cascade o unortunate events resulting rom project ri

that are characteristic o toxic applications.

1. Unplanned Eort: Perormance lapses and other types o quality

problems surace in system tests, requiring substantial unplanned

eort to fx beore the application can be released.

2. Delayed Time to Market: The application is delivered late to thbusiness, orcing business managers to alter their business plans a

destroying the project’s ROI.

3. Operating Cost Overruns: The system is fnally delivered but

still has undetected quality problems that will not become apparen

until the volume o transactions and data grow past the threshold th

was tested. When this happens, budget and resources will have to b

diverted rom other projects to fx the application.

4. Loss o Business Agility: As application size increases due to

continual modifcations and enhancements, its complexity grows, athe quality o its architecture degrades. Growing complexity orces

developers to spend more time understanding the system beore

modiying or enhancing it, driving up the cost o maintaining the

application while dramatically slowing the pace at which additiona

unctionality can be delivered to the business.

5. Stifed Innovation: As the cost o maintaining existing application

increases due to complexity and poor internal quality, the organizatio

has ewer resources, both fnancial and human, to invest in creating

business systems and capabilities. Business innovation and renewal

crippled, putting the business at a competitive disadvantage.

As application size increases

due to continual modifcations

and enhancements, its

complexity grows, and

the quality o its

architecture degrades.



Highlights




Page 11

These fnal points are illustrated in the fgure below. As the internal quality o

applications degrade over time, the cost o Requests or Change (RFCs) grow

dramatically, absorbing a disproportionate amount o the IT budget.

One reaction to this problem has been to purchase packaged applications. Bu

this “build” problem cannot be tamed with a “buy” solution. Commercial O

The Shel (COTS) Enterprise Resource Planning (ERP), Customer Relationsh

Management (CRM), or Human Capital Management (HCM) systems requirea great deal o customization. The risks introduced by extensive customizatio

quickly turn enterprise packages into toxic applications.

Figure: Innovation Killed by Runaway Customization Due to Requests for Change (RFC

Unortunately, the risks o poor internal quality cannot be outsourced or

packaged away. I not addressed, they wreak havoc throughout the lie cycle

an application.

The risks introduced by

extensive customization quickly

turn enterprise packages into

toxic applications.

Year 1 Year 2

Base Line Costs

RFC Costs

Uncontrolled RFC Costs

Year 3 Year 4 Year …

Time

C o s t s



Highlights




Page 12

In essence, we must accelerate

the pace o adding new

unctionality to an object

that already is growing more

complex daily – and do it or less

money each quarter.

Like it or not, as a business

executive, you are accountable

or the perormance o IT

applications that enable

your mission-critical

business processes.

V. It’s Time for Business Executives to Own Software Quality

Clearly, mission-critical applications come with risks that have sizeable busi

consequences. The conditions that produce these risks are growing steadily

worse as two trends collide. First, business applications are growing larger an

more complex by an order o magnitude every decade. Second, greater agility

increasingly required to compete in ast-moving markets.

In essence, we must accelerate the pace o adding new unctionality to anobject that already is growing more complex daily – and do it or less money

each quarter. These are exactly the conditions that lead to the types o disast

cited above. Since we cannot control the pace o markets, we must control the

internal quality o critical business applications so that the pace and quality

sotware development can scale with the size and complexity o our systems.

Like it or not, as a business executive, you are accountable or the perorman

o IT applications that enable your mission-critical business processes. Here

three concrete steps you must take to prevent the toxic application buildup th

destroys business value.

• First,demandthatITexecutivesandkeystakeholdersregularlymeasur

and report the perormance risks that are deeply embedded in your missi

critical applications.

• Second , insist on a clear and detailed plan to mitigate these risks.

• Third , use this risk inormation as the oundation o a continuing dialogu

withITexecutivesandkeystakeholdersaboutthefutureofapplications

support your business.

CIOs can (and will) spend buckets o your business dollars improving the

quality o mission-critical applications. Buthowmuchisqualityworth,ahow much is enough? You will never know i you don’t unearth and quanti

the business impact o your toxic applications and the business risks that lur

deep within them.



Bill Curtis is an industry luminary who is responsible or

inuencing CAST’s scientifc and strategic direction, as well

as helping CAST educate the IT market to the importance o

managing and measuring the quality o its sotware. He is best

known or leading the development o the Capability Maturity

Model (CMM) which has become the global standard or evaluating

the capability o sotware development organizations.

Prior to joining CAST, Dr. Curtis was a Co-Founder o TeraQuest,

the global leader in CMM-based services, which was acquired

by Borland. Prior to TeraQuest, he directed the Sotware

Process Program at the Sotware Engineering Institute (SEI)

at Carnegie Mellon University. Prior to the SEI he directed

research on intelligent user interace technology and the sotware

design process at MCC, the fth generation computer research

consortium in Austin, Texas. Beore MCC he developed a sotware

productivity and quality measurement system or ITT, managed

research on sotware practices and metrics at GE Space Division,and taught statistics at the University o Washington.

Dr. Curtis holds a Ph.D. rom Texas Christian University, an M.A.

rom the University o Texas, and a B.A. rom Eckerd College. He

was recently elected a Fellow o the Institute o Electrical and

Electronics Engineers or his contributions to sotware process

improvement and measurement. In his ree time Dr. Curtis enjoys

traveling, writing, photography, helping with his daughter’s

homework, and University o Texas ootball.

Dr. Bill Curtis

Senior Vice President and Chief Scientist

www.castsoftware.com

CAST Headquarters

North America: +1 212-871-8330

Europe: +33 1 46 90 21 00

About CAST

CAST’s unique technology is the result ofmore than $70 million in R&D investment.Top engineering talent, dedicated to buildingthe best technology for assessing complexapplications and their internal quality, has madeCAST the leader in Automated ApplicationIntelligence. CAST’s mission is to transformapplication development from a complex andobscure world, into one that’s transparent,driven by data, performance and operationalexcellence.

Founded in 1990, CAST has helped more than650 organizations worldwide speed IT deliveryto the business, mitigate risks in production,improve customer experience, and reducethe total cost of application ownership.CAST is listed on NYSE-Euronext (Euronext:CAS) and serves Global 2000 organizationsworldwide with a global network of locationsin the US and Europe.

Documents

Toxic Applications - An Actionable Primer