Upload
others
View
0
Download
0
Embed Size (px)
Citation preview
IT Risk Management –A Causal Modeling Approach forEnd-User Computing
Daniel J. Hinz
IWI Jour fixeFrankfurt/Main, February 14th, 2006
1
AGENDA
• Motivation and research history
• Theoretical foundation– IT infrastructure continues to be a hot topic for more than 15 years now– Surprisingly, research neglects the management of inherent infrastructure risks– Financial risk management theory suggests the use of causal modeling
approaches
• Study design– A single case study was used to create a design-oriented artifact– For desktop availability, experts identified and estimated a Bayesian Belief
Network (BBN)– The validation with historical loss data resulted in adaptions
• Key findings– Implications for decision making: user incidents are a sensitive figure– Findings can be efficiently communicated by a balanced scorecard
• Contributions, limitations, and further research
2
AGENDA
• Motivation and research history
• Theoretical foundation– IT infrastructure continues to be a hot topic for more than 15 years now– Surprisingly, research neglects the management of inherent infrastructure risks– Financial risk management theory suggests the use of causal modeling
approaches
• Study design– A single case study was used to create a design-oriented artifact– For desktop availability, experts identified and estimated a Bayesian Belief
Network (BBN)– The validation with historical loss data resulted in adaptions
• Key findings– Implications for decision making: user incidents are a sensitive figure– Findings can be efficiently communicated by a balanced scorecard
• Contributions, limitations, and further research
3
A PRACTITIONER'S PROBLEM …
Situation
• An international logistics company has an outsourced ITinfrastructure of end-user computers, servers, and network
• During contract renegotiation, end-user computers are hotly debated
– Major structural changes are the foundation to improve reliabilityand to reduce costs
– Corresponding tightened SLA figures and objectives werenegotiated
• Currently, transformation still in progress and SLA figures are notfully met
Who is responsible?Are fundamental assumptions of the
new contract correct?
4
?
… IS ADDRESSED BY MY RESEARCH QUEST(ION)
How can risks arisingfrom IT infrastructure beeffectively assessed andcommunicated?
Assessment
Main question
Sub questions
Communication
• How to identify IT risks in a structured way
• How to measure or predict the risk potential(e.g., downtime)
– In steady state (normal operation)
– In change scenarios (e.g., outsourcing,contract renegotiation)
• How to facilitate decision making bycommunicating IT risks and the potentialeffects of mitigitation measures effectively
• How to control risk mitigation strategiesthrough effective reporting
5
THESIS STRUCTURE AND PEER REVIEW PLAN
Introduction/Motivation
Application for IT managers• Communication and management
of IT risks with BSC
• Risk mitigation strategies
• Financial riskmanagementdomain
• IT manage-ment theory
• DSS
Causal modeling of IT risks• Development of classification
model for operational risk• Identification of key risk drivers
and dependencies• Modelling of Bayesian Belief
Network for PC desktop risks• Empirical validation (single case
study)
Outlook and further research
Thesis structure
Theoretical foundation
* Part of T-Systems agenda ** Best paper nomination
• HICSS-39: "Enhancing the Prognostic Power of IT BalancedScorecards with Bayesian Belief Networks" (with S. Blumenberg)**
• EFLQ 3/2005: "Management Communication of Complex RiskAssessment" (with S. Blumenberg)*
• ECIS 2006: "An Integrated Approach to Assess and Communicate ITRisks" (with Blumenberg, Weitzel)*
• IRMA 2006: "Mitigating Software Risk with Web Services"
Part ofthesisPeer review plan
• PACIS 2004: "A Framework for Classifying the Operational Risks ofOutsourcing" (with H. Gewald)
• WP: "IT risk assessment – methods and application" (withcluster 2: Pérez, Martinovic, Berbner, Steinmetz)
• IRMA 2006: "The Next Wave in IT Infrastructure Risk Management –A Causal Modeling Approach with BBNs" (with H. Gewald)
• (Journal): Empirical results
• HICSS-38: "High Severity IT Risks in Finance"• ECIS 2006: "Employing Bayesian Belief Networks for Measuring the
Operational Risk of Information Systems" (with H. Gewald)*
• GITMA 06: "IT Risks: Definition and Challenges" (part of roof paper)
• HICSS-39: "Assessing the Risks of IT Infrastructure – A PersonalNetwork Perspective" (with J. Malinowski)
�Accepted
�
�
�
�
02/06
( )� Submitted/completed
( )�
( )�
( )�
�
�
�
�
6
AGENDA
• Motivation and research history
• Theoretical foundation– IT infrastructure continues to be a hot topic for more than 15 years now– Surprisingly, research neglects the management of inherent infrastructure risks– Financial risk management theory suggests the use of causal modeling
approaches
• Study design– A single case study was used to create a design-oriented artifact– For desktop availability, experts identified and estimated a Bayesian Belief
Network (BBN)– The validation with historical loss data resulted in adaptions
• Key findings– Implications for decision making: user incidents are a sensitive figure– Findings can be efficiently communicated by a balanced scorecard
• Contributions, limitations, and further research
7
IT INFRASTRUCTURE CONTINUES TO BE A HOT TOPICFOR MORE THAN 15 YEARS NOW
1990 "IT infrastructure" appears forthe first time in the top 10issues identified by the survey
1995 "Building a responsiveinfrastructure" was rankednumber one challenge
2000
2001
"IT infrastructure management"ranked third in these twoinformal surveys
2003 "Infrastructure developments"ranked second in the categoryof top application andtechnology developments
• Survey to identify themost critical issues inIS management
• Conducted regularlyby the Society forInformationManagement (SIM),supported by the MISResearch Center(MISRC)
• Among theirmembers consistingof top executives aswell as ISresearchers
SIM survey
Source: [Niederman, Brancheau, Wetherbe 1991], Brancheau, Janz, Wetherbe 1996], [Luftmann, McLean 2004]
8
SURPRISINGLY, RESEARCH NEGLECTS THEMANAGEMENT OF INHERENT INFRASTRUCTURE RISKS
Main viewpointson IT infrastructure
• Flexibility• Standardization• Security• Web Services• SOA
IT architecture
• What are requirements andfeatures of a good ITinfrastructure?
• What is the enablingtechnology?
Main research questions Main answers
IT alignment
• Reflection of firm‘s strategicobjectives
• …
• How can IT and business bealigned?
• What is the economic impact?
IT infrastructureoutsourcing
• Risk assessment• SLA management
• What is a good managementinterface to the provider?
(Internal) ITmanagementprocesses
• Practical "handbooks" (ITIL)• Which internal supportprocesses are needed?
• How can IT (risks) internallybe managed?
Focus of my research
9
Measurement
FINANCIAL RISK MANAGEMENT THEORY SUGGESTSTHE USE OF CAUSAL MODELING APPROACHES
Identification Decision making ControlGeneric (risk)managementprocess
Classical sourceof information
• Expert judgment • Analysis ofhistorical lossdata
• Parallelconsideration ofprior analyses
Literature for operational riskmanagement suggests causalmodelling with Bayesian BeliefNetworks (BBN) [Alexander 2002]• To combine expert estimations
with loss data• To identify key risk indicators and
mitigation levers• To allow for upfront simulation
Balanced Scorecard(BSC) is a powerful andwell-established method tocommunicate causal de-pendencies to top manage-ment [Van der Zee, DeJong 1999]
Suggestionsfrom financialtheory
Main tasks • Systematicallyidentify mainsources of risk
• Estimateprobabilities andloss to quantifyrisk
• Decide onmitigationmeasures
• Monitor thesuccess ofmeasures andrisk change
• Monitoring ofkey figures
Assessment Communication
10
AGENDA
• Motivation and research history
• Theoretical foundation– IT infrastructure continues to be a hot topic for more than 15 years now– Surprisingly, research neglects the management of inherent infrastructure risks– Financial risk management theory suggests the use of causal modeling
approaches
• Study design– A single case study was used to create a design-oriented artifact– For desktop availability, experts identified and estimated a Bayesian Belief
Network (BBN)– The validation with historical loss data resulted in adaptions
• Key findings– Implications for decision making: user incidents are a sensitive figure– Findings can be efficiently communicated by a balanced scorecard
• Contributions, limitations, and further research
11
A SINGLE CASE STUDY WAS USED TO CREATE ADESIGN-ORIENTED ACTIFACT
Step 1: Model building andinitialization based on expertestimation
Step 2: Model adaption basedon historical data
Resulting model
• Two subject matter expertsfrom leading IT consultancies
– One with deep knowledge ofcompany's IT and contractintentions
– One with general knowledgeof IT infrastructure
• No prior input from logisticscompany
• Iterative approach according toEisenhardt and Yin
• Incident data from company'shelpdesk systems duringmigration phase
• Enriched with individualconfiguration data for eachsystem over time
• Key figures
– App. 30,000 computers– 4 months of observation
– Over 80,000 incidents
– App. 120,000 aggregateddata sets
Typical process forconstruction of Bayesian
Belief Networks
12
FOR DESKTOP AVAILABILITY, EXPERTS IDENTIFIEDAND ESTIMATED THE FOLLOWING BBN
Desktopoperational?
Serversavailable?
Softwareoperational?
Hardwareoperational?
Useroperational?
Hardwarecomplexity
Hardwarestandardized?
SW imagestandardized?
SW imagecomplexity
# of hardwareerrors
FieldserviceTTR
# of softwareerrors
Helpdesk TTR# user
incidents
Helpdesk load
Automationtool supportNetwork up?
Servers up?
Softwarematurity
User skill level
Data available
13
Helpdesk load
Automationtool supportNetwork up?
Servers up?
Softwarematurity
User skill level
Serversavailable?
THE VALIDATION WITH HISTORICAL LOSS DATARESULTED IN ADAPTIONS
Desktopoperational?
# of hardwareerrors
FieldserviceTTR
Hardwarecomplexity
Hardwarestandardized?
# of softwareerrors
Helpdesk TTR
SW imagestandardized?
SW imagecomplexity
# userincidents
Softwareoperational?
Hardwareoperational?
Useroperational?
? ��?� ?
• Edges connecting ostensiblyindependent nodes are kept in orderto reflect the experts‘ judgement
• New edges are added to incorporatenewly identified dependencies
Counter-intuitive findings Resulting adaptions
• Notebook computers are as reliable asdesktop computers
• Standardization has no effect on errorresolution times
• Software image standardization andcomplexity obviously also drivesnumber of user incidents
14
AGENDA
• Motivation and research history
• Theoretical foundation– IT infrastructure continues to be a hot topic for more than 15 years now– Surprisingly, research neglects the management of inherent infrastructure risks– Financial risk management theory suggests the use of causal modeling
approaches
• Study design– A single case study was used to create a design-oriented artifact– For desktop availability, experts identified and estimated a Bayesian Belief
Network (BBN)– The validation with historical loss data resulted in adaptions
• Key findings– Implications for decision making: user incidents are a sensitive figure– Findings can be efficiently communicated by a balanced scorecard
• Contributions, limitations, and further research
15
IMPLICATIONS FOR DECISION MAKING:USER INCIDENTS ARE A SENSITIVE FIGURE
52
30
61
12
172
8
% of total downtime
1 89
% of total incidents
Other*
Network*
Hardware
Software
User
Special attention has to be given to userincidents, as impact on desktop uptime isminor, but number of user incidents isenormous. Further analyses (e.g., bybenchmarking) may indicate, whetherfocus should be on reduction of incidentvolume instead of TTR
4,446Network*4,229Software
3,354Other*2,720Hardware
0,335User
Average TTR in days
Only 8% of total measured downtimeis due to user incidents and furtherreduction of TTR seems difficult …
… however, more than half of allincidents are user inquiries
* Not significant (e.g., network mass problems not included)
16
CLASSICAL APPROACHES FOCUS ON "OFFICIAL"INCIDENTS, BUT THEY ARE ONLY ONE PART
SituationClassical approaches measure mostly officialincidents (e.g., helpdesk calls) [Niessink and Van Vliet, 2000]
Complication• Not all incidents are reported but
instead solved by asking co-workers orknown experts
• How can those aspects be consideredto get a more realistic number ofincidents?
17
THE DENSITY OF THE SOCIAL NETWORK HAS ANINFLUENCE ON PROBLEM SOLVING
Based on expert interviews,two measures from SocialNetwork Analsis (SNA) werechosen to influence problemsolving
Socio-Centric Density (SCD)of the network of co-workers [Barnes 1974]
)1( −=
nn
lSCD
n
lECD =
Ego-Centric Density (ECD)of the helpdesk [Scott 2000]
18
SCD AND ECD CAN BE USED TO GET A MOREREALISTIC NUMBER OF USER INCIDENTS
+⋅=
ECD
SCDCHDUI α1
U3
U1
U2 HD0.9
0.8
0.8
0.6
1.0
0.8
Calculation of user incidents:
SCD = 0.68
ECD = 0.80
� UI = 185
UI: User incidentsCHD: Calls that reach the helpdeskα: Scaling factor
The ratio of both densities can be usedto predict the unkown number of totaluser incidents (UI) from the number ofknown incidents (CHD)
Example
19
THE INITIAL MODEL HAS TO BE EXTENDED TO REFLECTTHESE FINDINGS
Helpdesk qualityand social networkdensity nowdetermine the useraction upon anincident
20
BALANCED SCORECARDS CAN BE LAYERED UPON ABAYESIAN BELIEF NETWORK DUE TO SIMILARITIES
I
PC
F
Bayesian Belief NetworkF
PC
I
Balanced Scorecard
Balanced Scorecard (BSC)
Consists of entities (called figures),grouped within perspectives
Directed edges indicate causalrelationships
Loops are allowed, but should beomitted to be compatible with BBNs
Bayesian Belief Networks (BBN)
Consists of entities (called nodes), maybe grouped graphically
Directed edges describe causalrelationships and are used to calculateconditional probabilities
Loops are not allowed (graphs has to bedirected and acyclic)
21
FINDINGS CAN BE EFFICIENTLY COMMUNICATED BY ABALANCED SCORECARD
Desktop infrastructureavailability
Percentage ofdowntime
Fieldservice TTR
Avg. time per call
Helpdesk TTR
Avg. time per call
User incidents
Number of calls
Server availability
Percentage of downtime
SW standardization
Percentage ofstandardized systems
Tar
get
Inte
rnal
Bus
ines
sP
roce
sses
Cus
tom
er(U
ser)
Ext
erna
l
HW standardization
Percentage ofstandardized systems
SW image complexity
Percentage ofcomplex images
22
AGENDA
• Motivation and research history
• Theoretical foundation– IT infrastructure continues to be a hot topic for more than 15 years now– Surprisingly, research neglects the management of inherent infrastructure risks– Financial risk management theory suggests the use of causal modeling
approaches
• Study design– A single case study was used to create a design-oriented artifact– For desktop availability, experts identified and estimated a Bayesian Belief
Network (BBN)– The validation with historical loss data resulted in adaptions
• Key findings– Implications for decision making: user incidents are a sensitive figure– Findings can be efficiently communicated by a balanced scorecard
• Contributions, limitations, and further research
23
EX-ANTE SIMULATIONS ARE ONE OF THE MAJORSTRENGTHS OF THE APPROACH
Contributions to theory Contributions to practice
• Causal modelling techniques can beapplied to the assessment of ITinfrastructure risks
• Bayesian Belief Networks and BalancedScorecards can be combined to supporta seamless and fully integrated riskmanagement process
• Users seem to be a crucial pointconcerning risk, which are currentlyneglected by researchers andpractitioners
• Simulations of the causal model help toidentify most important risk mitigationlevers
• In change scenarios like outsourcingnegotiations, they may help to agreeon key figures in the SLA
• The process itself of building andtraining the model improves riskunderstanding
Limitations and further research
• Further research has to show, whether this approach is actually better than others whenapplied in real world scenarios