View
219
Download
4
Category
Tags:
Preview:
Citation preview
Ian M. Harper, CISA, CISM, CISSPChief Technology OfficerIan.Harper@twminc.com
Dealing with Your IT Process Failures
Improving Security with Root Cause Analysis
Lisa Johnson, CPA, CISA, CISM, CITP
Lisa.johnnson@twminc.com
Agenda
• Control Environment Survey• Risk Environment Survey• Control Weaknesses as Critical Process Failure• Root Cause Analysis Tools & Techniques• General Application of Root Cause Analysis to
Control Weaknesses• Common Pitfalls• War Stories
The Controls Equation
Increased complexity of control drivers
Increased complexity of systems
Dramatically increased complexity of IA controls environment
+
Explosion of IA Drivers
• General Business• US Sarbanes-Oxley (SOX)• EU 8th Company Law Directive• Japanese Financial Instruments and Exchange Law• Australia’s Corporate Law Economic Reform Program
• Specific Market Regulations• US Gramm-Leach-Bliley (GLB)• EC Markets in Financial Services Directive (MiFID)• US Health Insurance Portability and Accountability Act (HIPAA)
• Market Forces (the real driver)• Protection of business, shareholders and other market players’
interests per best practices
OrganizationalControl Frameworks
• May be home-grown or based off of internationally recognized frameworks• COSO, COBIT, etc.
• From the perspective of RCA, all implemented control frameworks are generally equally valid• Unless the framework is the cause for the
weakness!
The Role of Risk
• A necessary evil• "If the primary aim of a captain were to preserve his
ship, he would keep it in port forever." –Thomas Aquinas
• "Risk is what separates the good part of life from the tedium." –John “Johnny Zero” Foley
• But one that must be controlled• “From others' slips some profit from one's self to
gain.” –Publius Terentius Afer
Risk LifecycleU
NK
NO
WN
RIS
KA
CC
EP
TE
D R
ISK
IDENTIFICATION OF RISK
Concept Plans Components
InitialSystem
OperationalSystem
Initial RiskAssessment
RiskAssessment
Developmental ST&E
Inspection & Acceptance
Certification
Mo
nito
ring
&P
eriod
ic As
se
ssm
en
t
SYSTEM DEVELOPMENT
What Risks?
• Financial – risks that directly cost us money• Strategic – risks that affect our ability to meet our
various goals and objectives• Operational – risks that introduce inefficiencies in
achieving our goals• Compliance – risks that
may put us in jail or may cost us in fines
• Reputation – risks that undercut the trust in us held by others
Control WeaknessIdentification
• Risk Management Process• Allow us to fix before the weakness is
exploited
• Incident• Pressure-cooker environment• Personal and business credibility is at
stake
ControlWeaknesses
• Identification of presence of unacceptable risk
• Critical Process Failure (Key Point)• In manufacturing: a defect• Ultimately, a quality issue in the
manufacture of controls
“Weakness”in Context
Control Framework
Controls
Business desires to produce quality controls
Continued commitment to manage risks
Weakness Control framework or controls have failed (execution or quality)
Object Denotes
Control WeaknessObservations
• Usually occur in batches• Typically affect various areas of the
organization• Pareto Principle
• 80% of cost of control issues can be attributed to 20% of the total population of defective controls
Key Point: These observations typically indicate systemic causes
Root Cause Analysis
Enter Root Cause Analysis
Root Cause Analysis
• Definition:• A structured approach to identify factors causing
an undesirable event• Goal is to prevent recurrence
• A toolbox of tools and techniques tailored to a case, not a rote process• Each team/practitioner will use different
techniques and tools• Same team/practitioner may use different
approaches on different cases
RCA Roots &Applicability
• Manufacturing• Focus on quality production processes• Causes related to workforce, materials,
methods, and machinery• IT
• Need to view control framework as an assembly line
• Have to shift focus to those aspects that deal with IT and with virtual objects
• Machinery is never the cause
RCA is Not New to IT
• Auditors, remember the Cause statement of standard findings?
• So why don’t we do this in practice?• Blame game and rejection of findings by
management• Lack of resources and return on investment• Environment that does not promote creativity
RCA Stages
• Information Gathering & Problem Identification
• Process Analysis and Problem Understanding
• Brainstorming• Data Collection• Root Cause Determination• Root Cause Elimination
Information Gathering/Problem Identification
• Goals:• Gather information about the observed
weakness• Identify the actual problem
• Tools & Techniques:• Timeline
• Used throughout the RCA effort• Needs to identify pertinent data as a reference
point
Information Gathering/Problem Identification
• Tools & Techniques (continued):• WWWWH
• What? equipment involved, what was wrong.
• Who? who took what action, may be unknown
• When? day, date, time of occurrences, SDLC
stage, etc.• Where? physical, logical locations involved • How? rate of occurrence, scope of effect
Information Gathering/Problem Identification
• Concerns:• Integrity starts here• Data needs to be attributable• Data preservation is key here• Identified problem must be accepted by the
team and management• Problem definition should not jump to
conclusions (avoids “no” or “lack of” statements)
Process Analysis/Problem Understanding
• Goals:• Understand the basic function of applicable
business processes and controls• Understand the relationship of the problem
to processes and controls• Tools & Techniques:
• Document review• Interview of personnel
• Need to focus on developing rapport and furthering openness
Process Analysis/Problem Understanding
• Concerns:• Continued need to preserve data• Need to be resourceful in finding sources
of information• Trouble tickets, security plans, development
artifacts, audit logs, helpdesk statistics, etc.• Need to focus on developing rapport and
furthering openness of the process• Remember people develop personal worth from
their work
Brainstorming
• Goals:• Identify the full set of possible causes for the
defined problem based on information obtained• Prioritize possible causes for further review
• Tools & Techniques:• Unstructured Brainstorm Session
• Effective in symbiotic teams• All personnel provide possible causes while one member
records ideas• Trades risk of domination for more freedom
Brainstorming
• Tools & Techniques (continued):• Structured Brainstorm Session
• Effective in stratified teams (personality or position)
• Round Robin – Each individual gets a chance to provide ideas in turn
• Secret Ballot – Ideas are turned in on chits secretly
• Trades uninhibited expression for fairness of process
Brainstorming
• Tools & Techniques (continued):• Comparison Tables
• Used to resolve conflicts on prioritization• Priority of each possible cause is voted on by
each team member• The cause with the most votes tallied is
prioritized over the least number of votes• Can only be used for small sets of data
Comparison Table Example
– 3 causes, 10 team members– Resulting Prioritization: A, C, B
A
A/B 7
A/C 6
B
3
C
4
B/C 2 8
Total 13 5 12
Brainstorming
• Concerns:• At this point do not attempt to limit factors
based on feasibility or data collected to this point
• Prioritization is key to making sure resources are spent wisely
• Prioritization should focus on the best fit of the causes to the observed effect
• The team must be able to deal with strong personalities or perceived positions of power
Data Collection
• Goal:• Obtain any additional information needed to document
applicability of possible causes
• Tools & Techniques:• Interview and Document Review• Physical Inspection
• Concerns:• This step may not be needed if all necessary data was
collected in Information Gathering• Corroborative information may be necessary to assess
potential causes identified during Brainstorming
Determination of Root Causes
• Goal:• Organize and analyze potential causes to
identify the systemic root causes of an effect
Determination of Root Causes
• Tools & Techniques• 5-Whys
• A technique used by Toyota Motor Corporation• Based on the concept that by answering why 5
times, you get to the cause (may take less)• Prone to rote memorization of rationalized
causes, so need to ensure proper mindset• Also limited to the set of knowledge of the
questioner
Determination of Root Causes
• Tools & Techniques (continued)• Pareto Charting
• Based on 80/20 rule• Provides a visual way to distinguish the “vital
few” from the “important many”• Need to collect the occurrence of potential
causes related to an effect• Usually more effective with larger sets of data
Pareto Chart(Example)
• Potential Causes A and B are the “vital few”
Example Pareto Chart
0
0.1
0.2
0.3
0.4
0.5
0.6
A B C D E
Potential Cause
Oc
cu
ren
ce
(%
)
0
0.2
0.4
0.6
0.8
1
Cu
mu
lati
ve
P
erc
en
tag
ePercentage Cumulative
Determination of Root Causes
• Tools & Techniques (continued)• Fishbone Diagramming
• Based on Kaoru Ishikawa’s approach• Provides a graphical representation of causes in a
causal factor chain• Uses cause groupings for default causes:
• Policy – Unclear or non-existent policies• Procedure – Lack of sufficient, clear procedures• Organization – Ineffective business organization• People/Culture – Performance of personnel or groups
• Machines are not causes in IA control failures• Acceptance is an issue within other process groupings
Fishbone Diagram (Example)
Weak Passwords
Procedure
Password procedures not implemented
Procedures draftedbut not approved Password management
system being revamped
Organization People/Culture
Policy
Determination of Root Causes
• Tools & Techniques (continued)• Event Causal Factor Chart
• Based on Max Ammerman’s approach• Provides identification of potential and root
causes based on a workflow methodology• Standard charting methodology
• Circles – Start and end conditions• Rectangles – correctly-functioning events• Diamonds – Incorrectly functioning events• Light Ovals – Secondary causes• Dark Ovals – Root causes
Event Causal FactorChart (Example)
Baseline Established
XYZ Application Down 1/1/05
Helpdesk Creates
Trouble Ticket
System Moved to Production
Helpdesk Assigns TroubleTicket to Admin
Change NotDocumented
Admin FindsProblem/
Applies Fix
Helpdesk Contacts Admin
To Close
HelpdeskCloses Trouble
Ticket
Baseline Differs FromProduction
InsufficientResources
Helpdesk Not Trained
ElevatedHelpdesk Privileges
Admin Reassigned
Helpdesk Avoiding
Notification
Determination of Root Causes
• Concerns:• Go beyond the “pilot error” knee-jerk
defense• Make sure you don’t stop until fundamental
causes are identified• Be creative in the techniques used, use the
techniques that are most effective• “To thine own self be true”
Root Cause Elimination
• Goals:• Suggest controls or changes to existing
controls to eliminate the root cause of undesired events
• Identify metrics to ensure that proposed recommendations resolve root causes within the field
Root Cause Elimination
• Tools & Techniques:• Prioritized potential fixes are identified with effort
represented with all costs• For each root cause, metrics are defined to allow
measurement of the effectiveness of an implementation
• The findings of analysis and recommended fixes are provided to management for decision
• A champion is identified to pilot the implementation of recommended actions
Root CauseElimination
• Concerns:• Effectiveness should be provided as a
function of return cost of future events against up-front and ongoing costs for each recommendation
• Credibility of the team in front of management is key
• Typically the best champion is a member of the RCA team with sufficient organizational power to effect change
Common Pitfalls
• Human nature• Team dynamics
• Lack of organizational knowledge• Lack of necessary skills (technical and professional)• Lack of team credibility
• Organizational dynamics• “Gotcha” mentality• Blame game/personnel-rating• Insufficient resources• “Drive-by” RCA
War Stories
Conclusions
• IT organizations should consider effectiveness of IA controls as a key measure of quality for the processes used to develop or acquire systems
• Root cause analysis is directly applicable to and can provide great benefits to IT organizations, and specifically within the IA controls environment
Recommended