IT methods

INTOSAI IT AuditINTOSAI IT Audit

IT Methods AwarenessIT Methods Awareness

OutlineOutline

• Scope• Overview• It Methods• Methods Description• Methods Usage• Audit Reporting

ScopeScope

• It Methods Described For:

– Project Selection, Control, Evaluation– Systems Development– Systems Acquisition– Enterprise Architecture Development– Security Assessment

OverviewOverview

• Methods Listed Here Are Generally Accepted in The Community

• Methods Assess or Prescribe “What” Must Be Done Not “How” to Accomplish Activity

• Methods Provide a Framework to Audit It Activity

It MethodsIt Methods

Project Selection, Control, Evaluation

Information Technology Investment Management: A Framework for Assessing and Improving Process Maturity

http://www.gao.gov/special.pubs/ai10123.pdf

Systems Development SEI Software Capability Maturity Model

http://www.sei.cmu.edu/cmm/

Systems Acquisition SEI Software Acquisition Capability Maturity Model

http://www.sei.cmu.edu/publications/documents/99.reports/99tr002/99tr002abstract.html

Enterprise Architecture Development

A Practical Guide to Federal Enterprise Architecture, Chief Information Officer Council, Version 1.0, February, 2001

http://www.itpolicy.gsa.gov/mke/archplus/group.htm

Security Assessment Information Technology Security Assessment Framework, November 28, 2000 (Security, Privacy, and Critical Infrastructure Committee)

http://www.cio.gov/docs/federal_it_security_assessment_framework.htm

Methods Description

Module 1Project Selection, Control, Evaluation

Project Selection, Control, Project Selection, Control, Evaluation Evaluation

• Wisely Managed Investments in It Can Improve Organizational Performance

• Internet and Local Area Networks Enable Data Sharing and Research

• Data Warehouse Permits Organizations to Discover Unknown Fiscal or Physical Resources


• However, Along With the Potential to Improve Organizations, It Projects Can Become Risky, Costly, Unproductive Mistakes

• In Response, Gao Developed Guidance, That Provides a Method for Evaluating and Assessing How Well an Agency Is Selecting and Managing Its It Resources


• The Select/control/evaluate Model Has Become a Central Tenet of the It Investment Management Approach


• During the Selection Phase the Organization

– Selects Those It Projects That Will Best Support Its Mission Needs and

– Identifies and Analyzes Each Project’s Risks and Returns Before Committing Significant Funds to a Project.


• During the Control Phase the

– Organization Ensures That, As Projects Develop, the Project Is Continuing to Meet Mission Needs at Expected Levels of Cost and Risk

– If the Project Is Not Meeting Expectations Steps Are Taken to Address the Deficiencies


• Lastly, During the Evaluation Phase,

– Actual Versus Expected Results Are Compared to

• Assess the Project’s Impact on Mission Performance,

• Identify Any Changes or Modifications to the Project That May Be Needed, and

• Revise the Investment Management Process Based on Lessons Learned


• Gao’s Information Technology Investment Model (Itim) Model Is Comprised of Five Stages of Maturity

• Each Stage Builds Upon the Lower Stages and Enhances the Organization’s Ability to Manage Its It Investment Stages


Five Stages Of Investment Maturity

Project Selection, Control, Project Selection, Control, EvaluationEvaluation

Progressing Through the ITIM Stages of Maturity


• Itim Is a Tool for Assessing the Maturity of an Organization

– An Itim Assessment Can Be Conducted for an Entire Organization or For One of Its Lower Divisions

– Itim Is Applicable to Organizations of Different Sizes


• Itim Allows Auditors to Assesses the Maturity of Organizations to Manage Investments

• Itim Provides a Maturity Stage or “Level” for an Organization

• Each Maturity Stage or “Level” Has Required Practices or Activities


ITIM Required Processes


• Applying the Model Requires Assessing

– Critical Processes, Such As the Processes Used to Create an It Investment Portfolio

– Core Elements, (Purpose, Organizational Commitment, Prerequisites, Activities, and Evidence of Performance)

Questions / DiscussionQuestions / Discussion

• Questions• Comments• Discussion• Etc.

Methods Description

Module 2Systems Development

Systems DevelopmentSystems Development

• Systems Development Includes Activities Such As

– Project Management, – Requirements Management,– Configuration Management – Software Development, Testing, Etc.


• Many Organizations Rely on Software-intensive Systems to Perform Their Missions

• Software Quality Is Governed by the Quality of the Processes Used To Develop the Software

• (Provide Reference)


• The Software Engineering Institute Has Developed a Number of Models That Facilitate Assessing the Maturity of Organizations Developing Software

• The Models Are Called Capability Maturity Models (Cmm)


• What Is the Cmm?– An Ordered Collection of Practices for the Acquisition,

Development or Maintenance of Systems– Ordered by “Key Process Area”– Practices Determined by the Community Through Broad

Peer Reviews– Defines the Stages Through Which Organizations Evolve As

They Improve Their Acquisition Process– Identifies Key Priorities, Goals and Activities on the Road to

Improving an Organization's Capability to Do Its Job


• The Cmm Provides a Framework for

– Identifying an Organization’s Process Strengths and Weaknesses

– Assisting an Organization Develop a Structured Plan for Process Improvement


• Who Uses the Sw-cmm?– Organizations That Develop or Maintain Products

That Contain Software

– Organizations Who Want to Improve Their Software Development Processes

– Audit Organizations Who Want to Assess the Maturity Of Organizations Developing or Maintaining Software Products


• The Cmm Is Structured Into

– Five Maturity Levels• Each Level Has Key Process Areas (Kpa)

– Each Kpa Has Goals• Goals Require Certain Activities Be Performed

• Management Provides Support and Verifies That Activities Are Being Performed



• The Five Levels Are

• 1. Initial: The Software Process Is Characterized As Ad Hoc and Few Processes Are Defined

• 2. Repeatable: Basic Project Management Processes Are Established; Improvement Activities Are Begun

• 3. Defined: Software Processes Are Documented and Standardized; All Projects Use an Approved, Tailored Version of the Organization’s Standard Software Processes


• The Five Levels Are (Contd.)

• 4. Managed/quantitative: Detailed Measures of the Software Processes, Products, and Services Are Collected; the Software Processes and Products Are Quantitatively Measured and Controlled

• 5. Optimizing: Continuous Process Improvement Is Enabled by Quantitative Feedback From the Process and From Piloting Innovative Ideas and Technologies


Software CMM Levels and KPAs


• Cmm Common Features

– Commitment To Perform– Ability To Perform– Activities– Measurement & Analysis– Verification


• Commitment To Perform

– Describes What an Organization Must Do to ‘Set the Stage’ for Process Improvement / Implementation

• Involves Establishing Policy• Assigning Responsibility


• Ability To Perform

– Describes the Preconditions That Must Be Present to Facilitate Process Improvement / Implementation

• Assignment of Duties to Groups• Providing Trained or Experienced Personnel• Ensuring Adequacy of Resources


• Activities

– Describe the Activities, Roles, and Procedures That Are Necessary to Implement the Key Process Area

• Requires Formal and Informal Planning Documents• Requires Formally Documented Procedures• Requires (Depending on Kpa) Coordination With

Other Affected Groups, Tracking Contractor Performance, Etc.


• Measurement & Analysis

– Describes the Practices That Must Be Accomplished to Enable the Group to Track the Status of the Kpa

• Effort & Funds Expended by the Project Team in Conducting Its Activities

• Tracking Their Schedule and Progress (for Developing Formal Plans, Requirements, Etc.)


• Verification

– Describes the Practices That Must Be Performed to Ensure That Project and Senior Management Oversee the Activities of the Group

• Includes Periodic or As Needed – Project Level Reviews– Senior Management Level Reviews


Example From Model



Methods Description

Module 3Systems Acquisition

Systems AcquisitionSystems Acquisition

• Systems Acquisition Includes Activities Such As

– Project Management, – Requirements Management,– Solicitation, Contractor Tracking – Evaluation, Risk Management, Etc.


• Many Organizations Rely on Software-intensive Systems to Perform Their Missions

• Organizations Have Been Increasingly Contracting Out for Software or Engineering Services


• The Software Engineering Institute Has Developed a Number of Models That Facilitate Assessing the Maturity of Organizations That Acquire Software or Systems

• The Models Are Called Capability Maturity Models (Cmm)


• Just As For Software Development There Is the Sw-cmm (or Just Cmm)

• For Assessing or Improving Acquisition Related Activities, The Sei Has Developed the Software Acquisition Capability Maturity Model (Sa-cmm)


• Who Uses The Sa-cmm?– Organizations That Acquire or Support Acquisition of

Products That Contain Software, Including Software Support and Maintenance

– Organizations That Are Responsible for Acquisition Life Cycle From Requirements Development Through System Delivery and Support

– Audit Institutions That Want To Assess How Effectively Software or Services Are Being Acquired


• The Sa-cmm Is Also Structured Into

– Five Maturity Levels• Each Level Has Key Process Areas (Kpa)

– Each Kpa Has Goals• Goals Require Certain Activities Be Performed

• Management Provides Support and Verifies That Activities Are Being Performed



• The Five Levels Are

• 1. Initial: The Software Process Is Characterized As Ad Hoc and Few Processes Are Defined

• 2. Repeatable: Basic Project Management Processes Are Established; Improvement Activities Are Begun

• 3. Defined: Software Processes Are Documented and Standardized; All Projects Use an Approved, Tailored Version of the Organization’s Standard Software Processes


• The Five Levels Are (Contd.)

• 4. Managed/quantitative: Detailed Measures of the Software Processes, Products, and Services Are Collected; the Software Processes and Products Are Quantitatively Measured and Controlled

• 5. Optimizing: Continuous Process Improvement Is Enabled by Quantitative Feedback From the Process and From Piloting Innovative Ideas and Technologies



Example From Model



Methods Description

Module 4Enterprise Architecture Development

Enterprise Architecture Enterprise Architecture DevelopmentDevelopment

• An Enterprise Architecture (Ea) Establishes the Agency-wide Roadmap to Achieve an Agency’s Mission

• Eas Are “Blueprints” for Systematically and Completely Defining an Organization’s Current (Baseline) or Desired (Target) Environment


• Eas Are Essential for Evolving Information Systems and Developing New Systems That Optimize Mission Value

• For Eas to Be Useful and Provide Business Value, Their Development, Maintenance, and Implementation Should Be Managed Effectively


• An Ea Is A Strategic Information Asset, Which Documents the Mission And, – The Information , Technology, and the

Processes Required to Perform the Mission

• An Ea Includes a Baseline Architecture, Target Architecture, and a Sequencing Plan


• Eas Typically Include

– Business or Operational Architecture– Work Processes and Locations

– Information or Data Architecture – Data or Information Needed to Perform

Business

– Technical or Systems Architecture– Technology Standards, It Systems Description


Obtain Executive

Buy-In andSupport

Establish Management

Structure and Control

Define anArchitecture

Processand Approach

Develop Baseline Enterprise

ArchitectureDevelopTarget

Enterprise Architecture

Develop theSequencing Plan

Usethe

EnterpriseArchitecture

Maintain the Enterprise Architecture

Section 3.1

Section 3.2

Section 4

Section 5

Section 5

Section 5

Section 6

Section 7

Controland

Oversight

Controland

Oversight

EA Process

Sections Refer to:

A Practical Guide To Federal Enterprise Architecture


• Obtain Executive Buy-in and Support – Ensure Agency Head Buy-in and Support– Issue an Executive Enterprise Architecture Policy – Obtain Support From Senior Executives and Business Units

• Establish Management Structure and Control– Establish a Technical Review Committee – Establish a Capital Investment Council– Establish an Ea Executive Steering Committee– Appoint Chief Architect

• Define an Architecture Process and Approach– Define the Intended Use of the Architecture– Define the Scope of the Architecture– Determine the Depth of the Architecture


• Develop the Baseline Enterprise Architecture – Collect Information– Generate Products and Populate Ea Repository

• Develop the Target Enterprise Architecture – Collect Information– Generate Products and Populate Ea Repository

• Develop the Sequencing Plan– Identify Gaps– Define and Differentiate Legacy, Migration, and New Systems– Planning the Migration


• Use the Enterprise Architecture– Integrate the Ea With Cpic and Slc Processes– Train Personnel– Establish Enforcement Processes and Procedures

• Maintain the Enterprise Architecture As the Enterprise

Evolves – Reassess the Enterprise Architecture Periodically– Manage Products to Reflect Reality



Methods Description

Module 5Security Assessment

Security AssessmentSecurity Assessment

• Information and the Systems That Process It Are Among the Most Valuable Assets of Any Organization

• Adequate Security of These Assets Is a Fundamental Management Responsibility


• Agency Must

– Ensure That Systems and Applications Provide Appropriate Confidentiality, Integrity, and Availability

– Protect Information Commensurate With the Level of Risk and Magnitude of Harm Resulting From Loss, Misuse, Unauthorized Access, or Modification


• Agencies Must

– Plan for Security– Ensure Appropriate Officials Are Assigned

Security Responsibility– Authorize (or ”Certify") System Processing

Prior to Operations and Periodically As Necessary


• The Federal It Security Assessment Framework Provides a Method for

– Determining the Current Status of Their Security Programs – Establishing a Target for Improvements Where Necessary

• The Framework May Be Used to Assess the Status of Security Controls


• The Framework Comprises Five Levels to Guide Assessments of Security Programs and Prioritization of Improvement Efforts


• Level 1 Documented Policy• Level 2 Documented Procedures• Level 3 Implemented Procedures

and Controls• Level 4 Tested and Reviewed

Procedures and Controls• Level 5 Fully Integrated

Procedures and Controls


• Level 1 of the Framework Includes– Formally Documented and Disseminated Security

Policy

• Level 2 of the Framework Includes– Formal, Complete, Documented Procedures for

Implementing Policies Established at Level One


• Level 3 of the Framework Includes– Security Procedures and Controls That Are Implemented

• Level 4 of the Framework Includes– Routinely Evaluating the Adequacy and Effectiveness of

Security Policies, Procedures, and Controls

• Level 5 of the Framework Includes– A Comprehensive Security Program That Is an Integral

Part of an Agency’s Organizational Culture








Methods Usage

Methods UsageMethods Usage

• Most Methods – Have Specific Activities to Be Performed– Can Be Applied to Specific Projects– Need a Team of About 3 - 4 Auditors– Requires Training or Understanding of the

Method


• Since Methods Have Specific Activities– Questionnaires Can Be Generated– Results Can Be Tabulated– Analysis Can Be Formed Quickly From the

Results


• The Sw-cmm and Sa-cmm Methods Require the Audit Lead Be Specifically Trained


Sample Data Collection Instrument

Audit Reporting

Audit ReportingAudit Reporting

• Audit Report Can Be Briefing Slides or Full Reports

• Briefing Slides Can Contain Both Summary or Detailed Results


•Sample SA-CMM Summary Results


Sample SA-CMM Acquisition Risk Management Detailed Results



ContactsContacts

– Keith Rhodes• Phone 1 202 512 6412• Email [email protected]

– Madhav Panwar• Phone 1 202 512 6228• Email [email protected]