Audit of theCanadian Forces Health Information System … · Audit of the Canadian Forces Health Information System Project Final ... the Department of National Defence and the Canadian

AUDIT OF THE CANADIAN FORCES HEALTH INFORMATION SYSTEM

(CFHIS) PROJECT

October 2004

7045-71 (CRS)

FV1.0_CFHIS Rpt_Oct04_CH

Audit of the Canadian Forces Health Information System Project Final – October 2004

NOTICE OF CAVEAT TO THE READER

This review is not intended to assess the performance of contractors; rather it is an internal

ew of processes and practices within the DND/revi CF.

Chief Review Services


SYNOPSIS

In January 2000, the Department of National Defence and the Canadian Forces (DND/CF) initiated a major healthcare restructuring project known as Rx2000. The goal of Rx2000 is to develop and implement solutions for reported healthcare deficiencies thereby improving the standard of care provided to CF members at home and abroad. One of the areas of concern included the manner in which health information was being collected, stored, shared or utilized1. The Canadian Forces Health Information System (CFHIS) is intended to address this issue. The CFHIS is a $108 million project (excluding GST) to integrate a series of Commercial Off-the Shelf (COTS) applications that are to provide an enterprise-wide, health information system. The CFHIS project will be implemented in three phases over a five-year period. At the time of the audit, project completion was targeted for 2008. Due to the cost, schedule and significance of the CFHIS, Chief Review Services (CRS), in partnership with KPMG, conducted an audit of the first phase of the project to determine if appropriate processes and controls were in place to manage and successfully deliver the CFHIS. The CRS audit was conducted from August to October 2003. This report presents the results of the audit, which focused on the following areas: The effectiveness of the project’s management control framework; The soundness of the project’s risk management strategies; and The project’s use of information for decision-making and reporting.

Although the first phase of the CFHIS project was forecast to be on schedule and on budget at the time of the audit, over the long-term, fundamental weaknesses in key areas will make it difficult for the CFHIS project management office (PMO) to meet final deliverables within the prescribed budget and timeframes. Specifically, this audit report addresses weaknesses in the implementation of core project management processes, the application of risk management practices and the availability of project documentation and information. Without structured and rigorous project processes and outputs to manage the overall CFHIS project, critical interdependencies and risks will be overlooked and may significantly affect projected costs and schedule. The key recommendations of this audit are targeted at achieving a higher level of diligence and rigour in the management of the CFHIS project in order to increase the probability of success in downstream phases. Management action plans provided by the ADM(IM)/DGIMPD and the CFHIS project management team demonstrate constructive attention to the recommendations contained in this report and we are generally satisfied that corrective actions have been, or will be, implemented. It is important to emphasize that developing and implementing project plans and processes are only the start of achieving the required level of rigour. Equally important is ensuring that project plans and processes are maintained, communicated, used and monitored. Recommendations and corresponding management action plans are presented in matrix format at Annex F of this report.

Chief Review Services i/i

1 Additional information on the Rx2000 initiative can be found at the following website: http://www.forces.gc.ca/health/projects/rx2000/engraph/profile_orgfin_e.asp.

http://www.forces.gc.ca/health/projects/rx2000/engraph/profile_orgfin_e.asp


TABLE OF CONTENTS

RESULTS IN BRIEF......................................................................................................................................................................... I

INTRODUCTION .......................................................................................................................................................................... I

BACKGROUND ............................................................................................................................................................................ I

OVERALL ASSESSMENT.......................................................................................................................................................... III

KEY RESULTS ............................................................................................................................................................................V

KEY RECOMMENDATIONS.......................................................................................................................................................VI

MANAGEMENT ACTION PLANS ..............................................................................................................................................VII

ANNEX A – AUDIT OBJECTIVES, SCOPE AND METHODOLOGY ......................................................................................... A-1

ANNEX B – CONTROL OBJECTIVES FOR INFORMATION AND RELATED TECHNOLOGY (COBIT®)............................... B-1

ANNEX C – COBIT® CONTROL OBJECTIVES BY TB INTERNAL AUDIT POLICY GROUPING ........................................... C-1

ANNEX D – CFHIS ORGANIZATIONAL CHART (SIMPLIFIED VERSION) .............................................................................. D-1

ANNEX E – GLOSSARY OF DND ACRONYMS AND PROJECT MANAGEMENT TERMS......................................................E-1

ANNEX F – MANAGEMENT ACTION PLANS ............................................................................................................................F-1



RESULTS IN BRIEF

INTRODUCTION

The successful implementation of information technology (IT) projects is a recognized challenge within the IT industry and the federal government, including the Department of National Defence and the Canadian Forces (DND/CF). In response to this challenge, the Treasury Board Secretariat (TBS) developed the Enhanced Framework for the Management of IT Projects. To strengthen IM project management support and oversight within the DND/CF, the ADM(IM)/Director General Information Management Project Delivery (DGIMPD) Division was created in April 2000, devoted exclusively to the delivery of IM projects. One of the ADM(IM)/DGIMPD’s primary responsibilities is the implementation of a Project Delivery Management (PDM) discipline designed “to assure the successful delivery of major IM projects through the rigorous application of performance management processes and project management techniques”2. It is within this context that Chief Review Services (CRS) conducted an audit of the first phase of the Canadian Forces Health Information System (CFHIS) project. BACKGROUND

The DND/CF has determined that a requirement exists for an integrated, enterprise-wide, health information system for the collection, provision and sharing of health information as required by health care providers, CF members and decision-makers throughout the DND/CF. The CFHIS involves the integration of a series of Commercial Off-The-Shelf (COTS) applications and ancillary products that, upon completion, are to provide full health information capabilities and support including:

Master Patient Index; Laboratory; Diagnostic Imaging; Pharmacy; Order Entry and Results Reporting; Dental and Medical charting; and Clinical and Executive decision-making support.

The CFHIS project is linked to many other on-going health care reform initiatives and projects within the DND/CF, particularly Rx2000, the umbrella project to develop and implement solutions for reported CF health care deficiencies. Additional information on the Rx2000 initiative can be found at the following website: http://www.forces.gc.ca/health/projects/rx2000/engraph/profile_orgfin_e.asp.

Chief Review Services I/VII

2 Director Information Management Project Planning and Control (DIMPPC) Project Delivery Management Concept of Operations – Version 1.1, 29 June 2000.

http://www.forces.gc.ca/health/projects/rx2000/engraph/profile_orgfin_e.asp


In November 2002, a contract was awarded to a prime contractor to develop and implement the first phase of the CFHIS project with options for subsequent phases based on performance. At the time, this three-phased, $108 million project (excluding GST) was targeted for full implementation by February 2008. Phase I work began immediately following contract award. Overall responsibility for the project resides with the CFHIS project management office (PMO). This includes overseeing and contributing to prime contractor deliverables in addition to coordinating and managing all internal DND/CF project management and departmental activities required to implement and deliver the CFHIS. A breakdown of the approved3 CFHIS budgeted project funding requirements by phase is presented in Figure 1 below.

CFHIS Project Funding Requirements - $108M (excluding GST)

0.0

10.0

20.0

30.0

40.0

50.0

Escalation 0.5 2.2 1.3

Contingency 8.2 5.0 2.4

Internal DND/CF 5.0 6.4 10.1

Contract-VP 3.4 12.7 7.7

Contract-FP 15.5 21.2 6.6

Phase I - Dec 02 to Feb 04 Phase II - Mar 04 to Feb 06 Phase III - Mar 06 to Feb 08

Figure 1: Budgeted CFHIS Project Funding Requirements Source: CFHIS Budget and Escalated Costing (v3.15)

$M

$33M $28M

$47MCFHIS Project Funding Requirements - $108M (excluding GST)

0.0

10.0

20.0

30.0

40.0

50.0

Escalation 0.5 2.2 1.3

Contingency 8.2 5.0 2.4

Internal DND/CF 5.0 6.4 10.1

Contract-VP 3.4 12.7 7.7

Contract-FP 15.5 21.2 6.6

Phase I - Dec 02 to Feb 04 Phase II - Mar 04 to Feb 06 Phase III - Mar 06 to Feb 08

Figure 1: Budgeted CFHIS Project Funding Requirements Source: CFHIS Budget and Escalated Costing (v3.15)

$M

$33M $28M

$47M

The CFHIS contract consists primarily of a fixed-price (FP) portion with a variable-price (VP) portion for completing approved work packages. Work packages are required for delivering system functionality and for project management support that are not fixed-price deliverables (i.e., interfacing with other departmental systems or integrated project team (IPT) effort). At the time of the CRS audit, actual project expenditures totalled $9.8 million (excluding GST) and total commitments were $8.2 million (excluding GST). In total, 74 per cent (or $18 million) of Phase I budgeted project costs (excluding contingency and escalation) were spent or committed as at 08 September 2003. The CFHIS PMO was also forecasting that Phase I would be delivered on time and on budget without a requirement for contingency funds.

Chief Review Services II/VII

3 On 25 July 2002, the CFHIS Program Management Board (PMB) granted departmental approval for the CFHIS project at a revised indicative cost of $108 million (excluding GST). The project was formally approved on 21 November 2002.


Due to the overall cost, schedule and importance of the CFHIS, CRS, in partnership with KPMG, conducted an audit of the first phase of the project during the first year of its five-year implementation plan. The audit was conducted from August to October 2003 to determine whether appropriate processes and controls were in place to manage and successfully deliver the CFHIS. Industry accepted project management and IT governance frameworks were utilized to develop the audit methodology such as the Project Management Body of Knowledge (PMBOK®) and the Control Objectives for Information and related Technology (COBIT®). COBIT is a generally applicable and accepted standard for good IT security and control practices. It provides a reference framework for management, users and IT audit and security practitioners to manage the risks associated with IT-related processes. COBIT also provides a measurement tool to evaluate and rate the maturity of processes from non-existent (0) to optimized (5). Additional information on the PMBOK and COBIT can be found at the following websites: http://www.pmi.org/prod/groups/public/documents/info/pp_standards_overview.asp and http://www.isaca.org/Template.cfm?Section=COBIT6&Template=/TaggedPage/TaggedPageDisplay.cfm&TPLID=55&ContentID=7981. OVERALL ASSESSMENT

We encountered a team of people who are clearly dedicated to the success of the CFHIS project. However, the audit team has rated the project overall, at a “Level 2”, using the COBIT Maturity Model. A “Level 2” maturity rating, on a scale of 0 to 5, indicates that processes in place for the project are repeatable but intuitive. Essentially, the CFHIS PMO is placing a relatively high degree of reliance on the knowledge of individuals, and less on clearly defined, documented and communicated processes. Processes follow a regular pattern but many are only partially designed or implemented. This is evident in the COBIT maturity ratings summarized by individual project process in Figure 2, presented further on in this section. At the same time, team member experience and expertise is not being fully leveraged. The CFHIS PMO is largely making use of the prime contractor’s processes rather than developing and implementing overall project processes that integrate both prime contractor deliverables and internal DND/CF activities. Since the majority of Phase I activities are fixed-price deliverables, the CFHIS PMO is currently able to manage with this approach. However, project complexity will increase in successive phases of the project, as many more concurrent activities are required to be completed. The fixed-price portion of the contract will decrease from 65 per cent of total Phase I budgeted project costs (excluding contingency and escalation) to 53 per cent and 27 per cent in Phases II and III respectively. Unless structured and rigorous processes and outputs are implemented to manage the overall CFHIS project, critical interdependencies and risks will be overlooked and may significantly affect project cost and schedule. It is recognized that at this stage in the project, the observations can only speak to potential risks and impacts. However, there are certain attributes, or red flags, which have some similarity to those associated with a previous DND/CF IM project that was eventually cancelled after four years and a total expenditure of $65 million. In 1993, the RIIP/RAMS (Reserve Automated Management System, a component of the Reserve Integrated Information Project) project budget was set at $79.3 million and was scheduled for completion in five years. In 1997, it was concluded that the project could not deliver the key functional capabilities within the prescribed budget or timeframes. It was

Chief Review Services III/VII

http://www.pmi.org/prod/groups/public/documents/info/pp_standards_overview.asp

http://www.isaca.org/Template.cfm?Section=COBIT6&Template=/TaggedPage/TaggedPageDisplay.cfm&TPLID=55&ContentID=7981


subsequently cancelled and CRS was requested to perform an audit of the circumstances4. Although the RIIP/RAMS project faltered in large part due to an unproven procurement strategy, weaknesses in the application of project management systems and a lack of follow through on risk mitigation strategies exacerbated the situation. Another significant issue was the absence of clear project documentation that resulted in poor corporate memory and difficulty for new project members and stakeholders. While it is certainly not the intention to be alarmist, these same weaknesses are, at least in some degree, evident with respect to the CFHIS project and should not go unattended. Due to the known risks related to large, multi-year IM projects and the increasing complexity of the CFHIS project, weaknesses in project processes should be addressed immediately. The project processes shown in Figure 2 below, should have at a minimum, a COBIT Maturity Model rating of “Level 3”. This indicates that processes are clearly defined, documented and communicated. To demonstrate added due diligence and to maximize value for money, project teams should be challenged to set even higher target maturity levels. COBIT Maturity Model Ratings CRS Audit Objectives 0

Non- Existent 1

Initial / Ad Hoc2

Repeatable But Intuitive3

Defined Process4

Managed / Measurable 5

Optimized Project Management Control Framework - Project Management Framework 1 - Scope Management 2 - Master Project Plan 1 - Project Phase Approval 2 - HR Management 1 - Communications Mgmt (Internal) 1 - Communications Mgmt (External) 2 - Cost Management 3 - Quality Management 1 - End-User Participation 2 - Training Strategy 3 Risk Management Strategies - Risk Management 1 - Service Level Management 1

3 - System Security Plan / Compliance with External Requirements 2 Information for Decision-Making and Reporting - Project Documentation 1 - Business Processes 1

Figure 2: COBIT Control Objectives assessed in the audit have been categorized by TB audit policy grouping and PMBOK knowledge areas.

Chief Review Services IV/VII

4 CRS File Number 7045-74-5, October 2000. The CRS report can be found at the following website: http://www.forces.gc.ca/crs/rpt/riip_e.htm.

http://www.forces.gc.ca/crs/rpt/riip_e.htm


KEY RESULTS

Over the course of the audit, the ADM(IM)/DGIMPD and the CFHIS PMO team were very quick to respond to CRS concerns and actions were underway in October 2003 to begin resolving the identified issues summarized below.

The CFHIS project’s Management Control Framework lacks clear definition and structure:

There is no evidence of a CFHIS master project plan; a complete CFHIS work breakdown structure (WBS) for the overall project; an integrated CFHIS project schedule; an overall change control process for both project and product scope; or a CFHIS human resources management plan. Although many project sub-plans have been partially developed or exist in draft format, there are no overall plans that include both the prime contractor and internal DND/CF efforts.

The ADM(IM)/DGIMPD’s quality management of CFHIS project management processes is largely reactive rather than proactive in

nature. Key Project Oversight PDM Practices applied to the CFHIS project such as Delivery Management Meetings (DMMs) are not functioning as well as intended.

Senior-level oversight mechanisms such as a Senior Review Board (SRB) and a Program Review Board (PMB) are in place for the

project. The SRB met six times from March 1999 to June 2003 and the PMB met four times in approximately the same timeframe. Approval of the CFHIS WBS and the project implementation plan (PIP) is included as a responsibility in the SRB Terms of Reference. Discussion specifically related to these project plans were not found in the SRB or PMB meeting minutes provided.

Risk Management Practices are not well developed or actively followed:

Key project team members are not highly involved with risk management activities. The audit team noted that in one of the project’s risk registries, risks had been assigned to various project team members. When interviewed, project team members were unaware that they had been assigned this responsibility.

A CFHIS Risk Management Plan was developed at the beginning of the project by a consulting firm. However, it was not

implemented and identified risks have not been tracked or monitored over time. The CFHIS PMO is currently trying to revise and implement a more robust risk management process, but assigned project team members do not demonstrate strong project management experience in this area.

A total of $15.6 million in contingency funds has been included in the CFHIS project budget based on the CFHIS Project Profile and

Risk Assessment (PPRA), yet there is no evidence that identified risks are being actively managed or monitored. Risk Identification, Risk Mitigation and Risk Monitoring are key to keeping the CFHIS project on schedule and on budget.

Chief Review Services V/VII


There is a lack of project documentation and information from structured project management processes:

The CFHIS project documentation repository is not structured effectively and project documentation is not consistently updated. Management decisions are being made in the absence of project outputs from structured project management processes. For

example, additional project resources are being hired without a human resources management plan identifying required resources. A lack of documentation also results in poor corporate memory and weak project communication. New project team members and

other stakeholders will encounter difficulty in making informed decisions without clear documentation capturing prior events, decisions and reasoning. This is more significant for projects with lengthy and complex implementation plans such as the CFHIS.

KEY RECOMMENDATIONS

It is recommended that the following actions be taken to achieve a greater level of rigour and increase the probability of success in the downstream phases of the project: ADM(IM)/DGIMPD clearly identify, define and prioritize the core project management processes and outputs considered integral to

the successful delivery of the CFHIS project. At a minimum, CFHIS project outputs should include: a clear and succinct phased project scope document; a phased work breakdown structure (WBS) capturing total project scope; an integrated project schedule; and an integrated project master plan.

The CFHIS PMO establish a process to identify, monitor and manage key CFHIS project interdependencies using project outputs such

as the recommended WBS and project schedule and update project outputs as required (i.e., CFHIS budgeted funding requirements). ADM(IM)/DGIMPD and the CFHIS PMO develop a CFHIS human resources management plan including skills mapping, training

plans and clarification of roles and responsibilities for all project team positions and personnel. The CFHIS PMO develop a comprehensive CFHIS risk management plan by consolidating risks from existing risk registries as well

as categorizing, prioritizing, quantifying, assigning, monitoring and following-up on identified risks.

ADM(IM)/DGIMPD strengthen the monitoring and oversight of the CFHIS project to ensure that project management processes remain in place and are followed and that accurate and consistent outputs can be used for management decision making.

Chief Review Services VI/VII

ADM(IM)/DGIMPD and the CFHIS PMO develop a strategy and schedule to ensure that identified core project management processes are implemented in a timely manner. Consideration should be given to acquiring external project management expertise on a short-term basis to ensure that recommendations are implemented promptly and without undue delay to project progress.


MANAGEMENT ACTION PLANS

The ADM(IM)/DGIMPD and the CFHIS PMO acknowledge the audit findings and have accepted the audit recommendations. As indicated in the management action plans, provided in matrix format at Annex F, work has already commenced on addressing the key recommendations included in this report. Of particular note is the assignment of a risk manager, the completion of a comprehensive risk management plan and the implementation of risk management processes. We are generally satisfied that the management action plans appropriately address the concerns raised in this report and that corrective actions have been, or will be, implemented. It is important to emphasize that developing and implementing project plans and processes are only the starting point of achieving the required level of rigour. Equally important is ensuring that these plans and processes are maintained, communicated, used and monitored. It is this type of diligence and discipline that will help to increase the probability of successfully delivering the CFHIS in downstream phases of the project.

Chief Review Services VII/VII


ANNEX A – AUDIT OBJECTIVES, SCOPE AND METHODOLOGY

AUDIT OBJECTIVE

The objective of the audit was to assess the project management framework of the DND/CF Canadian Forces Health Information System (CFHIS) project and to determine whether appropriate processes and controls were in place to manage the project. The audit objective is aligned with the Treasury Board (TB) Internal Audit policy and includes:

the effectiveness of the project’s management control framework;

the soundness of the project’s risk management strategies; and

the project’s use of information for decision-making and reporting. SCOPE

The scope of the audit included the project management framework and controls established for the implementation, management and delivery of the CFHIS project. As the CFHIS project was in the first year of its five-year implementation plan at the time of the audit, the scope was limited to the processes and controls that were planned or in place for the first phase of the CFHIS project. METHODOLOGY

Industry accepted project management and IT governance frameworks were utilized to develop the audit methodology such as the PMBOK®, published by the Project Management Institute and COBIT®, published by the IT Governance Institute. COBIT is a generally applicable and accepted standard for good practices in IT security and control. It provides a reference framework for management, users and IT audit and security practitioners for managing the risks associated with IT-related processes. COBIT also includes a measurement tool to evaluate and rate the maturity of processes. The COBIT Maturity Model ratings range from 0 to 5 and are defined below:

0 - Non-Existent Management processes are not applied at all.1 - Initial / Ad Hoc Ad hoc approaches in place of standardized processes.2 - Repeatable But Intuitive Processes follow a regular pattern.3 - Defined Process Processes are documented and communicated.4 - Managed and Measurable Processes are monitored and measured.5 - Optimized Processes have been refined to a level of best practice.

COBIT Maturity Levels

Chief Review Services A-1/2


ANNEX A

The CRS audit team, in collaboration with KPMG, used the COBIT Maturity Model to measure and rate each of the COBIT Control Objectives assessed in the audit. Further detail on COBIT is provided at Annex B.

A summary of the individual COBIT Control Objectives assessed in the audit are provided at Annex C and have been cross-referenced to the TB audit policy groupings and PMBOK knowledge areas presented earlier in Figure 2.

Key project team members interviewed by the audit team are highlighted in the CFHIS organizational chart provided at Annex D.

Chief Review Services A-2/2


ANNEX B – CONTROL OBJECTIVES FOR INFORMATION AND RELATED TECHNOLOGY (COBIT®)5

5 COBIT® 3rd Edition Management Guidelines, July 2000. Released by the COBIT Steering Committee and the IT Governance Institute. 6 DM Review Online, October 2003. Editorial Staff.

▼ COBIT is published by the ITGovernance Institute and isrecognized internationally as agenerally applicable and acceptedstandard for good practices in ITgovernance, security and control.

▼ COBIT is used or recommended by

auditors and managementworldwide, including the USNational Association of State ChiefInformation Officers (NASCIO) andthe Office of Inspector General(OIG) of the US House ofRepresentatives6.

▼ For the CFHIS audit, the audit team

tailored and assessed 19 of the 34COBIT High-level ControlObjectives from the Planning andOrganization (PO), Acquisition andImplementation (AI) and Deliveryand Support (DS) domain areas.COBIT High-level ControlObjectives assessed in the audit areprovided at Annex C.

Chief Review Services B-1/3

Audit of the Canadian Forces Health In

COBIT also provides the ability for omodels are derived from the Softwarprocesses from non-existent (0) to opt

0 - Non-Existent Ma1 - Initial / Ad Hoc Ad 2 - Repeatable But Intuitive Pro3 - Defined Process Pro4 - Managed and Measurable Pro5 - Optimized Pro

COB

A fundamental feature of the maturitydeveloping strategies to achieve themorganizations. Benchmarking Information In response to the many COBIT user rsurvey from March to June 2002, to awere received, distributed over differe Figure 3: Survey Respon

Location, Size and Se

7 “Capability Maturity Model® for Software,”February 1993.


8 “Control and Governance Maturity Survey: E. Guldentops, CISA, W. Van Grembergen, P

formation System Project Final – October 2004

ANNEX B rganizations to measure and rate the maturity of their management processes. The COBIT maturity e Engineering Institute (SEI) Capability Maturity Model7 and allow organizations to grade existing

imized (5). COBIT maturity levels are defined as follows:

nagement processes are not applied at all.hoc approaches in place of standardized processes.cesses follow a regular pattern.cesses are documented and communicated.cesses are monitored and measured.cesses have been refined to a level of best practice.

IT Maturity Levels

model is that it enables organizations to define “to be” or target maturity levels and also assists with . Additionally, it provides a reference tool for organizations to benchmark themselves against other IT

equests, the Information Systems Audit and Control Association (ISACA) conducted a self-assessment ssess the average maturity of the 15 most important COBIT processes8. In total, 168 valid responses

nt geographies, sizes and sectors as presented below in Figure 3.

dents by ctor

Version 1.1 Technical Report CMU/SEI-93-TR-024, Software Engineering Institute, Carnegie Mellon University, PA,

B-2/3

Establishing a Reference Benchmark and a Self-assessment Tool.” Information Systems Control Journal, Volume 6, 2002. h.D., and S. De Haes.


ANNEX B In line with the overall focus of the CFHIS audit, the CFHIS overall project rating has been presented below in Figure 4 along with the ISACA unweighted average survey results for PO 10, the COBIT Control Objective designed for managing projects. Due to the nature of self-assessment surveys, it is important that these results are interpreted with necessary care and that they are positioned in context. For the purposes of this report, the survey results are to be used as a point of reference. PO 10 - Manage Projects

CFHI

S Pr

ojec

tO

vera

ll Re

sults

Larg

e O

rg'n

Mid

Org

'nSm

all O

rg'n

Publ

ic Se

ctor

Fina

nce

Sect

orRe

tail /

Mfg

Sec

tor

Glo

bal O

rg'n

Asia

/ O

cean

iaEM

EA

0

1

2

3

4

5

CO

BIT

Mat

urity

Lev

els

PO 10 - Manage Projects

Figure 4: ISACA Control and Governance Maturity Self-Assessment survey results for PO 10 – Manage Projects.

Due to the cost, schedule and significance of the CFHIS project, project processes should have, at a minimum, a COBIT Maturity Model rating of “Level 3” indicating processes are clearly defined, documented and communicated. This is similar to the ISACA survey results for global and large organization respondents. To demonstrate added due diligence and to maximize value for money, project teams should be challenged to set even higher target maturity levels.

Chief Review Services B-3/3


ANNEX C – COBIT® CONTROL OBJECTIVES BY TB INTERNAL AUDIT POLICY GROUPING

The CRS audit team, in collaboration with KPMG, tailored and assessed 19 of the 34 COBIT High-level Control Objectives from the Planning and Organization (PO), Acquisition and Implementation (AI) and Delivery and Support (DS) domain areas.

CRS Audit Objectives / TB Audit Policy COBIT Control Objectives Assessed in the Audit Project Management Control Framework - Project Management Framework PO 10 Manage Projects – PM Framework - Scope Management PO 10 Manage Projects – Project Definition AI 1 – 3 Automated Solutions AI 6 Manage Changes - Master Project Plan PO 10 Manage Projects – Master Project Plan - Project Phase Approval PO 10 Manage Projects – Phase Approval - HR Management PO 4 Define the IT Organization PO 7 Manage Human Resources PO 10 Manage Projects – Project Team - Communications Mgmt (Internal) PO 6 Communicate Aims and Directions - Communications Mgmt (External) PO 6 Communicate Aims and Directions - Cost Management PO 5 Manage the IT Investment - Quality Management PO 10 Manage Projects - Quality Plan PO 10 Manage Projects - Test Plan PO 11 Manage Quality AI 5 Install and Accredit Systems - End-User Participation PO 10 Manage Projects - User Involvement - Training Strategy PO 10 Manage Projects - Training Plan Risk Management Strategies - Risk Management PO 9 Assess Risks PO 10 Manage Projects – Project Risk - Service Level Management DS 1 Define and Manage Service Levels DS 3 Manage Performance and Capacity DS 13 Manage Operations

PO 10 Planning of Assurance Methods DS 5 Ensure Systems Security - System Security Plan /

Compliance with External Req’ts PO 8 Compliance with External Requirements

Information for Decision-Making and Reporting - Project Documentation PO 2 Information Architecture Definition - Business Processes AI 4 Develop and Maintain Procedures

Chief Review Services C-1/1

Audit of the Canadian Forces Health Information System Project

ANNEX D – CFHIS ORGANIZATIONAL

Denotes CFHIS personnel interviewed *Denotes those not interviewed

Project LeadADM (IM)/DGIM

Project ImplemeADM (IM)

Admin Coordinator

Test Manager

Deputy Project DPL Project Director

PD

Functional Le

Project Control Office PCO

Project ManagPM

Resou

* Interviews were also conducted with the PWGSC contracting officer and ke Manager, Chief Engineer and Project Scheduler).

Train

CFHIS PMO

Project Sponsor ADM (HR-Mil)

Project Leader ADM (HR-Mil)/DGHS


CHART (SIMPLIFIED VERSION)

ProcuremADM (Mat)/D

er PD

ntation

Deputy Project Manager DPM

Leader

ad Technica

er

rces

DCPS ProcuOfficer

y members of the prime contractor’s team

ing Lead TrL

Re

CFHIS Comm

Senior R

Project Man

Final – October 2004

ent CPS

l Lead

rement

(LM Program

sources

ittee Governance

eview Board

agement Board

D-1/1


ANNEX E – GLOSSARY OF DND ACRONYMS AND PROJECT MANAGEMENT TERMS

ADM(IM) Assistant Deputy Minister Information Management OIG Office of Inspector GeneralADM(Mat) Assistant Deputy Minister Materiel LC Limited Capability AI Acquire and Implementation (COBIT® domain area) MA&S Materiel Acquisition and Support CF Canadian Forces OPI Office of Primary InterestCFHIS Canadian Forces Health Information System PCO Project Control OfficerCMM Capability Maturity Model PD Project Director COBIT® Control Objectives for Information and related Technology PDM Project Delivery ManagementCOTS Commercial Off the Shelf PIR Project Independent Review (PDM process)CRS Chief Review Services PM Project ManagerDCPS Director Common Procurement and Supply PMB Program Management Board DESC Defence Enterprise Service Centre PMBOK® Project Management Body of Knowledge DGHS Director General Health Services PMO Project Management OfficeDGIMPD Director General Information Management Project Delivery PO Planning and Organizing (COBIT® domain area)Dir IM Secur Directorate of IM Security PPMP Project Performance Management Plan (PDM process)DMM Delivery Management Meetings (PDM process) PPRA Project Profile and Risk AssessmentDND Department of National Defence PSR Project Status Reports (PDM process)DPL Deputy Project Leader PWGSC Public Works and Government Services CanadaDPM Deputy Project Manager RAMS Reserve Automated Management SystemDS Delivery and Support (COBIT® domain area) RIIP Reserve Integrated Information ProjectFAA Financial Administration Act RFP Request for Proposal FC Full Capability SEI Software Engineering Institute FMAS Financial Managerial Accounting System SLA Service Level Agreements HR Human Resources SME Subject Matter ExpertHRMS Human Resources Management System SRB Senior Review BoardIM Information Management TB Treasury Board ISACA Information Systems Audit and Control Association TBS Treasury Board Secretariat IT Information Technology TrL Training lead NASCIO National Association of State Chief Information Officers WBS Work Breakdown Structure

Chief Review Services E-1/1


ANNEX F – MANAGEMENT ACTION PLANS

CRS Recommendations OPI(s) Management Action Plans

1. Clearly identify, define and prioritize the core project management processes and outputs considered integral to the successful delivery of the CFHIS project. At a minimum, CFHIS project outputs should include:

a. a clear and succinct phased project scope document;

b. a phased work breakdown

structure (WBS) capturing total project scope;

ADM(IM)/DGIMPD DGIMPD’s Project Delivery Management (PDM) has established cost, time, scope and risk as key project management processes that must be managed. Project Management processes for these processes will include the development and maintenance of performance indicators. These performance indicators will be incorporated into the monthly Project Status Reports (PSRs) to assist in the management of these processes. An updated Project Performance Management Plan (PPMP) will be produced for CFHIS Phase II, and the CRS audit findings and detailed analysis and recommendations will be incorporated into the creation of the PPMP for CFHIS Phase II. It is expected that the Phase II PPMP will require additional changes to existing project management processes in use within CFHIS. Progress has already been made since the audit was performed, towards the implementation of a more robust risk management process. a. While the project scope is outlined in various documents such as the

Synopsis Sheets, the PPRA, and Project Charter, and more completely defined the project SOR document which is supplemented by the Enterprise Architecture Plan and detailed requirements in a CFHIS PMO maintained DOORS database, we acknowledge that there does not exist a single document that consolidates and links CFHIS scope and aligns this with the three phases of the project. The project will prepare a “Scope consolidation” document that will outline CFHIS scope by phases, provide linkages to other scope documentation such as the SOR and DOORS requirements database. As well, this document will address the detailed findings of the CRS audit related to phased project scope ambiguities.

b. DND and the prime contractor, LMC, are working together for the Phase II

integrated WBS. A preliminary WBS for Phase III will also be developed during Phase II. The integrated WBS will ensure all project scope is addressed.

Chief Review Services F-1/4


ANNEX F


c. an integrated project schedule; and

d. an integrated project

master plan.

c. DND and the prime contractor, LMC, are working together for the project schedule.

d. CFHIS will develop a Project Master Plan consistent with DND project

management guidance contained in the Capability Initiatives Database (please refer to paragraph 1a).

2. Establish a process to identify, monitor and manage key CFHIS project interdependencies using project outputs such as the recommended WBS and project schedule.

Update project outputs as required (i.e., CFHIS budgeted funding requirements).

CFHIS PMO A number of improvements in formalizing the processes, documented capture, execution and control have been made since the CRS audit was performed. An informal process for the identification of internal project interdependencies had been followed since project inception. A formal process has now been implemented within which Technical and Functional Team Leads. On a weekly basis, any issues that could have an effect on the other teams are identified. The issues are documented and presented to a Senior Management Committee composed of the PM, PD and Contractor personnel for resolution in the event the issue cannot be resolved at the team level. If necessary, schedule, WBS and budgets are updated weekly. As part of developing an integrated project schedule, project internal dependencies will be identified and incorporated into the project schedule.

3. Develop a CFHIS human resources management plan including:

a. skills mapping;

ADM(IM)/DGIMPD and CFHIS PMO

The project acknowledges that no documented HR Plan existed or was explicitly communicated at the time of the audit. a. A process has now been established whereby requirements for specific

expertise are identified on an on-going basis and the skilled resources are obtained to fulfill these specific roles. This may be through government staffing or through contract. The prime contractor may also be considered as a source or personnel resources, depending on the expertise required and available.



ANNEX F


b. training plans; and

c. clarification of roles and responsibilities for all project team positions and personnel.

b. Each situation brings its own specific training issues. Training requirements are identified, prioritized and then actioned on an individual basis, based on any experience or formal training gaps.

c. The situation is being reviewed and an HR Plan will be finalized. In

addition, specific responsibilities will be reviewed and information will be communicated to all project personnel to ensure that all understand their responsibilities.

4. Develop a comprehensive CFHIS risk management plan by consolidating risks from existing risk registries as well as categorizing, prioritizing, quantifying, assigning, monitoring and following-up on identified risks.

CFHIS PMO A risk manager has been assigned to consolidate and maintain an integrated and comprehensive CFHIS Risk Management Plan. The plan includes processes for categorizing risks, prioritizing, quantifying, assigning, monitoring, developing mitigation plans and reviewing identified risks through interaction with all team members.

5. Strengthen the monitoring and oversight of the CFHIS project to ensure that project management processes remain in place and are followed and that accurate and consistent outputs can be used for management decision-making.

ADM(IM)/DGIMPD DGIMPD has been monitoring the CFHIS project on an ongoing basis. PSRs are reviewed during the monthly Delivery Management Meetings (DMMs) between DPDMIS (Director Project Delivery Management Information Systems), the PM and key project staff. There have been ongoing improvements to the project management processes used within the CFHIS project. In part, this is as a result of the DGIMPD oversight provided as part of the PDM practices of a monthly PSR and DMM. Since the audit was conducted from August to October 2003, there has been progress made in finalizing documented project management plans. Of note is the completion of a Risk Management Plan, the appointment of a Risk Manager and implementation of risk management processes.



ANNEX F


The Project Performance Management Plan (PPMP), which establishes the reporting standard to be followed in the PSR, will be updated for Phase II of CFHIS. The update will use the findings of the CFHIS audit as input. This will form the baseline for CFHIS monitoring throughout Phase II of the project.

A DGIMPD self-validation, a Project Readiness Review, will also be undertaken to confirm the presence and appropriateness of documented project management processes as well as other project documentation observed on by the CRS audit. PDM includes a Project Independent Review (PIR) process, performed on a recurring basis throughout the life of a project, to assess the project management processes in use within a project. The CRS audit performed the function of the DGIMPD PIR of CFHIS for Phase I. For the next PIR of CFHIS, the CRS audit report will be used as input to confirm appropriate action has been taken within the project.

6. Develop a strategy and schedule to ensure that identified core project management processes are implemented in a timely manner.

ADM(IM)/DGIMPD and CFHIS PMO

The core project management processes required for Phase II will be reviewed during the update of the PPMP. Where new processes or updates to existing processes are required they will be noted and added to the DMM action items list and status reviewed at the monthly DMMs. Identified required resources will be acquired when needed. This was the case in the Risk Management processes where a risk manager was acquired and formal risk management processes implemented during Phase I. These processes will continue to be monitored by the DPL through monthly DMM meetings.


Documents

Audit of theCanadian Forces Health Information System … · Audit of the Canadian Forces Health Information System Project Final ... the Department of National Defence and the Canadian