Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
eManifest Review Review Engagement
December 2012
1
TABLE OF CONTENTS
1.0 INTRODUCTION ......................................................................................... 2
2.0 SIGNIFICANCE OF REVIEW ................................................................... 2
3.0 STATEMENT OF CONFORMANCE ........................................................ 2
4.0 KEY FINDINGS ............................................................................................ 3
5.0 SUMMARY OF RECOMMENDATIONS ................................................. 4
6.0 OVERALL MANAGEMENT RESPONSE ................................................ 4
7.0 FINDINGS ...................................................................................................... 5
7.1 Scope and user requirements ................................................................................................. 5 7.2 Project Plan ........................................................................................................................... 8 7.3 Cost Management ................................................................................................................ 10
7.4 Project Status Reporting ...................................................................................................... 12
APPENDIX A – ABOUT THE REVIEW ........................................................... 15
APPENDIX B – GLOSSARY ............................................................................... 17
APPENDIX C – ACRONYMS ............................................................................. 18
2
1.0 INTRODUCTION eManifest is a major crown project underway at the Canada Border Services Agency (CBSA). Once
fully implemented, eManifest will change the commercial import process. It will require members
of the trade community such as carriers, freight forwarders, and importers in all transportation
modes to electronically transmit data on cargo, conveyance and crew to the CBSA before shipments
arrive at the border.
The Internal Audit Division completed a review of the eManifest project which included controls
over the project scope, project management practices, cost management processes and project status
reporting. The review included interviews and examination of project documentation that took place
during the period from January 1, 2012 to August 30, 2012.
2.0 SIGNIFICANCE OF REVIEW
This review is significant because large government Information Technology (IT) projects like
eManifest are inherently complex, expensive and risky, and have had a history of cost overruns and
delays, and not delivering what had been planned. This review of eManifest will provide CBSA
management with information on the adequacy of its project management practices, in general and
for the eManifest project.
OBJECTIVE
The objective of this review was to determine the extent to which eManifest will meet the needs of
end users; the adequacy of the project plan with respect to managing the project’s scope, schedule
and resources; the management of costs; and the effectiveness of tools and processes for status
reporting.
3.0 STATEMENT OF CONFORMANCE
This review conforms to the Internal Auditing Standards for the Government of Canada, as
supported by the results of the quality assurance and improvement program. The approach and
methodology for this review followed the International Standards for the Professional Practice of
Internal Auditing as defined by the Institute of Internal Auditors and the Internal Auditing
Standards for the Government of Canada as required by the Treasury Board Internal Audit Policy.
This review was intended to provide a low level of assurance commensurate with a review.
3
4.0 KEY FINDINGS
Consultations are occurring with the future end users of eManifest. This is being done through
project working groups and governance committees. Some end users have expressed an interest in
having improved discussion on business requirements and feedback on decisions taken. Project
management understands the importance of good stakeholder consultations and considers this a low
risk in the overall context of the project.
The eManifest project has performed multiple types of testing on previous releases. However, at the
CBSA, standards for test plans have not been developed and final acceptance testing does not
include end users. Test results have not been shared with stakeholders to allow for a more informed
review and evaluation of the system as implemented. Ineffective acceptance testing could lead to
issues and change requests after the system is in the production environment.
The trade-off between project efficiency and end user involvement presents risks that need to be
considered and weighed. For example, the initial release of Build 1 for March 31, 2013 could be at
risk. The preliminary estimated effort to conduct testing was greater than the time available. If this
becomes an issue, management will have to weigh the trade-offs before making a decision. Project
governance committees at all levels are in place to facilitate the decision-making process.
For project planning and monitoring, management does not have a complete set of project
management information to monitor progress with respect to time, scope and cost. Each individual
manager is responsible for managing and reporting monthly on their piece of the project. There is
reliance on this reporting process and confidence that issues and risks will be identified and
addressed in a timely and effective fashion.
Earned value management (EVM) is a process that provides management with information on
investment performance and necessary data points to estimate probable completion costs and dates.
As part of the Agency’s cost management plan for eManifest, it had agreed to use EVM to measure
performance. While this work was initiated, it was discontinued due to the complexity and lack of
agency tools for the EVM process.
Multi-year projects are expected to forecast costs based on a costing template that aligns with the
Treasury Board Submission. eManifest is not following this process. Project managers are
negotiating their budget requirements with the eManifest project office without knowing the cost of
the detailed project plans. Cost forecasts to complete eManifest are based on the original Treasury
Board submission and workplans developed by project managers.
Comptrollership Branch has indicated that it is unable to provide reasonable assurance on the
project’s financial health because project costs and forecasts are not done by deliverable.
Comptrollership does monitor and discuss actual cost information with project management to
ensure it is consistent with corporate information.
Project management is reporting regularly on the status of the project. Clarification is required on
the information that should be reported, including the completeness of some of the dashboard
4
elements. This includes reporting by deliverable, rather than project phase. Oversight on the
dashboard could be strengthened to ensure all significant issues from CBSA stakeholders are
reported. The risk is that CBSA executive management may not be informed of significant issues
that could have broader implications for operations.
5.0 SUMMARY OF RECOMMENDATIONS
The review makes six recommendations to:
Review the current consultation process with end users to ensure it is understood and issues
are properly addressed.
Develop and implement CBSA project standards specific to test plans and user acceptance
testing (UAT).
Assess whether the system development life cycle and project management practices that
support agile techniques undertaken by the eManifest project are acceptable and applicable
to other Agency projects and update project management policies and guidance accordingly.
Develop and implement a process to demonstrate benefits realization of eManifest.
Provide assurance on the financial health and the remaining cost forecast of the eManifest
project and sign off as such on the dashboard.
Require that the Director General (DG) Projects Steering Committee reviews and discusses
all project risks and issues and include significant risks and issues on the eManifest project
dashboard.
6.0 OVERALL MANAGEMENT RESPONSE
The Programs Branch, Comptrollership Branch, and the Information, Science and Technology
Branch (ISTB) agree with this review and its findings.
Programs Branch will implement an improved consultation process with end users, will
demonstrate the benefits realization of eManifest, and will take steps to ensure that the DG Projects
Steering Committee is able to review and discuss all significant project risks and issues.
ISTB will assess whether the system development life cycle and project management practices that
support agile techniques, undertaken by the eManifest project are acceptable and applicable to other
Agency projects and will update project management policies and guidance accordingly. ISTB will
also move forward with a modernization and transformational proposal for a testing
framework/approach to address the recommendations.
Comptrollership, in consultation with Programs Branch (Office of Primary Interest), ISTB (Office
of Collateral Interest) and the Enterprise Project Management Office (EPMO) will implement a
process to strengthen the quality assurance of project financial information and the remaining cost
forecast of the eManifest project and sign off as such on the dashboard.
5
7.0 FINDINGS 7.1 Scope and user requirements
Project Management Objective: The project will meet its scope and end user requirements.
The success of a project depends on the effectiveness of the final project output in meeting its initial
business objectives and requirements of the business process owners and end users. Good project
management practices assist project managers to meet the project scope and end user requirements.
These practices include a sign-off of the scope definition and user requirements, having appropriate
safeguards in place to manage changes to project scope definition and user requirements, and
conducting user acceptance testing before the system is released into the production environment.
Project Scope:
Prior to 2010, sponsor branches (Admissibility, Enforcement and Operations branches) approved
business requirements. As of the April 1, 2010 reorganization, the Major Projects Directorate of
Programs Branch assumed the role of Office of Primary Interest (OPI) for the eManifest project.
Offices of Collateral Interest (OCI), including other directorates within the Programs Branch, and
other branches, now serve the role of Subject Matter Experts (SMEs). The purpose of this
reorganization was to improve the governance structure efficiencies and to streamline the escalation
process. As a result, internal end users (Operations Branch and Programs Branch, other than Major
Projects Directorate) are no longer required to sign off business requirements and they no longer
have the approval authority on policy issues and scope definition. The trade-off between efficiency
and end-user involvement presents itself with different risks that need to be considered and
weighed.
Currently, the Commercial Unit of the Pre Border Programs Directorate is tasked with coordinating
consultations on requirements between the project team and representatives of future end users. The
Commercial Unit identified relevant OPIs and provides relevant documentation for comments.
Once comments are received, the Commercial Unit will summarize them and attempt to find a
solution to issues. If an issue cannot be resolved, it is sent for a decision from the Director Policy
Steering Committee. Other avenues for feedback include an internal e-mail box, where users can
send issues for the policy team to review and action accordingly. The process has since evolved and
improved through the creation of the Managers Policy Committee in February 2012. This
committee was created to review new system functionality and discuss policy issues every two
weeks. These sessions should provide an opportunity to discuss and validate user needs. Although
consultations are occurring with future internal users, some end users expressed an interest in
having improved discussion on business requirements and feedback on decisions taken. Project
management understands the importance of good stakeholder consultations and considers this a low
risk in the overall context of the project.
6
Project Change Controls:
Regarding safeguards to manage changes to the project scope definition and end user requirements,
the eManifest project has adopted a more iterative approach to managing the project. The approach
used by the eManifest project team is inspired by agile project management concepts and is a
recognized approach to project management in the industry. Agile requirements elicitation requires
regular consultations with users through various methodologies to accommodate acceptable
business demands.
The eManifest project teams and information technology representatives must agree on which
project artefacts define the scope and the business requirements associated with each of the
remaining three builds that will form the basis for the development of systems. In most cases, the
business teams within the eManifest project are producing user requirements specifically aligned to
each build. An agreement on requirements forms the basis of the baseline, for which systems are
developed. The baseline for the first two builds was established in July 2012 creating a risk that IT
teams had been working using project artefacts that were not finalized. A requirements baseline is
an important element of change control, as all subsequent modifications to requirements after the
baseline is established require a formal change request. Without a baseline, user requirements may
fluctuate during the development process and cause unwanted consequences such as rework to
developed systems.
An additional consideration is the requirement for collaboration with future users during
requirements elicitation. As noted, the future three builds have incorporated the more iterative agile
approach to requirements definition. The current approach adopts aspects of an agile approach with
some important deviations; the eManifest project does not employ smaller release cycles, in which
the product is continuously ready for release. Additionally, continuous feedback between end users
and the development teams is not carried out. Consequently, there can be important lags between
when comments are received from stakeholders and when these groups will validate the
functionality developed.
User Acceptance Testing:
Typically, before system changes are released into the production environment, the application is
tested in a controlled environment to determine functionality and usability using scenarios that
resemble how the application will be employed by the end user.
The eManifest project has a testing plan and testing of previous releases has occurred. However, at
the CBSA, organizational guidance or standards specific to test plans or user acceptance have not
been developed.
With respect to testing activities for eManifest releases, areas to strengthen would include more
involvement of end users, and improved documentation and communication of test results.
Specifically, end users were not involved in acceptance testing and there was no separation of
duties between the individuals who defined the requirements and those who tested the requirement.
7
The test plan was insufficient in detail. Test plans should include details on the scope of testing,
schedule, test deliverables, release criteria and risks, and contingencies. The test results reviewed
were not adequately documented and not shared with all stakeholders to allow for an informed
review and evaluation of the system implemented by internal project stakeholders such as Operation
Branch, Border Programs and Pre-Border Programs.
In May 2012, a release pertaining to the Passage Deployment for Rail had a reduced testing period.
There were problems with the implementation of the system and resolution of these issues created a
delay of one month to the development schedule for the next release. The May 2012 release
proceeded to meet project time commitments. It is recognized that management needs to balance
project scope requirements, cost and schedule and manage the risks associated with them.
Summary
In regard to this project management objective as noted above, the Agency is exposed to risks that
system functionality may not meet the initial needs of end users and additional costs may be
incurred to make change requests to respond to users’ concerns. As well, expected results to
enhance employees’ efficiency may not be achieved and management may not be able to obtain the
information it requires to effectively manage the business process.
Within the project, there are different levels of risk tolerance. The trade-off between project
efficiency and end-user involvement presents risks that need to be considered and weighed by the
project governance committees.
RECOMMENDATION 1
The Vice-President of Programs Branch should review the current consultation process with end
users to ensure the process is understood, and end user feedback is considered and issues are
properly disposed of.
MANAGEMENT RESPONSE
Programs Branch agrees with this recommendation. The eManifest project team, in
consultation with Operations Branch, will enhance stakeholder consultations by conducting
focus groups with end users on end state passage and risk assessment systems. It will also
review and document the current process for consultation with end users, and will identify
any significant gaps in this process. By March 2013, Programs Branch will implement a
revised consultation process that will ensure the process is understood, the views of end
users are considered, and any issues identified by the process are addressed appropriately.
8
RECOMMENDATION 2
The Vice-President of ISTB in collaboration with the Vice-President of the Programs Branch
should develop and implement CBSA project standards specific to test plans and user acceptance
testing (UAT).
MANAGEMENT RESPONSE
The Information, Science and Technology Branch (ISTB) agrees with this recommendation.
ISTB will move forward with a modernization and transformational proposal for a testing
framework/approach to address standards specific to test plans and UAT. By April 2013, a
Testing Model Pilot will be implemented.
7.2 Project Plan
Project Management Objective: The project plan is sufficiently detailed to guide the project
throughout its lifecycle.
Project management guidance at the CBSA has established a common approach for managing
projects. Project plans are required documentation that includes scope, deliverables, work
breakdown structures, roles and responsibilities, cost estimates, milestones, key dependencies and
the identification of the project's critical path. An effective project plan establishes a baseline for the
project and enables management to track progress in terms of scope, time and costs. This guidance
developed by the Enterprise Project Management Office is largely based on traditional project
management principles.
The eManifest approach to project scheduling adopts some agile concepts to realize efficiencies.
The Agency’s guidance does not provide for alternative methodologies such as agile project
management, which is recognized as a valid project management methodology. In summary, the
eManifest project management approach includes:
A scope definition that aims to realize efficiencies. In December 2010, the eManifest
Strategic Deployment View Task Force was established to develop a high-level deployment
sequence for the remaining eManifest deliverables. This scope definition was amended to
allow deliverables to be released in parallel and realize efficiencies in the schedule.
A time box approach was adopted for project planning, where deadlines (end dates) related
to system implementation are fixed. The scheduled dates for tasks are permitted to fluctuate
to meet deadlines. Each project team is accountable for meeting its deadlines.
The master schedule contains the duration and estimated hours per task; a high number of
detailed plans are aggregated to the project master schedule. The granular project plans
contain the deliverables agreed upon in the scope definition document.
Three Builds for eManifest remain with release dates; for Build 1 – March 2013, Build 2 –
July 2014 and Build 3 – December 2014.
9
In project planning, a work breakdown structure helps accurately define and organize the scope of
the project by breaking down the project deliverables into work packages across different levels of
the project plans. Based on a review of project plans, the following was observed:
Project deliverables were indicated on detailed project plans.
Activities have been assigned an estimated duration, but did not provide specifics on
resource assignments, estimated effort or standard costs.
For detailed project plans, the 8/80 rule was not employed; this guideline stipulates that no
work package should be less than 8 hours or greater than 80 hours. This rule allows project
managers to govern the size and scope of the smallest chunks of work and apply consistent
rules across work breakdown structures.
A critical path method is used to predict project duration by analyzing the sequence of activities that
has the least amount of scheduling flexibility. For eManifest, the traditional critical path method
was not employed. The eManifest project schedule assumes no flexibility in delivery for builds.
The initial release management timelines for Build 1 could be at risk. The preliminary estimated
effort to conduct testing was greater than the time available to meet the delivery date of March
2013. Project governance committees will need to engage if potential issues arise in order to
facilitate decision making.
Summary
With respect to the current project plan for eManifest, management does not have a complete set of
project management information and as such it limits management's ability to monitor progress
with respect to time, scope and costs. Each individual manager is responsible for managing his/her
piece of the eManifest project including resources, scope and schedule. Managers are required to
report regularly on the status of their component. Once a month, the results from managers are
consolidated into the overall eManifest project dashboard. There is considerable reliance on these
management processes and confidence that issues and risks will be identified and addressed in a
timely and effective fashion.
RECOMMENDATION 3
The Vice-President of ISTB should assess whether the system development life cycle and project
management practices that support agile techniques, undertaken by the eManifest project are
acceptable and applicable to other Agency projects, and update project management policies and
guidance accordingly.
MANAGEMENT RESPONSE
The ISTB agrees with this recommendation. ISTB will assess whether the system
development life cycle and project management practices that support agile techniques,
undertaken by the eManifest project are acceptable and applicable to other Agency projects,
and update project management policies and guidance accordingly. The planned completion
date is April 2013.
10
7.3 Cost Management
Project Management Objective: Costs are monitored; deviations are identified and addressed in a
timely manner. Performance will be measured and validated.
Under Treasury Board Policy on the Management of Projects, projects are expected to achieve
value for money and deliver outcomes within cost constraints. The Agency also expects that
projects achieve value for money; the CBSA submitted a cost management plan as part of the
effective project approval that stated that the earned value management technique would be used to
ensure that costs were managed effectively, and performance was measured and validated.
Earned value management (EVM) is a process that provides project managers and stakeholders with
insight into investment performance, and provides the necessary data points to estimate statistically
probable completion costs and dates. The implementation of EVM is a recognized parameter of
good project management. It ensures that cost, schedule, and technical aspects of the investment are
integrated and:
• Relates time-phased budgets to specific tasks;
• Indicates work progress;
• Properly relates cost, schedule, and technical accomplishment to scope;
• Is valid, timely, and auditable;
• Allows for statistical estimation of completion costs;
• Supplies managers with information summarized at a practical level;
• Facilitates risk management; and
• Provides information on benefits realization.
For eManifest, EVM techniques are not currently being employed. Project managers indicated that
it was difficult to meet EVM requirements with the tools they presently have.
Currently, the eManifest project team is tracking actual costs through CAS. Different groups
however are reporting their costs differently. For example, ISTB employees use internal orders to
report their time. Other groups do not employ internal orders for their timesheets; rather they
perform a salary allocation to the project based on actual or estimated costs. The internal order code
is not a mandatory field in CAS. As such this creates a risk that costs are not distributed accurately
by projects.
At present, it is difficult to relate the full project cost to the actual and future project deliverables
and builds. Multiple systems are employed to support the cost management process such as
COMPASS and CAS but are not integrated with the eManifest project plans.
Multi-year projects are expected to forecast costs based on a costing template that aligns with the
Treasury Board submission. eManifest is not following this process. Project managers are
negotiating their budget requirements with the eManifest project office without knowing the cost of
the detailed project plans. Cost forecasts to complete eManifest are based on the original Treasury
Board submission and workplans developed by project managers.
11
As per the CBSA project guidance, the Comptrollership Branch and the EPMO have important
roles related to cost management. The EPMO is responsible for providing project oversight,
maintaining project management guidance, reviewing and analyzing the dashboard reports to ensure
accurate reporting on key project metrics, and validating and challenging the dashboard
information. Comptrollership is responsible for providing and completing the project financial
information and for performing quality assurance of project financial information. Comptrollership
is unable to provide assurance on project financial health because the project costs and forecasts are
not done by deliverables. Comptrollership does review, monitor and discuss with project
management the cost information to ensure the financial information aligns with CAS and
COMPASS.
Summary
In regard to the project management objective, the Agency is exposed to the risk that CBSA
management may not be made aware in a timely manner of potential cost overruns to the eManifest
project. Actual costs are monitored and the eManifest burn rate is tracked to the original eManifest
budget.
Management recognizes these challenges to integrate all costs and forecasts by deliverable and
relate them back to the project benefits. EVM is one approach to accomplish this but requires the
capacity, tools and discipline to implement.
RECOMMENDATION 4
The Vice-President of Programs Branch should develop and implement a process to demonstrate
the benefits realization of eManifest.
MANAGEMENT RESPONSE
Programs Branch agrees with this recommendation. The eManifest project will complete a
draft report demonstrating the benefits realization of the eManifest project by January 2013.
The project will then submit this report to the Vice-President-level Projects Steering
Committee for approval by March 2013.
RECOMMENDATION 5
The Vice-President of Comptrollership Branch should provide assurance on the financial health and
the remaining cost forecast of the eManifest project and sign off as such on the dashboard. This
includes a discussion with the Programs Branch on the financial information requirement to provide
the necessary assurance.
12
MANAGEMENT RESPONSE
The Comptrollership Branch agrees with this recommendation.
Since the eManifest Review, Comptrollership has worked collaboratively with key
stakeholders (OPI, OCI and EPMO) to establish the necessary elements to allow for the
provision of quality assurance on the financial health.
An Internal Order structure (broken down by Components and Builds) has been established
and agreed upon. Costs and timelines will be assigned to each Component and Build to
further enhance the ability to monitor and report on the financial health, commencing April
1, 2013. This measure will help the Agency advance towards the principle of Earned Value
Management (EVM).
The Branch Chief Financial Officer will be signing off, and/or providing comments, where
necessary, on the monthly dashboards to confirm that an assurance on the financial health,
including the remaining cost forecast, has been conducted.
7.4 Project Status Reporting
Project Management Objective: Tools and processes for status reporting are used appropriately.
eManifest project management is required to report the project status to CBSA senior executives
and the Treasury Board Chief Information Officer Branch on a monthly basis. This is accomplished
through a project dashboard designed to report on the core project metrics of cost, schedule, scope,
risks and issues. Data requirements are stipulated for each metric within the Guide to Executive
Project Dashboards for Treasury Board Oversight Projects. For example, to report on the project
schedule, project management specifies the approved completion date, forecasted completion date
and variance by deliverable. According to the Treasury Board Secretariat’s guide, the variance
indicator is calculated based on the difference between the revised completion date (approved) and
the forecasted completion date.
The status of each metric is colour-coded using green, yellow and red indicators. Green indicates a
variance of less that 10%, yellow is a variance between 10 and 20% and red is a variance of over
20%. The eManifest Project Support and Control Office completes the project reporting process by
collecting the information from various systems, and documents and validates the information with
the Project Director's Steering Committee before it is inserted on the dashboard. As noted above,
the quality assurance of financial information is to be provided by the Comptrollership Branch. The
EPMO reviews the reports and provides a challenge function on the accuracy and completeness of
the reporting. The final dashboard is signed off by the Major Project’s Director General.
The guidance on roles and responsibilities is sufficiently defined and key stakeholders understand
their roles and responsibilities for approvals and quality reviews.
13
For eManifest, there are some deviations from Treasury Board Secretariat’s guidance for reporting
on the project schedule. The schedule section of the status reports asks that variances be reported
for key milestones or deliverables based on the approved completion date and forecasted
completion date. For eManifest, the EPMO has raised concerns regarding the definition of a
milestone or deliverable. Currently, the project management uses project phases (e.g. construction
and testing) as their definition for milestones or deliverables. The EPMO requests that the
dashboard reflect key deliverables by the remaining Builds. This would be more effective in
monitoring progress and potential slippage.
Additionally, the calculation of variance (slippage as per the Treasury Board Secretariat and EPMO
guide) deviated from the policy. Project management currently uses a project schedule that includes
the entire project development time for eManifest (2007 to 2014). The current project schedule was
baselined and approved in January 2012 with final project implementation by December 2014.
Therefore, the slippage calculation should be based on the revised baseline project duration.
During the course of the review, various stakeholders raised a couple of issues and they appear to
be of significance for reporting on the project dashboard:
Key regulations that needed to come into force by November 2012 to enforce the mandatory
requirement for highway carriers to transmit information electronically are not in place. As a
result the anticipated benefits have been delayed.
Related to the first point, there is a slow uptake of highway carriers to employ electronic
data interchange. As of October 2012, 3,339 out of 15,000 active carriers were registered
which account for 22%, however it represents 77% of total trade volumes. Of the remaining
non-registered carriers, 5,651 carriers had averaged less than one release per month of which
1,381 had only one release during the year. This could potentially create a demand from
carriers when the mandatory requirement is put in place which the agency is unable to deal
with in a timely matter.
Regarding project risks, project risks are identified on the dashboard and they include mitigation
strategies. In reviewing some of the mitigation strategies, it is questionable whether the mitigation
will achieve a desired outcome in response to the risk. For example, a project risk identified is the
resource capacity of the ISTB to deliver on eManifest in addition to other Agency projects such as
the Beyond the Border Action Plan and the CBSA Assessment and Revenue Management initiative.
The mitigation was that eManifest is the foundation of many initiatives and the Agency’s
commitment is to continue to implement eManifest and secure required resources. The review team
has heard that this risk is now manifesting itself. As such this should be noted as an open issue and
managed accordingly.
Summary
In regard to this project management objective, project management is reporting regularly on the
status of the project and the tools and dashboards are being used. Clarification is required on the
information that should be reported in the dashboard and the completeness of some of the
dashboard elements, particularly the risk and issues section. Oversight on the dashboard reports
14
could also be improved to ensure all significant information from CBSA stakeholders is reported.
The risk is that CBSA executive management may not be informed of significant issues that could
have broader implications for operations.
RECOMMENDATION 6
The Vice-President of Programs Branch should require the DG Projects Steering Committee to
review and discuss all project risks and issues, and include significant risks and issues on the
eManifest project dashboard.
MANAGEMENT RESPONSE
Programs Branch agrees with this recommendation. By January 2013, detailed risk and
issue logs will be standing items on the DG Projects Steering Committee agenda where they
will be reviewed and discussed, and significant items will be included on the Project
Dashboard.
15
APPENDIX A – ABOUT THE REVIEW
REVIEW OBJECTIVES AND SCOPE
The objective of this review was to determine the extent to which eManifest will meet the needs of
end users; the adequacy of the project plan with respect to managing the project’s scope, schedule
and resources; the management of costs; and the effectiveness of tools and processes for status
reporting.
The review scope included all aspects of the eManifest project; there were no exclusions. The
engagement was a review and not an audit as detailed testing was not undertaken. The level of our
work was commensurate with a review, which provides a low level of assurance.
RISK ASSESSMENT
A number of risk areas were identified in the planning phase of the review:
Scope: In the absence of adequate controls over scope management there are risks that the
project will fail to meet business and user requirements leading to the rejection of delivered
capabilities. Additionally, there is a risk that insufficient testing will result in unsatisfactory
systems to business users.
Project Management: In the absence of reasonable project management practices, there are
risks of undetected deviations in scope, schedule, and costs. Additionally, without consistent
project management practices there are risks of inconsistency across interdependent
projects.
Cost Management: In the absence of sufficient controls over costs, there are risks of a lack
of control over project costs resulting in misspending of IT investments.
Project Reporting: In the absence of satisfactory controls over project reporting there are
risks that reporting is ineffective on project progress which may lead to unidentified issues,
undetected project showstoppers or undetected project risks. There is also a risk of loss of
focus on customer expectations and business needs.
APPROACH AND METHODOLOGY
The review was based mainly on interviews and documentation reviews using two frameworks:
Control Objectives for Information and Related Technology (COBIT) version 4.1; and
Project Management Body of Knowledge (PMBOK).
COBIT identifies the IT processes that should exist to ensure that IT is aligned with and supports
the business in an effective manner. COBIT and its supporting publications identify control
objectives, techniques and practices commonly required for each process.
PMBOK identifies the best practice process for project management, together with the knowledge
and techniques required for those processes to be effective. PMBOK can be designed to be applied
to any industry, including IT.
16
THE REVIEW CRITERIA USED FOR THE ENGAGEMENT PHASE WERE:
LINES OF
ENQUIRY
CRITERIA SUBCRITERIA
1. Scope and
user
requirements
The project will
meet its scope and
the requirements of
end users.
Project Scope: The project scope is defined and
formally approved by the program and project
sponsors.
Project Change Control: The project plan reflects
changes to scope for approved changes, a
baseline is established and the project plan is
updated to reflect approved change requests.
User acceptance testing: The system meets the
requirements of the business end users through
successful user acceptance testing.
2. Project Plan The project plan is
sufficiently detailed
to guide the project
throughout its life
cycle.
Integrated Project Plan: The scope, deliverables,
required resources and responsibilities, work
breakdown structures, estimates of resources
required, milestones, key dependencies, and
identification of a critical path are documented in
the integrated project plan.
3. Cost
management Costs are
monitored;
deviations are
identified and
addressed in a
timely manner.
Cost Management: A cost management process
has been implemented comparing actual costs to
budgets. Costs are monitored and reported, where
deviations are identified and assessed in a timely
manner.
4. Project Status
Reporting Tools and processes
for status reporting
are used
appropriately
Roles and responsibilities related to status
reporting are clear and understood.
Project Reporting: Project status reporting
complies with the relevant organizational
standards.
17
APPENDIX B – GLOSSARY
Acceptance testing: Applying performance and capability measures to project deliverables
to ensure that they meet specifications and requirements and satisfy the customer.
Agile Project Management: This is a highly iterative and incremental process, where
developers and project stakeholders actively work together to understand the domain,
identify what needs to be built, and prioritize functionality. Agile methods are used when
these conditions are present: project value is clear; the customer actively participates
throughout the project; the customer, designers, and developers are co-located; incremental
feature-driven development is possible; and visual documentation (cards on the wall vs.
formal documentation) is acceptable.
Baseline: Original Plan (for a project, work package, activity), plus or minus any approved
changes. May be used with a modifier (for example, cost baseline, schedule baseline,
performance measurement baseline).
Baseline Measurement: Set of approved estimates for scope, cost and schedule that forms
the basis for change management during project execution. A project is baselined at the end
of the planning phase.
Build 1: Functionality that allows freight forwarders in all modes of transportation to
transmit pre-arrival detailed data (i.e. House bill) is expected for March 2013.
Build 2: Functionality for commercial passage and core risk assessment is expected for July
2014. It will also include the functionality to allow importers to transmit Advance Trade
Data.
Build 3: Enhancements to Risk Assessment and Passage are expected in December 2014,
and will include enhancements to the commercial passage and risk assessment
functionality, as well as new functionality for crew risk assessment in all modes.
Milestone: 1. An event with zero duration and requiring no resources. Used to measure the progress of
a project or program and signify the completion or start of a major deliverable or other
significant metric such as cost incurred, hours used or payment made.
2. Identifiable point in a project, program or set of activities that represents a reporting
requirement or completion of a large or important set of activities.
Project Scope Management: Includes the processes required to ensure that the project
includes all the work required, and only the work required, to complete the project
successfully. Project scope management is primarily concerned with defining and
controlling what is and what is not included in the project.
18
APPENDIX C – ACRONYMS
CARM - CBSA Assessment and Revenue Management
CAS - Corporate Administrative System
CBSA - Canada Border Services Agency
COBIT - Control Objectives for Information and Related Technology
COMPASS - Compliance Measurement, Profiling and Assessment System
DG - Director General
EPMO - Enterprise Project Management Office
EVM - Earned Value Management
ISTB - Information, Science and Technology Branch
IT - Information Technology
OCI - Office of Collateral Interest
OPI - Office of Primary Interest
PMBOK - Project Management Body of Knowledge
SME - Subject Matter Expert
UAT - User Acceptance Testing