CITY AUDITOR'S OFFICE Software Acquisition and …...This audit was included on the City Council-approved fiscal year (FY) 2016/17 Audit Plan as a contracted information technology

CITY AUDITOR'S OFFICE

Software Acquisition and Implementation March 13, 2017 AUDIT REPORT NO. 1708

CITY COUNCIL Mayor W.J. “Jim” Lane Vice Mayor Suzanne Klapp Virginia Korte Kathy Littlefield Linda Milhaven Guy Phillips David N. Smith

TABLE OF CONTENTS

AUDIT HIGHLIGHTS ......................................................................................... 1

BACKGROUND ............................................................................................... 3

Figure 1. Information Technology Department Organization ............................................... 3

Figure 2. Software Acquisition Key Roles ....................................................................... 5

Figure 3. Software Implementation Phases ..................................................................... 6

Figure 4. Grant Thornton Assessment Framework ............................................................. 7

OBJECTIVES, SCOPE, AND METHODOLOGY ............................................................. 11

FINDINGS AND ANALYSIS .................................................................................. 13

1. Strategy Management ............................................................................... 13

2. Benefits Realization Management ................................................................. 14

3. Information Technology Governance .............................................................. 15

4. Project Management ................................................................................ 16

Figure 5. PMBOK Project Management Processes and Knowledge Areas ................................ 17

Table 1. PMBOK Project Management Groups and Knowledge Area Mapping .......................... 18

Table 2. Potential Role Assignments for Project Management ............................................ 21

5. Procurement and Contract Management ......................................................... 22

6. Solution Delivery ..................................................................................... 24

7. Maintenance and Operations ....................................................................... 25

MANAGEMENT ACTION PLAN ............................................................................. 27

Page 1

This audit was included on the City Council-approved fiscal year (FY) 2016/17 Audit Plan as a contracted information technology (IT) audit. We contracted with Grant Thornton, LLP, to evaluate the City’s software acquisition and implementation processes by reviewing recent information system projects.

The IT department’s Project Management Office assists other departments with software acquisition of systems specific to the business area’s operations. Departments can request implementation assistance. The user department typically assigns a contract administrator to manage the project, including the acquisition and implementation phases. The Purchasing and Risk Management departments and the City Attorney’s office, are involved for matters related to their areas of responsibility. In addition to Grant Thornton reviewing IT policies and processes, we selected four recent acquisitions for a more in-depth review: •ActiveNet – Community Services system used to manage class registrations, facility reservations/bookings, and cashiering services. •HSCAMS – Human Services Client Assistance Management System used to track client records and schedule meetings. •Munis – Citywide human resources and payroll system. •Fleetmind – A combined hardware and software solution used by Solid Waste intended to optimize collection practices.

AUDIT HIGHLIGHTS

WHY WE DID THIS AUDIT

BACKGROUND

Software Acquisition and Implementation March 13, 2017 Audit Report No. 1708

WHAT WE FOUND Projects lacked consistent guidance and oversight. While the IT Project Management Office provided needed assistance during software procurement activities, there was not consistent guidance for department management and contract administrators during project initiation and software implementation. As a result, some key tasks were not being consistently completed:

• Goals and desired outcomes were not clearly defined when a project was initiated and later evaluated at project closeout.

• Adequate market research of alternatives was not documented prior to choosing sole-source (non-competitive) procurements.

• Quality was not being monitored or controlled by requiring approved software testing plans and review of testing results.

Further, contract administrators may not have sufficient prior experience or training and guidance for effective project management of software acquisition and implementation. Specifically, contract administrators were not always conducting:

• Risk management activities, such as risk identification, analysis and planning risk responses.

• Organizational change management activities, such as preparing users for the new systems, overcoming resistance to changes and influencing attitudes about the new system.

• Verification of project deliverables prior to phase or project acceptance.

Projects did not maintain or create necessary documentation. None of the projects were able to provide all of the necessary documentation. Critical items that were not maintained or created included: proposal evaluation results, project status reports, test plans, test results, infrastructure plans, and security plans. In some cases, the contract administrator was unaware that some of these documents were contractually required.

WHAT WE RECOMMEND We recommend the Information Technology department require the Project Management Office (or equivalent) to:

• Establish standard processes, including planning templates, to assist department contract administrators in managing software acquisitions and implementation.

• Provide more consistent oversight of the implementation phase as well as the acquisition phase.

MANAGEMENT RESPONSE The Information Technology and Purchasing departments generally agreed with the recommendations. City Auditor’s Office

City Auditor 480 312-7867 Integrity Line 480 312-8348

www.ScottsdaleAZ.gov

Page 2 Audit Report No. 1708

Software Acquisition and Implementation Page 3

BACKGROUND

This audit of Software Acquisition and Implementation was included on the City Council-approved fiscal year (FY) 2016/17 Audit Plan as a contracted information technology (IT) audit. We contracted with Grant Thornton, LLP, to perform an evaluation of software acquisition and implementation processes.

The City’s IT department is responsible for providing technical design, support and maintenance for a variety of systems and services needed to support city business functions and communications. As shown in Figure 1, the Department staffs an IT Project Manager within the Technology Infrastructure section to coordinate IT projects, including software acquisitions and implementation.

Figure 1. Information Technology Department Organization

SOURCE: Auditor analysis of IT department organizational chart.


Software Acquisition

In addition to managing projects within the IT department (projects and systems related to technology infrastructure or affecting multiple business areas), the IT Project Management Office (PMO) may assist other departments with software acquisition and implementation of systems specific to the business area’s operations. At the time of the audit, the IT PMO consisted of 1 full-time project manager, who retired in late December 2016.

Typically, the user department identifies a business need for a software solution and performs the initial market research to learn more about the types of software applications available. The department may or may not engage the IT PMO at this point. Larger departments have their own technology support staff and will assign one as the project manager/contract administrator (CA). For small departments without technology personnel, the IT Project Manager will provide project management. The recently retired IT Project Manager was a certified Project Management Professional, though when there is a department project manager/CA, the IT PM takes an advisory role.

According to City Procurement Code, software purchases over $25,000 must follow the Request for Proposal (RFP) process. The Purchasing department executes the RFP process, but relies on the IT PMO to coordinate technology software, hardware, and Software as a Service (SaaS) procurements. The IT PMO often will assist the user departments in drafting the RFP scope, and over time, has developed standard RFP requirements based on IT infrastructure and security requirements and template contract terms and conditions for software acquisitions.

While Risk Management and the City Attorney’s office generally review all City contracts, they are significantly involved in the negotiations of Software as a Service (SaaS) contracts. Unlike traditional “off-the-shelf” software products that are hosted (installed and operate) on City servers, SaaS systems may rely on the vendor for application hosting, data storage and/or processing, making data security a concern. However, SaaS vendors are resisting the City’s standard liability and indemnity contract terms and thus, contract negotiations are requiring more Risk and Legal involvement.

(continued on next page)


Figure 2. Software Acquisition Key Roles

Source: Auditor analysis of interviews with the various departments.

The Contract Administrator (CA) is typically a technology staff within the user department. The CA also serves as the Project Manager for the system being acquired.

The user department engages IT Project Manager to assist. The IT Project Manager acts as liaison between the department and Purchasing, providing technical expertise in writing RFP requirements and reviewing proposals for compliance with IT department infrastructure and security requirements.

Purchasing staff review for procurement policy compliance.

Risk Management and the City Attorney’s office are involved in contract development and negotiations, primarily focusing on liability and indemnity contract terms.

All technology hardware and software purchases, including peripherals, must be approved by IT. For software purchases, the Chief Information Security Officer must also review the contract’s security terms.

Department identifies the business need that may be addressed by a software solution. Department then initiates market research to identify potential solutions.


Software Implementation

The IT PMO may or may not be involved in the implementation phase of a software system. According to the PMO, the department can request assistance, though it has been nearly twelve years since the project manager has managed an implementation. The implementation process is often vendor-driven, and the department’s contract administrator is responsible for monitoring the process. According to the IT department, the City rarely customizes vendor solutions.

Software implementation is often identified as having several significant phases, with one analysis shown in Figure 3. Clockwise, the phases include identifying the desire to change (business need) and forming an implementation team, developing and executing the implementation plan, and reviewing effectiveness of the solution. Each of these phases should be guided by policies and procedures to help ensure successfully addressing the business need.

Figure 3. Software Implementation Phases

SOURCE: COBIT® 5, A Business Framework for the Governance and Management of Enterprise IT (figure 17) and COBIT® 5 Implementation (figure 6) from ISACA.


Many standards, guides and best practices have developed for software project phases; Figure 4 summarizes the framework that our contracted specialists, Grant Thornton, used. These include established resources such as the Project Management Institute’s (PMI) body of knowledge and standards for project and program management and the CMMI Institute’s model for assessing the processes used in software acquisition.

Figure 4. Grant Thornton Assessment Framework

Phas

es

Planning and Project

Initiation

Project Management

Implementation and Transition Guide/Standard/Methodology/Best Practice

Asse

ssm

ent

Cate

gori

es

Strategic Management

Managing Successful Programmes (MSP) Benefits Realization, Association for Strategic Planning (ASP) Body of Knowledge, Project Management Institute (PMI) Standard for Program Management 3rd Edition

Benefits Realization Management MSP Benefits Realization

Governance PMI Standard for Program Management 3rd Edition

Project Management PMI A Guide to the Project Management Body of Knowledge (PMBOK® Guide)

Procurement and Contracts Management Capability Maturity Model Integration (CMMI®) - Acquisition

Organizational Change Management and Training

Prosci ADKAR Model (Awareness, Desire, Knowledge, Ability, Reinforcement)

Risk and Issue Management PMBOK®

Solution Delivery

Industry Best Practices (Agile, Waterfall, RUP), CMMI® – Software Engineering, The Open Group Architecture Framework (TOGAF), Quality Assurance Institute Common Body of Knowledge (CBOK)

Quality Management American Society for Quality (ASQ) Guide to the Quality Body of Knowledge (QBOK)

Maintenance and Operations IT Infrastructure Library (ITIL) 4th Edition

Focu

s

Past Present Future

SOURCE: Grant Thornton, LLP

The strategic management category assesses the analysis, decisions and actions taken to create and sustain competitive advantages. To evaluate the strategy, Grant Thornton used Managing Successful Programmes (MSP) and the Project Management Body of Knowledge (PMBOK). MSP provides a framework for managing a portfolio of projects, referred to as a program, while PMBOK provides a framework and standard for managing projects.

Benefits realization management involves the process of organizing and managing so that potential benefits arising from an investment in change are actually achieved. Grant Thornton used MSP for this evaluation as it focuses on defining “benefits” early and measuring success


against the defined benefits. Specifically, MSP defines a benefit as a “measurable improvement that results from an outcome identified as an advantage by one or more stakeholders. Benefits contribute towards organizational objectives.”

Governance consists of systems and methods by which a program and its strategy are defined, authorized, monitored, and supported by its sponsoring organization. Project governance is the framework used to apply a consistent set of rules and processes for a project. Control and governance requirements vary among organizations based on a variety of factors, such as structure, size and culture. Currently the City of Scottsdale uses a highly decentralized model, resulting in the contract administrator being responsible for creating the project governance framework. The PMI, in its Standard for Program Management, recommends 11 elements be included in a governance framework:

• Success and acceptance factors • Issue resolution • Roles • Responsibilities • Communication processes • Strategic alignment • Project life-cycle approach • Phase/stage review process • Change management • Stakeholder management

Project management is the application of knowledge, skills and techniques to execute projects effectively and efficiently. The PMI includes ten knowledge and five process groups in its PMBOK. The knowledge areas identify the key project management factors, such as integration, scope, time, cost, quality and others. The process groups are the stakeholders who will be responsible for initiating, planning, executing, monitoring and controlling, and closing each of those project management areas.

Procurement and contracts management considers the practices used to purchase or acquire the products, services or results needed from outside the project team to perform work. Grant Thornton evaluated these practices based upon the CMMI Acquisition standard, which states that procurements should clearly align with broader business objectives. Once signed, the contract should be actively managed so that there are no discrepancies between work performed and the contractual requirements. In the case when business needs change, the change approval process should follow a consistent set of procedures and is often documented in a contract modification.

The organizational change management and training category evaluates the application of a structured process and set of tools for leading the “people” side of change to achieve a desired outcome. For this, Grant Thornton relied on the Prosci ADKAR methodology, which stands for Awareness, Desire, Knowledge, Ability and Reinforcement. These represent the five outcomes that must be achieved for a change to be successful.

Risk and issue management reviews processes concerned with conducting risk and issue management planning, identification analysis, responses and monitoring and control on a project. PMBOK stresses that risks should be planned, not encountered. Therefore, most risk management activities are performed during the planning process. Further, risk management is supposed to assess both qualitative and quantitative factors that may affect reputation and/or finances.


Solution delivery corresponds to the cross-cutting aspects of solution development, such as security, caching, data access, validation, and exception management, among others. Grant Thornton used the CMMI framework in evaluating solution delivery. Three of the four evaluated projects used Software as a Service (Saas) solutions that required minimal customization and configuration. Therefore, a number of aspects that would be required in larger, customized solutions were not applicable.

Quality management represents the processes and activities that determine quality policies, objectives and responsibilities. The American Society for Quality states in ISO 9000 that quality management systems should incorporate four factors: identified processes, responsibilities, procedures (implemented and maintained) and effective processes. Grant Thornton looked for these quality management techniques in the selected software projects.

The maintenance and operations category consists of the processes and activities to ensure that IT services are delivered effectively and efficiently. Using the ITIL standards, Grant Thornton reviewed how these software tools were managed, monitored and supported.



OBJECTIVES, SCOPE, AND METHODOLOGY

An audit of Software Acquisition and Implementation was included on the City Council-approved fiscal year (FY) 2016/17 Audit Plan as a contracted information technology (IT) audit. The audit objective was to evaluate the effectiveness of the City’s software acquisition and implementation policies and practices.

We contracted with specialists from Grant Thornton, LLP, to perform this audit. As required by Government Auditing Standards, we evaluated the qualifications and independence of the specialists and documented the nature and scope of the specialists’ work, including the objectives and scope of work, intended use of the specialists’ work to support the audit objectives, and the specialists’ procedures and findings.

To achieve the objective, the audit team reviewed the City’s current policies and procedures for software acquisition and implementation to determine whether the City follows industry standard practices. Further, taking a case study approach, the team performed an in-depth review of selected recent system acquisitions. We selected several systems identified through recent IT risk assessments and performance audits:

• ActiveNet—Community Services uses this system to manage class registrations, facility reservations/bookings, and cashiering services. This system was acquired in 2015 through sole-source procurement.

• Human Services Client Assistance Management System (HSCAMS)—The Human Services department, within Community Services, uses this case management system to track client records and schedule client meetings. This system was acquired in 2014 through a Request for Proposals (RFP) process.

• Munis—The human resources and payroll software of this Enterprise Resource Planning (ERP) system is to replace existing systems and processes to enhance the work flow and automation of the Human Resources department and Payroll staff. The software was acquired in 2016 through an RFP process, and implementation is still underway.

• Fleetmind—A combined hardware and software solution, this system intends to optimize solid waste collection practices. The Solid Waste department purchased the hardware through a regional cooperative purchasing agreement, and then acquired the software through sole-source procurement in 2014.

To gain an understanding of the City’s policies and practices, auditors reviewed City administrative regulations and procurement code, and the IT department’s documented policies and procedures related to system acquisition and implementation. Auditors also interviewed representatives from the IT department (including the Chief Information Officer, the IT Project Manager, Technology Director, and Chief Information Security Officer), the contract administrators involved in the selected implementations (Community Services Systems Integrator, Finance IT Director, and Solid Waste Business Analyst), as well as the Risk Management Director and Sr. Assistant City Attorney involved with these contracts.

For the selected systems, auditors obtained and reviewed available project documentation, including bid proposals, contracts, proposal evaluations, project charters, project planning documents, testing plans and results, status reports, risk management plans, design documents, and other related records.

To assess the City’s practices, Grant Thornton compared them with the standards and best practices in the firm’s IT Assessment framework. Illustrated in Figure 4 on page 7, the


framework encompasses a set of best practices across ten different categories of software acquisition and implementation.

The audit found that projects lacked consistent guidance and oversight, contract administrators were not always performing key project management tasks, and necessary documentation was not maintained.

We conducted this audit in accordance with generally accepted government auditing standards as required by Article III, Scottsdale Revised Code §2-117 et seq. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Audit work took place from November 2016 to February 2017.


Project Charter

A planning document that defines roles and responsibilities, business needs, project objectives, stakeholders, assumptions and desired outcomes. It establishes an agreement between the project manager and project sponsors regarding shared understanding and desired outcomes. Approval of the project charter by project sponsors formally authorizes the project start.

Strategy Management Document

Defines: • Current state evaluation • Known problems and opportunities • Desired future state • Analysis of alternatives • Strategy and action plan

FINDINGS AND ANALYSIS

1. Strategy Management

Using the Managing Successful Programmes (MSP) and Project Management Book of Knowledge (PMBOK) established standards, strategy management was evaluated for the four selected software acquisitions.

A. There is not a documented IT strategic plan to guide acquisitions. However, three of the four projects were Software-as-a-Service (SaaS) solutions, and management has indicated that their use of SaaS has grown in recent years. A strategic plan could make the business case for these tools and provide guidance for choosing the most appropriate solution.

B. A project charter template serves as the strategy document for software projects.

1. Only two of the four projects prepared the project charter. For these two charters, the information provided was not responsive to the required elements. It appears the project charters are not reviewed for adequacy. Further, one appeared to have been created after sole-source procurement had been decided but did not include evaluation of competitive alternatives.

2. The existing template contains only two of the five elements that should be included as part of the strategy: known problems and analysis of alternatives. Three key elements were not included: current state evaluation, desired future state, and a strategy and action plan.

The IT department’s Project Management Office, consisting of one staff, provides support during software acquisition and, if requested, implementation. The experienced staff in this position recently retired. Information system staff in the various departments serves as project managers and contract administrators during all phases.

Recommendations:

We recommend the Information Technology department require the Project Management Office (or equivalent) to:

A. Develop a strategy management template or expand the project charter template to incorporate the five strategy management elements, including:

• current state evaluation, • desired future state, and • strategy and action plan.


Also, develop a completed sample that demonstrates the type of information to be provided.

B. Provide strategy management training to information system support staff who serve as project managers and contract administrators for department software projects and the project teams.

C. Review the strategy document and project charter for completeness and adequacy during the software acquisition phase before the solution is selected.

2. Benefits Realization Management

Benefits realization management is the process of organizing and managing so that potential benefits arising from an investment in change are actually achieved. However, there are limited processes in place to evaluate the benefits of a software acquisition and implementation project.

The Project Charter template contains a section to document benefits of the project. Use of this template is not required as only two of the four projects reviewed had a Project Charter. Of these, one had vague benefits listed (e.g. “time”, “money”) and the other listed reasons for the procurement method (sole source) rather than the business value of the solution. A third project separately documented a Return on Investment (ROI) analysis; however, this analysis was provided by the vendor and the department did not validate its accuracy.

Some strategies from the Managing Successful Programmes (MSP) framework recommended by Grant Thornton include:

• Benefits identification and changes – Review ways to identify potential benefits, such as through City strategy documents and/or meeting with stakeholders to understand their perspectives.

• Benefits tracking – Create a benefits realization plan to understand how and when project benefits should happen.

• Benefits optimization – Identify lessons learned from previous projects that may enhance benefits.

• Managing benefits performance – Create key performance indicators to understand and measure benefits realization.

• Documenting and tracing benefits – Creating benefits maps to illustrate how benefits are related and how they trace to capabilities, outcomes, and objectives.

Recommendations:


• Update the project charter to clarify that benefits should include measurable improvements tied directly to stakeholder and/or organizational objectives.

• Require contract administrators to track benefits realization throughout the course of the project and report progress to the governance (steering) committee.


3. Information Technology Governance

Project governance is the framework used to apply a consistent set of rules and processes for a program or project, and the requirements would be tracked in a project management plan. Control and governance requirements may vary among organizations based on a variety by factors, including size, structure and culture. Because the IT department uses a decentralized model for project management, the user department’s contract administrator is responsible for creating the project governance framework, resulting in inconsistent application.

A. Goals and objectives are not well defined, and acceptance testing is not documented.

Best practices indicate success should be defined by “improved business outcomes,” rather than simply by the software being actively in use. These success factors should be defined in the project charter, and can then drive acceptance testing plans and help to ensure successful outcomes.

The City’s standard contract requires a written “Acceptance Test Plan” to verify that the product operates in accordance with the requirements. Acceptance testing may be done at the end of each project phase or at other logical points as detailed in the plan. Then, after evaluating test results, the standard contract terms require formal written acceptance before the project moves to the next phase or is closed out.

However, three of the four reviewed projects were unable to provide acceptance testing plans or the test results; one project was not yet at this stage. Further, some of these contract administrators were unaware of the Acceptance Test Plan requirement.

B. Governance roles and responsibilities could be better defined to provide project oversight.

While the project charter template identifies the key roles of project sponsor, project owner, and project manager, it does not define their responsibilities.

• For the four projects reviewed, some contract administrators were not fully aware of their project management responsibilities. For example, three of the four contract administrators were not aware of the specific deliverables listed in their contracts. Contract administrators are not required to provide evidence of phase completion or deliverables before authorizing payments to the vendor. For three projects, excluding the HR/Payroll system still in the implementation phase, auditors did not find evidence that contract deliverables were verified before payment.

• Additionally, software acquisition and implementation projects typically have decision-making hierarchies, but the project charter template does not require approval levels to be defined. Further, the City’s standard contract language indicates all decision-making responsibility resides with the contract administrator.

• Traditionally, IT projects report progress to a governance committee, however only one project held regular steering committee meetings. The other project managers stated they provided irregular updates to the sponsors and/or steering committees. Updates were typically regarding issues and timelines, but not on success factors such as vision, goals, and outcomes.


As well, a stakeholder management plan would have helped the Fleetmind and HSCAMS projects to more effectively implement their solutions. In both of these cases, the primary stakeholders (system users) resisted the new tools and may have played a significant role in implementation delays. Seeking stakeholder input and buy-in may have driven greater success.

Recommendations:


A. Provide guidelines for and oversight of the Acceptance Test Plan development and execution.

B. Define responsibilities of the key roles in project oversight and governance, ensure they are established, and monitor progress, including status updates, deliverables acceptance and evaluation process.

4. Project Management

The acquisition and implementation of software systems are finite projects with a specific goal (successful system implementation) and defined scope and resources, and the project team often includes individuals from the vendor, different departments and various organizational roles. As such, project activities have to be properly managed to meet the project goals.

Grant Thornton used project management standards established by the Project Management Institute (PMI) in the Project Management Body of Knowledge (PMBOK). As shown in Figure 5 on page 17, the PMBOK identifies five process groups and ten knowledge areas that are important for effective project management.



Figure 5. PMBOK Project Management Processes and Knowledge Areas

SOURCE: Auditor illustration of PMBOK project management processes and knowledge areas.

A. More oversight from the IT PMO is needed for department-managed projects.

While the City provides contract administrator training, it does not equip contract administrators with the skills to perform project management duties. The Project Management Institute requires a minimum of 4,500 hours of experience and 35 hours of project management education before attaining project manager certification. However, contract administrators are sometimes assigned to perform the duties of a project manager without prior experience leading projects of similar size and scope or relevant training.

Further Table 1 on page 18 summarizes the City’s performance based on the PMBOK framework of processes and knowledge areas. As shown, projects performed better in the earlier process areas, primarily because the contract administrator has additional assistance from the IT PMO. Once the contract is signed, the IT PMO does not provide any formal assistance to the contract administrator, but will provide support and guidance if requested.

As the shading in Table 1 also shows, some of these four projects performed many of the tasks in the Initiating and Planning process groups, but did not perform (or performed ineffectively) tasks in the Executing, Monitoring & Controlling, and Closing processes when the IT PMO was no longer actively involved. Additionally, many of these tasks were vendor-led with inadequate monitoring by the contract administrator.


Table 1. PMBOK Project Management Groups and Knowledge Area Mapping

Knowledge Areas

Project Management Process Groups

Initiating Planning Executing Monitoring and Controlling Closing

Integration Management

Develop Project Charter

Develop Project Management Plan

Direct and Manage Project Work

Monitor and Control Project Work

Perform Integrated Change Control

Close Project or Phase

Scope Management

Plan Scope Management

Collect Requirements

Define Scope

Create Work Breakdown Structure

Validate Scope

Control Scope

Time Management

Plan Schedule Management

Define Activities

Sequence Activities

Estimate Activity Resources

Estimate Activity Durations

Develop Schedule

Control Schedule

Cost Management

Plan Cost Management

Estimate Costs

Determine Budget

Control Costs

Quality Management

Plan Quality Management Perform Quality Assurance Control Quality

Human Resources Management

Plan Human Resource Management

Acquire Project Team

Develop Project Team

Manage Project Team

Communication Management

Plan Communication Management

Manage Communications

Control Communications

Risk Management

Plan Risk Management

Identify Risks

Perform Qualitative Risk Analysis

Perform Quantitative Risk Analysis

Plan Risk Responses

Control Risks

Procurement Management

Plan Procurement Management

Conduct Procurements Control Procurements Close

Procurements

Stakeholder Management

Identify Stakeholders

Plan Stakeholder Management

Manage Stakeholder Engagement

Control Stakeholder Engagement

Green – Performed Yellow – Partial Performance Red – Ineffective or non-performance

SOURCE: Grant Thornton analysis of PMBOK project management standards.


B. Improvements could be made to the following project management areas:

1. Risk Management – The PMBOK stresses that risks should be planned, not encountered. Therefore, most risk management work is performed in the planning process. A project manager should create a plan to manage risks, by identifying them, performing qualitative and quantitative risk analysis, and planning risk responses. To control risks, the identified risks should be reviewed regularly for likelihood, impact, and effectiveness of response.

Auditors reviewed whether contract administrators planned for and monitored risks, and how any resulting issues were mitigated.

One of the four projects had a vendor-developed risk management plan and a project risk log. The risk management plan did not include qualitative and quantitative risk analysis that would meet the PMBOK standards. Because the project was still in the early stages of implementation, it was not clear yet whether the risks would be monitored and controlled. Three of the reviewed systems did not develop a risk management plan and did not maintain a risk log or other record of issues that occurred. Further, this is not an area commonly covered in contract administration training; therefore some assigned contract administrators may not be aware of this practice.

2. Quality Management – A quality management system is a way of defining how an organization can meet the requirements of stakeholders affected by its work. It encompasses policies, objectives and processes.

Two of the four projects had a quality management plan.

• The first indicated that the project manager would measure quality against standards, but did not specify which standards would be used. The plan also stated that the quality standards would be approved by the project sponsor and communicated to stakeholders. But there was no evidence that this occurred.

• The second project, though still in its early implementation phase, had a quality management plan with relevant details on how the team will manage the quality software delivery, including nine different types of testing. Grant Thornton noted that the testing approach described appeared rigorous and, if followed, should be effective.

Establishing a quality management system or plan is not currently required for the City’s software implementations.

3. Human Resources Management – while the vendors listed internal roles and responsibilities for projects, the City does not create roles and responsibilities for its staff. This is problematic because interview feedback indicated that contract administrators are still responsible for their typical workload. This may be reasonable for smaller projects, but for larger and critical system implementations, such as the HR/Payroll system, this poses a potential risk. Also, time commitments may conflict with other projects and cause delays. One project paid up to six months of unnecessary licensing fees before the product was installed due to staff schedule conflicts with another project.

4. Organizational Change Management - Organizational change management (OCM) refers to the practices used to lead the “people” side of change management to


achieve a desired outcome. For software implementation, this means preparing users for the new system, overcoming resistance to change and influencing attitudes about the new system. Though OCM activities may be performed by the City or by the vendor, the audit team did not find evidence that these activities were being performed.

For two of the reviewed systems, user acceptance issues appear to have caused delays to any benefits that may have been expected by implementing the new systems.

• One system was not fully adopted by the department staff until 12 to 18 months after installation because of employees’ resistance to adopting the new processes.

• Another system has yet to be effective or functional almost two years after installation. The project manager believes some of the issues are related to employees’ attitudes about the new system and unwillingness to use the equipment properly.

The City does not have a formal change management requirement for IT projects, and the contract administrators did not recognize it as necessary or within their scope of work.

Table 2 on page 21 summarizes Grant Thornton’s analysis of possible role assignments in the Project Management process groups and knowledge areas.



Table 2. Potential Role Assignments for Project Management

Knowledge Areas

Project Management Process Groups

Initiating Planning Executing Monitoring and Controlling Closing

Integration Management

Lead: IT PMO Support: Contract administrator

Lead: Vendor Support: Contract administrator, IT PMO

Lead: Contract administrator Support: IT PMO

Lead: IT PMO Support: Vendor, Contract administrator


Scope Management


Lead: IT PMO Support: Contract administrator, Vendor

Time Management



Cost Management



Quality Management


Lead: Vendor Support: Contract administrator


Human Resources Management



Communication Management




Risk Management



Procurement Management

Lead: IT PMO Support: Purchasing, Contract administrator

Lead: Purchasing Support: IT PMO, Contract administrator, Risk management, Legal


Lead: Contract administrator Support: Purchasing, IT PMO

Stakeholder Management





SOURCE: Grant Thornton analysis.

Recommendations:


A. Lead tasks in other IT project management areas, particularly monitoring and controlling processes, currently led by the contract administrator.

B. Assist departments’ IT project management, by providing support:

1. Establish guidelines requiring the use of a project charter at project initiation.

2. Develop a project risk management plan template with clear criteria for risk categorization and escalation and provide risk management training to contract administrators. As well, the PMO should assist in monitoring and responding to project risks.


3. Develop a quality management plan template that demonstrates well-defined quality management processes, roles and responsibilities, and performance measures. Review and approve the quality management plans and provide oversight of the quality of project deliverables.

4. Establish guidelines requiring the creation of a resource plan at project initiation. The plan should include an estimate of internal resource requirements and how the time spent on the project will be mitigated for regular City operations.

5. Establish an organizational change management plan template to be prepared by the project manager or vendor.

5. Procurement and Contract Management

The CMMI Acquisition standards state procurements should clearly align with broader business objectives. Once the contract is signed, active contract management is necessary to monitor that work performed matches the contractual requirements and contracts are formally modified when there are changes to business needs. However, the audit found that:

A. Procurement evaluation criteria and scoring is not retained.

For these projects, Purchasing only retained the final total score for each vendor. According to the Department, all documentation of evaluation panel members, evaluation criteria and individual scoring details are destroyed three months after the contract award to protect the evaluators’ identity. Besides retention of procurement documentation being required by state public records laws, this practice may inhibit the City’s ability to negotiate and evaluate contract changes.1 If a vendor was awarded points for its proposed solution and later fails to deliver, the evaluation materials may help to determine an appropriate penalty. Further, evaluation criteria can serve as a tool to consider lessons learned and improve future procurement.

B. Performance standards are not included in the standard Request for Proposal (RFP) template.

While the City’s RFP template incudes seven of eight CMMI criteria (terms and conditions, a statement of work, requirements, technical standards, development milestones, acceptance criteria and delivery dates), it does not include performance standards. These standards may be based on cost, schedule and quality requirements. For example, there may be a performance standard that 99.5% of data must be successfully converted to the new system.

C. For the sole-source purchases reviewed, market research was not adequately performed.

Two of the reviewed software acquisitions were completed as sole-source. This means the City determined only one vendor could provide the product/services and a competitive purchasing process could not be conducted. However, no market research was documented for one system, although there were several reasonable alternative

1 Access to public records is covered in ARS Title 39. Procurement records are specifically addressed in ARS §41-2550 and are required to be retained according to approved retention guidelines. The City’s records retention schedules state that contract-related records should be retained for six years after contract fulfillment or cancellation.


solutions in the marketplace. In the second instance, the contract administrator evaluated more than 20 vendors solely based on internet research, without obtaining information directly from the vendors. From the top choices among these, department staff selected a vendor and directly procured specialized hardware through a regional cooperative agreement. The software needed for the selected hardware was then handled as a sole-source procurement.

D. For all Software as a Service (SaaS) solutions, IT requires vendors to submit a SaaS questionnaire prior to the contract approval. SaaS systems may rely on the vendor for application hosting, data storage and/or processing, and typically has some web-based component, making data security a concern. The SaaS questionnaire developed by IT asks questions about security of City data, backup and recovery procedures, data storage and encryption, and disaster recovery.

• The Information Services and Information Security staff review and approve the vendor’s responses based on professional judgment that the vendor solution complies with the City’s requirements. Formal City standards have not been established although best practices indicate that standards, such as backup and recovery, should be driven by criticality. IT leaves this judgment to the user departments, and based on interviews, contract administrators vary in experience levels.

• Because usually only one RFP response is sent to IT for review, it appears the SaaS questionnaire is reviewed after the RFP evaluation process has been completed. Thus, the security component is not part of the evaluation criteria.

E. Contract management practices could be improved.

1. For three of the four projects, documentation was not retained to verify that contract requirements were followed and deliverables were provided. The critical items that were not available include: evaluation results, project status reports, test plans, test results, infrastructure plans, and security plans. This impaired the City’s ability to evaluate the quality of the vendor’s work. Also, lack of software documentation shortens the lifespan of software tools, inhibiting the City’s ability to identify and make improvements, and creates the potential for production errors and vulnerabilities.

2. An internal kick-off meeting including the contract administrator, IT PMO, Purchasing, and Risk Management would help to share information and review key points of the contract requirements. Often, the project is assigned to the contract administrator after the award but some contract administrators were not experienced managing these types of contracts, as demonstrated by the lack of evidence for acceptance testing or deliverables. Also, in some cases, it is not clear whether negotiated risk management and legal risks were explained to the contract administrator.

Recommendations:

A. We recommend that the Purchasing department retain software RFP evaluation materials in accordance with public records law and record retention guidelines.


We recommend that the Information Technology department require the Project Management Office (or equivalent) to:

B. Work with the Purchasing department to incorporate performance standards into the RFP template.

C. Work with the Purchasing department to establish standards for and review software market research prior to approving sole-source purchases.

D. Establish SaaS criteria and standards, and require the SaaS questionnaire to be included in the RFP evaluation process of each vendor’s proposal.

E. Facilitate contract administration and improve oversight by assisting the contract administrator to:

1. Maintain a contract file that includes all contract-related documentation, including the approved deliverables.

2. Require the vendor to produce deliverable evaluation documents including agreed-upon acceptance criteria, prior to acceptance.

3. Hold an internal kick-off meeting after the vendor contract is signed, including the contract administrator and involved staff from the IT PMO, Purchasing, Legal and Risk Management.

6. Solution Delivery

According to the IT department, the City does not govern the Software Development Life Cycle (SDLC) for vendors or require them to provide their own. However, the City’s contract has SDLC-like elements that require the vendor to provide project planning, requirements definitions, software installation, infrastructure planning, security planning, software configuration, data conversion, testing, performance tuning and startup. Despite the contract deliverables for these requirements, there was no evidence the vendors conducted all of these phases, excluding the project still in the implementation phase.

As mentioned previously, three of the projects reviewed lacked acceptance testing plans and test results to demonstrate that the software met functional requirements. These are critical elements for evaluating the quality of the solution that was delivered.

Recommendations:


A. During the proposal phase of the project, verify the vendor has a suitable SDLC.

B. Provide oversight for product testing and approve phase acceptance based upon a contractual checklist.



7. Maintenance and Operations

The maintenance and operations standards review how software tools are managed, monitored and serviced.

A. Problem resolution can be better managed.

• IT has an enterprise change and issue management tool, but it is not used for department-based tool problem management. Instead, each software project requires its own problem and issue management system.

• The standard City software contract includes problem response times. For one system, there was no evidence the vendor was following the maintenance plan. For another system reviewed, the contract administrator did not seem to be aware of the contractual requirements despite experiencing significant issues with system functionality.

B. Vendors are not required to provide a transition plan for the City to take over the system. This was particularly problematic for the Solid Waste system which the City has struggled to understand how to administer the system.

Recommendations:

We recommend the Information Technology department require the Project Management Office (or equivalent) to assist the contract administrator with:

A. Monitoring maintenance and support service levels.

B. Requiring vendors to provide a transition plan and execute it.



MANAGEMENT ACTION PLAN

1. Strategy Management

Recommendations:


A. Develop a strategy management template or expand the project charter template to incorporate the five strategy management elements, including:

• current state evaluation, • desired future state, • strategy and action plan.

Also, develop a completed sample that demonstrates the type of information to be provided.

B. Provide strategy management training to information system support staff who serve as project managers and contract administrators for department software projects and the project teams.

C. Review the strategy document and project charter for completeness and adequacy during the software acquisition phase before the solution is selected.

MANAGEMENT RESPONSE: Agree

PROPOSED RESOLUTION:

A. A strategy management template will be developed to include: current state evaluation; known problems and opportunities; desired future state; analysis of alternative; and strategy and action plan. A strategy management document (SMD) will be completed for all RFP or sole source procurements over the city’s $25,000 procurement code threshold.

B. Strategy management training will be developed and offered to Information Technology support staff who serve as project managers and/or contract administrators for a given project or projects.

C. The Project Management Office will review the strategy management document for completeness and adequacy prior to selecting a solution.

RESPONSIBLE PARTY: Project Management Office (PMO)

COMPLETED BY:

The date will be dependent on staff availability, their existing workload and the City’s long term project management strategy. Currently the city’s IT department has one position in its Project Management Office (PMO) and that positon is currently vacant. In addition, prior to the individual retiring, they were working at capacity leading procurement efforts citywide as well as major projects within IT.


2. Benefits Realization Management

Recommendations:


• Update the project charter to clarify that benefits should include measurable improvements tied directly to stakeholders and/or organizational objectives.

• Require contract administrators to track benefits realization throughout the course of the project and report progress to the governance (steering) committee.



The project charter and SMD will be updated to clarify that benefits (measurable improvements) tie to the stakeholders and/or their organizational objectives. In addition, the Project Management Office (PMO) will require contract administrators to track benefits realized throughout the course of the project and report progress to the governance (steering) committee.


COMPLETED BY:


3. Information Technology Governance

Recommendations:


A. Provide guidelines for and oversight of the Acceptance Test Plan development and execution.

B. Define responsibilities of the key roles in project oversight and governance, ensure they are established, and monitor progress, including status updates, deliverables acceptance and evaluation process.



A. Guidelines for an Acceptance Test Plan will be developed and oversight will be provided on the execution of the plan.

B. As part of the governance process key roles and their responsibilities in project oversight and governance will be defined. In addition, the Project Management Office


(PMO) will insure the project manager and/or contract administrator has established these key roles and is monitoring progress which includes status updates, deliverables acceptance and evaluation process.


COMPLETED BY:


4. Project Management

Recommendations:


A. Lead tasks in other IT project management areas, particularly monitoring and controlling processes, currently led by the contract administrator.

B. Assist departments’ IT project management, by providing support:

1. Establish guidelines requiring the use of a project charter at project initiation.

2. Develop a project risk management plan template with clear criteria for risk categorization and escalation and provide risk management training to contract administrators. As well, the PMO should assist in monitoring and responding to project risks.

3. Develop a quality management plan template that demonstrates well-defined quality management processes, roles and responsibilities, and performance measures. Review and approve the quality management plans and provide oversight of the quality of project deliverables.

4. Establish guidelines requiring the creation of a resource plan at project initiation. The plan should include an estimate of internal resource requirements and how the time spent on the project will be mitigated for regular City operations.

5. Establish an organizational change management plan template to be prepared by the project manager or vendor.

MANAGEMENT RESPONSE: Agree, with changes to recommendation


A. The project manager will assist in leading tasks and/or providing oversight related to other IT management areas related to monitoring and control processes currently led by the contract administrator.

B. 1. IT does not have the authority to mandate the use of the project charter. IT also

believes that this concept applies to all projects in the city not just Information


Technology (IT) projects and that should be taken into consideration when addressing this recommendation.

2. A project risk management template will be developed that covers risk categorization and escalation. Training will be provided to the contract administrators regarding the use of the template. The Project Management Office (PMO) will help monitor and respond to project risks as time permits.

3. A quality management plan template will be developed that demonstrates quality management processes, roles and responsibilities, as well as performance measures.

4. Guidelines will be developed that will require the creation of a resource plan at project initiation. The plan will include a high-level estimate of key internal resources required as well as how their existing regular city duties will be accomplished in the absence. Depending on the size, complexity, duration and risk of the project contract labor may need to be hired to create staff availability.

5. An organizational change management plan template will be created. If the vendor does not provide their own change management strategy the template will be used by the project manager, contract administrator and departmental management to create a unique change management plan.

RESPONSIBLE PARTY: Project Management Office

COMPLETED BY:


5. Procurement and Contract Management

Recommendations:

A. We recommend that the Purchasing department retain software RFP evaluation materials are maintained in accordance with public records law and record retention guidelines.

We recommend that the Information Technology department require the Project Management Office (or equivalent) to:

B. Work with the Purchasing department to incorporate performance standards into the RFP template.

C. Work with the Purchasing department to establish standards for and review software market research prior to approving sole-source purchases.

D. Establish SaaS criteria and standards, and require the SaaS questionnaire to be included in the RFP evaluation process of each vendor’s proposal.


E. Facilitate contract administration and improve oversight by assisting the contract administrator to:

1. Maintain a contract file that includes all contract-related documentation, including the approved deliverables.

2. Require the vendor to produce deliverable evaluation documents including agreed-upon acceptance criteria, prior to acceptance.

3. Hold an internal kick-off meeting after the vendor contract is signed, including the contract administrator and involved staff from the IT PMO, Purchasing, Legal and Risk Management.

MANAGEMENT RESPONSE (5A): Agree


Previously the evaluation materials in the Purchasing files were considered copies that were supplied by the Contract Administrator. Therefore they were not retained as records to our retention schedules.

Going forward the CA’s will be told to provide all originals to Purchasing, and they will then be kept in the Purchasing files as official documents per the retention schedules.

Purchasing will include such direction in a notice to Contract Administrators, update the evaluation instructions and update the CAA training covering the same issue.

RESPONSIBLE PARTY: Purchasing Director

COMPLETED BY: 12/31/2017

MANAGEMENT RESPONSE (5B-E): Agree, with changes to recommendation


B. IT will work with the Purchasing department to incorporate performance standards into the RFP template.

C. IT will work with the Purchasing department to recommend processes for, and review of, software market research prior to approving sole-source purchases. Because Purchasing is responsible for the management of all sole source purchases throughout the city, including non-IT purchases, it will ultimately be their responsibility to approve, adopt and enforce the recommendations.

D. The existing SaaS questionnaire will be reviewed and updated as needed. Once complete, the questionnaire will be a requirement of each vendor’s response and will be evaluated as part of the RFP process.

E. Contract administration and oversight:

1. While Information Technology can assist in the process, management believes that the maintenance of a contract file applies to all city contracts and should be covered as part of the city’s larger contract administrator training conducted by Purchasing.


2. Vendors will be required to produce agreed-upon acceptance criteria and the criteria will be confirmed complete by the contract administrator and verified by the PMO prior to signing off on acceptance.

3. After the vendor contract is signed, an internal kick-off meeting will be held with the contract administrator and staff from the IT PMO. Given that Purchasing, Legal and Risk Management roles pertain to the contract itself, and the contract is already signed, IT will invite them to the meeting as optional and if they have comments, questions or feedback they can attend.

RESPONSIBLE PARTY: Project Management Office and contract administrator

COMPLETED BY:


6. Solution Delivery

Recommendations:


A. During the proposal phase of the project, verify the vendor has a suitable SDLC.

B. Provide oversight for product testing and approve phase acceptance based upon a contractual checklist.



A. As part of the RFP process, the city will ask for the vendors to describe their SDLC process. From there, the Project Management Office (PMO), in conjunction with the RFP evaluation team, will evaluate their response to ensure it is suitable.

B. The Project Management Office will confirm with the contract administrator that product testing has been successfully completed and that the approval phase of acceptance is properly documented based upon a contractual checklist.


COMPLETED BY:



7. Maintenance and Operations

Recommendations:

We recommend the Information Technology department require the Project Management Office (or equivalent) to assist the contract administrator with:

A. Monitoring maintenance and support service levels.

B. Requiring vendors to provide a transition plan and execute it.



A. Upon project closure, the Project Management Office will work with the contract administrator to ensure they understand their role in monitoring maintenance and support service levels.

B. Prior to the projects closure, the Project Management Office will work with the contract administrator and vendor to ensure they have a transition plan and that they execute it.


COMPLETED BY:


City Auditor’s Office 7447 E. Indian School Rd., Suite 205 Scottsdale, Arizona 85251 OFFICE (480) 312-7756 INTEGRITY LINE (480) 312-8348 www.ScottsdaleAZ.gov/auditor

The City Auditor’s Office conducts audits to promote operational efficiency, effectiveness, accountability, and integrity.

Audit Committee Vice Mayor Suzanne Klapp, Chair Councilmember Virginia Korte Councilwoman Kathy Littlefield City Auditor’s Office Kyla Anderson, Senior Auditor Lai Cluff, Senior Auditor Cathleen Davis, Senior Auditor Brad Hubert, Internal Auditor Dan Spencer, Senior Auditor Sharron Walker, City Auditor

Documents

CITY AUDITOR'S OFFICE Software Acquisition and …...This audit was included on the City Council-approved fiscal year (FY) 2016/17 Audit Plan as a contracted information technology