Performance Measures Indicating IT Project Effectiveness and Investment Success / 1 2010 Performance Measures Indicating IT Project Effectiveness and Investment

Performance Measures Indicating IT Project Effectiveness and Investment Success / 12010

Performance Measures Indicating IT Project Effectiveness and Investment Success

INTOSAI Working Group on IT AuditThe 6th Performance Auditing Seminar

Beijing, China

April 2010

Prepared By

State Audit Bureau of Kuwait Information Technology Audit

Technical Support Department


Benefits of Performance Measurement

1. Benchmark the contribution and alignment of an IT investment with strategies and business goals.

2. Provides an accurate and a quantifiable assessment of improving and degrading measured areas.

3. Assist in assigning focus to new areas of improvement to help in directing funds in the appropriate direction and magnitude.


IT Investment Measurement Approach

• The term “IT investment” is broad. It can be a product, service or program.

• Areas of measurement can include financial management, IT service/infrastructure availability, quality, delivery time, cost, customer satisfaction, resource utilization…etc


IT Investment Measurement Approach

• Having a set of measurements to cover multiple IT areas helps in establishing a metric.

• Gradually building up with the investments goals in mind will yield more specific measurements.

• Linking of measurements to Organizational and IT goals gives a strong insight on areas with potential development.


Measurements

• Focus based: Financial, Customer, Internal, Learning and Growth.

• Measurement unit: Percentage, number, Ratio …etc

• Measurement direction and Target


IT Investment Case

• Initial audit conducted revealed findings that were reported.

• Measurements can be applied to in order to track performance areas of the findings.


IT System Description

1. Our case required the audit of a Rationing system developed by the IT department of a Government agency to automate and optimize the delivery of essential subsidized commodities to eligible beneficiaries. The Ration Department is in charge of setting the rationing regulations and laws by Setting eligibility criteria and Determining quotas and prices.

2. The IT department of the ministry is in charge of the Rationing System to:

• Issue, renew, amend the ration card data for beneficiaries.

• Distribute the goods to consumers using the system in branches of Co-operative wholesale societies.


Expected System Benefits

As reported by the Ration Department, the system is expected to deliver the following benefits:

1.Provide the information on beneficiaries, ration cards and transactional data in an electronic format for easier accessibility and accuracy.

2.Non-issuance of duplicate ration cards per beneficiary.

3.Avoid the current manual issuance and indexing of ration cards.


Expected System Benefits

4. Link the branches to the main system at the ministry to automate change.

5. Issue/print the new ration cards.

6. Better management of inventory and governing of transactions.

7. Provide accurate statistical reports in a timely fashion with ease.


Audit Goal

The goal of the audit is to:

1. Verify the efficiency and effectiveness of the system of issuing ration cards and the system of distribution/sales.

2. Verification of the systems availability.

3. Alignment to benefits and requirements.

4. Measure the security and safety of the systems.


Initial Observation

There was an absence of organizational and IT goals and objectives.

Impact: Limits the possibility of relating the outcome of performance measures to the Organizational and IT goals as IT investment processes, resources and activities cannot be aligned/linked to a higher level business objectives. Thus, the measures will be focused on IT processes, resources and activities surrounding the Rationing system.


About the Findings

Next, we will discuss some of the findings of our audit. Since there were no pre-established measures, these audit findings will be used to suggest measures to the concerned agency.


Audit Findings Where Performance Measures are Applicable

Finding (1): Absence of documented and approved procedures and processes for project management.

Potential performance measures:

1. Percent of meeting the predefined business benefit

2. Percent of budget deviation value compared to the assigned budget

3. Percent of project progress (on time and on budget)



4. Percent of following project management standards and practices



Finding (2): Absence of standardized procedures and processes for IT resource procurement.


1. Amount of reduction of the purchasing cost

2. Percent of key stakeholders satisfied with suppliers

3. Percent of initial requirements addressed by the selected solution

4. Percent of compliance with standing procurement policies and procedures



5. Percent of variance amongst budget, forecast and actual cost

6. Percent of overall cost that is allocated according to the agreed-upon cost models

7. Percent of disputed costs by business



Finding (3): Absence of technological direction.


1. Number and type of deviations from technology infrastructure plan

2. Percent of compliance with defined IT architecture and technology standards

3. Number of emergency changes related to the infrastructure components



Finding (4): Absence of IT investment feasibility study.


1. Percent of stated benefits that were not achieved due to incorrect feasibility assumptions

2. Percent of stakeholders satisfied with the accuracy of the feasibility study



Finding (5): Absence of disaster recovery and business continuity plan and incident logs.


1. Percentage of downtime due to unplanned outages

2. Percent of availability SLAs met

3. Frequency of service interruption of critical functions



Finding (6): Operation and user manuals are not updated.


1. Percent of business owners satisfied with application training and support materials

2. Number of incidents caused by deficient user and operational documentation and training

3. Satisfaction scores for training and documentation related to user and operational procedures



Finding (7):

1) The Ration card system does not restrict the issuance of cards to ineligible individuals.

2) The Ration card system does not implement any restrictions regarding the creation of non-existing beneficiaries.

3) The Ration card system allows the creation of beneficiaries with out names or contact information.




1. Percentage of ineligible beneficiaries

2. Percentage non-existing beneficiaries

3. Percentage of beneficiaries with missing or incomplete information

4. Percent of data integrity

5. Frequency of internal control incidents

6. Cost of non-compliance, including settlements and fines



Finding (8):

1) The Ration distribution/sales system transactional database contains sales of commodities that are unidentifiable on the system.

2) The Ration distribution/sales system transactional database contains sales with miscalculated commodity prices.




1. Percentage of transactions with unidentifiable commodity.

2. Percentage of transactions with miscalculations.

3. Percent of data integrity.



Finding (9): The distribution centers run a client software that saves the transactions on an unprotected local database that is reconciled with the main collective database of transactions at the end of the day due to design limitations. Some databases were reported to be found emptied at the time of reconciliation which suggests also the possibility of any other database being tampered with.




1. Number of security incidents with business impact

2. Number and type of suspected and actual access violations

3. Number of occurrences of an inability to recover data critical to business process

4. Revenue lost due to loss of data


Summary

• The audits findings we communicated to the agency in order to:

1. Resolves issues permanently where possible.

2. Apply appropriate controls and adopt suggested measures to follow up with actual performance measurement.

The audit concluded for the Ration system provided the agency with the means to establish areas of improvement, measurement baselines and targets to achieve.


Discussion

Thank you…

Documents

Performance Measures Indicating IT Project Effectiveness and Investment Success / 1 2010 Performance Measures Indicating IT Project Effectiveness and Investment