Towards a Systematic Approach Towards a Systematic Approach

to Improving Quality in the to Improving Quality in the

Investment CentreInvestment Centre

Learning from the Focus Group, the Partner Survey and the Pilot Experience

December 2010

OutlineOutline

� Learning from feedback:◦ ID-2009◦ Quality Focus Group◦ Partner Survey 2010

� Learning from Q Pilots� Proposed approach to Q management◦ task level◦ service level◦ division level

� Next Steps: implementing a quality system in TCI

LEARNING FROM LEARNING FROM FEEDBACKFEEDBACK

Feedback from ID Feedback from ID -- 20092009

� Quality review process perceived to be significantly reduced. Partly explained by the changing products and services provided by TCI, and the downsizing Human Resources in TCI services and TCID

� Need to revitalize and strengthen Quality of work

Feedback from Focus GroupFeedback from Focus Group1. Business Development is key aspect to QE/QA approach;

2. Focus on quality upstream (at-entry, ToR) minimize drudgery

3. Need for independent quality system if FAO to be a honest broker

4. Avoid unnecessary checkbox approach, to avoid delays in delivering

5. Q process not only paper system; involves personal interactions and behaviours

6. Feedback is important and has been missing from within TCI and from IFIs and countries

7. Link to IFI quality processes. Don’t duplicate.

8. TCI to invest in quality support as more substantive work coming

9. PEMS as part of Q process

10. Q system to take into account time pressures

11. Q review work needs to be billable

12. Q Process to provide for learning

13. Q to apply to all partners including other FAO units

14. GS Staff’s role in Q Process

15. Leaders for ORs have a review responsibility

Feedback from Partner SurveyFeedback from Partner Survey

◦ Overall TCI doing good professional job ◦ TCI in demand, brings value-added◦ But they use trusted individuals who know WB/IFAD

system, nervous of using others◦ Demand growing but ad-hoc-ism too◦ Partner’s own Q systems evolving◦ Highly satisfied with technical expertise, much less

with linkages/facilitation and innovation◦ Issues with capacity development◦ Behavioural issues: need to listen more◦ See TCI more as a technical service provider, rarely

as a partner that shares resources, risks and... results

LEARNING FROM THE LEARNING FROM THE PILOTSPILOTS

Q Process: DefinitionsQ Process: Definitions

� Quality Enhancement (QE): a system to support and improve the quality of a product before approval. QE involves key steps (screening, peer reviews, partner feedback, etc.) to check against the TOR that objectives are met along good practice and sound technical standards. Reviews must be constructive and offer improvements where needed. Mentoring of junior staff by senior staff can be included.

� Quality Assurance (QA): a system to prevent poor quality products being delivered by TCI and its partners. QA would ensure that a sub-standard product is detected and not released. The QA function acts independently from the QE function, and provides a final control that products meet minimum standards.

Piloting Q at Task LevelPiloting Q at Task Level

1. Screening based on ToR2. Planning3. Enhancement:◦ Review of draft reports (optional)◦ Mentoring (throughout the assignment)

(optional)

4. Self Assessment (optional)5. Quality Assurance◦ Partner feedback ◦ Independent Quality Assurance Review

(optional)

Criteria for Initial ScreeningCriteria Measure Scoring

1. Investment Value Value of underlying investment (total value including financier and government contributions)

o>$50m=2

o$10-50m=1

o<$10m=0

2. Type of Agency Partnering with TCIAgency is New, or Task Manager is either inexperienced or 'risky'

oNew Partner or Manager=2

oNew or Difficult Manager=1

oNeither =0

3. Level of TCI influence in the investment

Potential for TCI to have influence on the quality of the investment (e.g. involved from upstream through to approval in case of design, or leading supervision or evaluations)

oHigh influence / leverage =2

oModerate Influence =1

oLimited =0

4. Level of innovation, learning TCI can expect to deliver a high level of innovation or provide best practice for the country through its involvement

oHigh = 2

oMedium = 1

oLow=0

5. Complexity ratingNo. of partners (donors, agencies, govt. depts) involved, geographical or sector wide coverage

oVery Complex =2

oModerately Complex =1

oLow complexity=0

6. New country / type of work

New assignment started (rather than continuation of previous), or a new country (for the TCI staff involved)

oNew task & new country= 2

oNew country=1

oContinuation =0

7. Quantity of Staff

Total number of staff plus consultant weeks needed for whole project (combining missions where appropriate)

oOver 10 weeks =2

o5-10 weeks =1

oless than 5 weeks= 0

8. Experience and type of staff The degree of experience and skill available for the assignment

oLow level of skill & experience =2

oMedium level = 1

oHigh experience = 0

9. Level of local co-preparationHigh reliance on local personnel to complete the assignment

oHigh reliance =2

oSome reliance =1

oLittle or no reliance =0

10. Strength of external quality control Limited reliance on external quality processes

oNo or limited external Q=2

oSome external Q = 1

oMainly reliant on external Q =0

Quality Process DiagramQuality Process Diagram

Q ScreeningQ Screening

Quality Pilots ProgressQuality Pilots ProgressPilot Exercise for QE/QA Instruments/Steps : September - December 2010

Pilot Assignment Lead TCI Staff Internal External

Zambia COSOP F.Ohler 11 � x x x x x �

Turkmenistan E. Hidier

W.Africa Study F. Hollinger 13 x x x x x x x x x

Design Jamaica I.Christensen 13 � x x x x � � x x

Bangladesh B. Veillerette 17 � x x x x x x

Gambia P.Pozarny 11 � � x x � �

C.Asia TCP A.Gulliver 10 not yet approved x x x

Cambodia CBAPP N.Santos 9 � x x x

Mozambique IrrigationJ.Risopoulos 7 � x x x x x

Evaluation Peru INCAGROII ICR K.Urban 6 dropped

Self-

Assess

ment

Client

Feedback

Study (policy, sector

work)

QA StepsQE Steps by TCI

Activity / Product

Process entry

Screening

Score

Open a Q

Note (a)

ToR

Review

Pre-

mission

briefing

Mentoring

Plan

QA

Review &

Rating (c)

QA

Decisio

n (d)

Post

Mission De-

Brief

Peer Reviewing

Progress to dateProgress to date

� Six pilots on track: performed Q screening, selection, planning Q steps,

� Gambia and Jamaica completed, and Zambia, West Africa Study, Bangladesh and Cambodia making progress

� Three are slow or suspended: Peru is dropped (to be substituted), Turkmenistan (?), Central Asia TCP not approved yet

� Screening scores range from 7 – 17 out of 20� Guidance manual in process: Formats prepared

for Screening, TOR, Self Assessment, Mentoring� Client feedback – mainly through online survey� Mentoring – not tested

Lessons LearnedLessons Learned1. Q Screening is easy to apply to TCI’s main work

2. Q screening may need to be adapted for upstream work “bodyshopping”

3. Self-Assessment produced useful feedback and learning

4. Scoring criteria were refined and a criterion on the strength of partners’quality system added

5. Useful for planning reviews, but issues with mobilizing reviewers and providing them incentives (e.g. fee, billable time)

6. Need to provide TCI feedback to Partner (especially if the Task Leadership lies with Partner)

7. IT issues: need to link to MIS to reflect Q in management processes

8. Q Management at Service and Division levels:

◦ Service Chiefs have to be involved in Q Management

◦ QA, not tested yet

◦ Role of TCID to be further defined

◦ Use of Q process to foster knowledge sharing and learning

9. Q Process still to be anchored to TCI Strategic Framework!

PROPOSED APPROACH PROPOSED APPROACH TO QUALITY TO QUALITY MANAGEMENTMANAGEMENT

Setting Clear TargetsSetting Clear Targets

� % budget allocated to quality control

� % assignments rated as satisfactory by clients / partners

� % investments approved on time

� % investments rated as Satisfactory at Completion

� % products rated as best practice

� The indicators should be included in TCI Strategic Objectives and Results Framework as a way of measuring “quality-assured investments (LO3)”

PrinciplesPrinciples

� Products delivered should meet minimum standards

� All assignments should be screened (upstream) for relevance and risks

� All assignment should receive feedback from Partner

� Most critical assignments should received the highest QE/QA efforts

� Q Process should lead to learning� Q managed along subsidiarity principle:

task, service and division levels

Managing Q at Task LevelManaging Q at Task Level

1. Screening based on ToR2. Planning3. Enhancement:◦ Review of draft reports (optional)◦ Mentoring (throughout the assignment)

(optional)

4. Self Assessment (optional)5. Providing Feedback to Partner6. Quality Assurance◦ Partner feedback ◦ Independent Quality Assurance Review

(optional)

Managing Q at Service LevelManaging Q at Service Level

� Focal Point (FP) for Q management

� Q Meeting every two weeks with SC, FP, TLs and TCID: screening, monitoring and learning

� Budget resources

� Support mentoring

� Foster knowledge sharing and learning: capturing lessons and feeding them back

Managing Q at Division LevelManaging Q at Division Level

� Developing common Q standards� Q Process to be integrated with the R-B

Management, & planning and budgeting systems

� Oversight and guidance� Managing QA (independent function)� Establishing more effective IT platform� Managing pool of “certified” reviewers� Fostering learning and linking up with the

thematic networks and country focus groups (to come)

Who does what? Who does what?

� Task Leader - Planning and managing the quality process at entry and during the task, Q Note (including choice of reviewers)

� Service Chief, assisted by Q Focal Point, endorsing following: TOR, screening score, assignment report (ASR), self-assessment, Feedback to Partner, deciding on product release to Partners or to TCID for QA, learning / linking to country focus group

� TCI MIS: records for each mission and project code: link to TOR, screening score, link to the Q Note, Self-Assessment

� TCID: QA Independent Review, decision to release high-risk outputs, monitoring quality process, providing support (pool of reviewers, IT), learning across services and linkages to thematic networks

NEXT STEPS: NEXT STEPS: IMPLEMENTING A IMPLEMENTING A QUALITY SYSTEM IN TCIQUALITY SYSTEM IN TCI

Key stepsKey steps

� Agreeing on an implementation plan at management level

� Set up a Quality Team

� Dialogue with partners

� Seeking feedback from GS and Professional TCI staff

� Complete testing (body-shopping, upstream work)

� Refining Q process: guidance, scoring at exit(?)

� IT Platform for better management and for knowledge sharing

� Link with the larger TC and FAO reform

� Revisiting TCI business model (e.g. business lines, pricing)

Key AssumptionsKey Assumptions

� Management should take the lead

� Focusing on strategic aims and results, prioritizing areas of work, reducing fragmentation

� Allocating human resources accordingly and providing incentives thru e.g. PEMS

� Promoting learning

� Encouraging behavioral change (e.g. genuine exchange of views, providing and seeking feedback)