Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
WHO Reform Stage 2 Evaluation Final report Appendices
18 November 2013
WHO Reform Stage 2 Evaluation - Final report Appendices
PwC 2
Table of content
Appendix A- Inception Report 3
Appendix B- Addendum to Inception Report 40
Appendix C- Evaluation questions 61
Appendix D- List of interviews 67
Appendix E- List of documentation 71
Appendix F- Analysis of staff survey 74
WHO Reform Stage 2 Evaluation - Final report Appendices
PwC 3
Appendix A- Inception Report
With offices in Aarau, Basel, Berne, Chur, Geneva, Lausanne, Lugano, Lucerne, Neuchâtel, Sitten, St. Gallen, Thun, Winterthur, Zug and Zurich, PricewaterhouseCoopers SA is a provider of auditing services and tax, legal and business consultancy services. PricewaterhouseCoopers AG is a member of a global network of companies that are legally independent of one another; the network is represented in some 150 countries throughout the world.
PricewaterhouseCoopers SA Avenue Giuseppe-Motta 50 P.O Box 2895 1211 Geneva 2 Phone +41 58 792 91 00 Fax +41 58 792 91 10 www.pwc.ch
Office 7121 Bid Ref: IOS Stage two Attn: Mr David Webb, Director, Office of Internal Oversight Services World Health Organization 20, Avenue Appia CH-1211 Geneva 27 Switzerland Geneva, 8 July 2013 Dear Sir / Madam, Thank you for giving PricewaterhouseCoopers (“PwC”) the opportunity to be selected as the providers for the Stage 2 Independent Evaluation (“the evaluation”) of the World Health Organization (“WHO”). We understand the strategic importance of this evaluation and look forward to working closely together to make it a successful assignment. This document sets out our inception report which has been prepared in accordance with the Letter of Agreement for the Independent evaluation of the WHO reform: Stage 2. The purpose of the Inception report is to present our detailed approach to conducting the Stage 2 Evaluation of the WHO reform and presents our project management plan including timeline and milestones of the project. We remain at your disposal for any further information or clarification on our inception report and look forward to hearing from you. Yours faithfully, PricewaterhouseCoopers SA
Gill Sivyer Antoine Berthaut
Engagement Partner - Geneva Engagement Director–Geneva
WHO Reform Stage 2 Evaluation - Final report Appendices
PwC 5
1. Understanding of the project scope
1.1. Understanding the WHO reform
The proposed evaluation is part of a far reaching reform agenda initiated by the WHO in 2010. The aim of the
reform is to improve the overall performance and accountability system of the organisation to better address
the changing public health needs of the population going forward into the 21st century.
One of the major reasons for kick starting the process of reform has been the continued unpredictability of
funding and the difficulty WHO has had to secure financing for its priority activities and programmes.
Additionally, and as highlighted by the report called ‘Review of management, administration and
decentralisation in the WHO of the Joint Inspection Unit’ (January 2013), WHO has been slow to reform in the
areas of internal and external governance, organisational effectiveness and transparency.
Some of the main challenges identified were around the:
Need for better internal governance and alignment between global and regional bodies
Difficulty of allocating resources across various layers of governance structures
Lack of predictability of funding and associated challenges with priority setting
Weak resource mobilisation capacity at all levels of the organisation
Ballooning of administrative and management costs
Rise of other global health actors and the role of the WHO in a changing environment
The High-level implementation plan of the WHO reform, referred to as Implementation plan in this report,
was developed through an extensive consultation process initiated in 2011 with the discussion of the future
financing of the WHO that has resulted in the definition of four reform areas (programmatic, governance,
managerial and change management) and associated outcomes and related outputs. Whilst the reform areas
and outcomes have been validated by the Secretariat and Member States and agreed upon as the set direction
for the reform, the Implementation plan is a flexible document that continuously aims to improve the quality of
its outputs and responsibility framework.
1.2. WHO reform evaluation process
As part of the above, the Executive Board at its special session in November 2011 requested a two stage
independent evaluation of the WHO reform.
The Stage 1 evaluation of the reform was conducted by WHO’s external auditors, the Comptroller and Auditor General of India, between February and March 2011, with the aim to validate the completeness, comprehensiveness and adequacy of WHO’s reform proposals centred around governance, management and programmatic reform. The recommendations elaborated by the Stage 1 evaluation have included:
Strengthening linkages between governing bodies at the regional and HQ level for greater organisational coherence;
Strengthening the accountability systems throughout all three levels of the organisation (i.e. HQ, regional and country level) which could be supported by a more robust results-based and performance management system;
Focusing on change management through the development of a communication strategy on reform that clearly identifies the champions of the reform at WHO;
WHO Reform Stage 2 Evaluation - Final report Appendices
PwC 6
Implementing a sound monitoring and evaluation framework that accompanies the reform and is supported by clearly defined outcomes and outputs indicators.
The modalities for Stage 2 of the evaluation were endorsed by the Executive Board at its 132nd session in January 2013 to assess the readiness of the WHO to fully implement the reform agenda and to identify the necessary conditions for success.
1.3. Objectives of the Stage 2 Evaluation
This independent evaluation has at its core the objective of supporting the reform process and assisting the WHO in identifying the mechanisms and tools required to unlock some of the areas of the reform which have been lagging behind. Member States have demonstrated increasing confidence in the evaluation process accompanying the reform and have several expectations in relation to seeing the reform hit the ground. The primary audience for the evaluation will be the Executive Board, represented by the Evaluation Management Group. This evaluation represents an opportunity to validate current reporting to Member States and to foster trust and confidence in the reform process. The secondary audience will be Secretariat management and staff involved in Reform activities. The evaluation will favour exchange of knowledge between the Independent Evaluation Team and the Secretariat and will contribute to paving the way for efficient, effective and sustainable change.
As several reform initiatives defined in the Implementation plan are underway and at different stages of implementation, this independent evaluation will equally cover all aspects of the reform while bringing more depth into areas which the Independent Evaluation Team will have identified has being critical to the success of the reform program and the WHO as a whole.
Also as the Reform is still in its early day, the evaluation will not seek to assess the impact or outcome of the Reform.
The evaluation will follow UNEG norms and standards for evaluations, as well as ethical guidelines.
1.3.1. Assess WHO reform status
The Terms of References for the evaluation, as well as discussions the Independent Evaluation Team has had with members of the WHO Office of Internal Oversight Services (IOS), the Reform Team responsible for overseeing and reporting on the execution of the reform agenda, and a limited number of WHO Member States, indicate a preference for a recommendation-oriented evaluation that addresses the following objectives:
Assess the status of actions taken on the Stage 1 recommendations: In this context the evaluation will in the first place assess whether the recommendations mentioned in Section 1.2 above made in the context of the Stage 1 Evaluation have been addressed and are integrated as part of the Implementation plan.
Assess progress made to date on the WHO reform implementation: The evaluation will assess the status of the reform agenda which will involve reviewing and confirming progress made to date on outcomes, outputs and deliverables defined in the Implementation plan.
Assess the completeness of the WHO reform Implementation plan: Stage 1 of the evaluation validated whether the overall objectives of the reform agenda were in line with the challenges that the WHO needed to address. In contrast, Stage 2 focuses on validating whether the implementation strategy supporting these initiatives addresses all required elements needed to effect change and achieve the intended results. A specific focus relates to assessing whether the theory of change or result-chain of the reform is robust and adequately translated into reform operational plans. This comes down to validating the completeness of the Implementation plan.
WHO Reform Stage 2 Evaluation - Final report Appendices
PwC 7
1.3.2. Assess the modalities of implementing the reform proposal and the sufficiency of the change management strategy
The second objective of the evaluation relates to validating whether the enabling conditions for success of
the reform are met in order for the reform in general and each initiative in particular to succeed. This includes:
Prioritisation of various components of the reform proposal;
Identification of change agents;
Capability of accountability structures to support the reform process;
Resource requirements for the reform proposal;
Timelines defined for the implementation of the reform proposal;
Performance indicators defined to measure movement towards reform process;
Strategy to deal with hindrances, enablers and dependencies;
Changes in internal procedures and structures to implement the reform process.
WHO Reform Stage 2 Evaluation - Final report Appendices
PwC 8
Structure (incl. internal governance and accountability)
Strategy
Process (incl. transparency)
People (incl. role)
Technology (incl. Instruments)
Programme Management
Change Management (incl.barriers to change )
Assess & Strategize
Rolling-out new ways of working
Constructing, testing and validating detailed changes
Designing blueprints and detailed execution plans
Building the case for reform, assessing options and validating reform strategies
Operating the reformed organization and ensuring continuous improvement
Ch
an
ge
E
na
ble
rs
De
liv
er
ing
Ch
an
ge
Design Construct Implement Operate & Review
Outcome
Imp
ac
t
Relevance
I A O I A O I A O I A O
4
3
2
1 Sustainability
Efficiency/Effectiveness
I A OInput Activity Output
2. Proposed evaluation approach
2.1. Conceptual framework
Transformational change occurs when there is a fundamental shift in strategy resulting in significant changes in the way that an organisation operates, is structured or sets out to satisfy its stakeholders. Transformational change is deep and pervasive, affecting the interplay of strategy, process, people, technology and structure often requiring changes in culture (mind-set and organisational behaviour). This is the essence of WHO’s reform.
Change is often both misunderstood and also underestimated by organisational leaders because capturing its benefits requires insight into how this change impacts almost every other part of the enterprise, from strategy and structure to people, process and technology. At its heart, change is about people. It is about values and culture, the “soft stuff” that many leaders find so chronically hard to address.
The conceptual framework used for the evaluation builds on best practices for transformation programmes. It provides a logical and systematic framework to cover the full scope of the practical challenges in shifting an organisation away from addressing incremental change to implementing lasting transformation.
The framework takes into account the following analytical elements:
1. UNEG Norms and Standards for Evaluation in the UN System
2. Dimensions of a transformation programme
3. Stages of execution of a transformation programme and result-chain or theory of change to drive impact
4. Enablers for a transformation programme
It is summarised in the picture below:
Figure 1: Conceptual framework
WHO Reform Stage 2 Evaluation - Final report Appendices
PwC 9
2.1.1. UNEG Norms and Standards for Evaluation in the UN System
The evaluation incorporates the UNEG Norms and Standards for Evaluation in the UN System with a particular focus on the criteria of efficiency, effectiveness and sustainability. Effectiveness will be measured against the qualitative benchmark of whether the WHO is undertaking the right steps towards meeting its reform objectives and if the activities laid out in the Implementation plan adequately address the goals of the reform. The core question here is: How well is the WHO doing in reaching its objectives? When assessing the efficiency of WHO’s reform process, we will assess whether the inputs placed towards the activities detailed in the Implementation plan include the right resources, offer the best value for money and are executed according to plan. The core question here is: Is the WHO doing this in the right way in the set timeframe? Finally, the sustainability of the WHO reform program will be addressed by assessing if it is designed to bring about deep institutional and cultural changes at the WHO. To ensure sustainability, all elements of the change vision need to be present, with strong interaction of those responsible for change leadership and management. The core question here is: Is the solution being implemented clear, supported by appropriate tools, reinforcers and ownership that by those who operate the solution in order to bring about institutional and culture change at the WHO?
Each evaluation questions will clearly identify to which of the above mentioned criteria they relate to.
2.1.2. Dimensions of a transformation programme
Transformation encompasses five dimensions which are considered essential for delivering change within an organisation. These dimensions provide a structure for systematically analysing an organisation’s internal working model, structure, processes and culture.
A strategy that provides the strategic direction and vision for the organisation. The criteria to assess this stage mostly relate to relevance and effectiveness.
A clear organisational structure that defines reporting lines between governing bodies and the various levels of the organisation which will directly affect internal governance and accountability systems. The criterion to evaluate this stage is that of effectiveness.
A set of processes, tools and guidelines which are readily available and incorporated into the day to day work of WHO staff for greater transparency. The criterion to assess this stage is that of effectiveness.
A pool of qualified, adequately incentivised human resources with clearly defined roles and terms of references, supporting by an enabling culture. The criterion to assess this stage is that of effectiveness.
Technology, systems and data which support the direction of change. The criterion to assess this stage is that of effectiveness.
The resulting questions for the evaluation are as follows:
Domain Evaluation questions Criteria
Overall Have all five dimensions of transformation been considered in the reform implementation strategy?
Relevance
Effectiveness
Strategy Has the overall reform plan been translated into coherent sub-strategies for each of the 12+1 initiatives under review?
Is the result-chain robust?
Relevance
Effectiveness
Structure Does the reform adequately address factors relating to internal governance, reporting lines between governing bodies, accountabilities, organisational structure, geographical footprint and relationships with external stakeholders?
Effectiveness
WHO Reform Stage 2 Evaluation - Final report Appendices
PwC 10
Does each reform initiative take the required enabling structure into account?
How is reform cascaded down into the various levels of the organisation (HQ, Region and country)?
Processes Are processes consistent and standard across the organisation, at the HQ, regional and country level?
Are strategies and policies supported by adequate processes and procedures? Have these been documented? Are these fit-for-purpose?
Is the required data available to support the new processes? How is data quality ensured (availability, accuracy, completeness, timeliness and confidentiality)?
Have internal controls and standardisation of processes and data been considered?
Are tools and guidelines readily available and incorporated into the day to day work of WHO staff?
Effectiveness
Efficiency
People Are governance mechanisms and accountability documented and clear?
Do leaders own change efforts, or are they delegated to others?
Are the skill-sets of staff adequate to meet the objective of each reform initiative?
Has a performance management system and incentives been considered to support the targeted changes?
Are the culture and individual behaviours aligned with and supportive of the reform strategy?
Effectiveness
Technology Are IT enablers such as data management and IT infrastructure been considered to support the direction of change?
How does technology enable the reform and its initiatives?
Does the IT strategy appear to be aligned with the reform Implementation plan? Is technology appropriately funded in the context of the reform?
Effectiveness
The strategy dimension shapes the other four, and is often the starting point for a dialogue that results in
identifying improvement opportunities that are connected across the five dimensions. While each reform
initiative will be analysed through the lens of these five dimensions the relevance of the dimensions per reform
initiative will be considered throughout the evaluation.
2.1.3. Stages of execution of a transformation programme and Result-chain to drive impact
In order for change to be affected, a structured approach to change delivery is required. Successful
transformations are typically conducted through five following stages. These include:
Assess and Strategize - used to understand the strategy and business needs that drive the change initiative and envision the future state and expected benefits at a high level; set-up the required programme management arrangements; develop the change vision and define the best-fit change approach and integrated change strategy to support the change initiative. The criteria to assess this stage mostly relate to relevance and effectiveness.
Design - used to design the changes to the organisation's operating model with a focus on optimising the delivery of benefits. The criterion to assess this stage mostly relates to effectiveness.
Construct - used to build and test the changes to the organisation's operating model to ensure they deliver the expected benefits. The criteria to assess this stage mostly relates to effectiveness and efficiency.
Implement - used to successfully implement the operating model changes and plan for subsequent benefits delivery. The criteria to assess this stage mostly relates to efficiency and sustainability.
WHO Reform Stage 2 Evaluation - Final report Appendices
PwC 11
Operate and Review - used to hand over the operating model changes to business as usual, realise the
expected benefits and set the foundation for continuous improvement, assess the success of change, finalise
the sustainability plan and complete knowledge transfer and hand over. The criteria to assess this stage are
mostly that of sustainability and impact.
The resulting questions for the evaluation are as follows:
Domain Evaluation questions Criteria
Strategize and Assess
Was the current state of affairs analysed and the baseline created? Is there a clear definition of the problem being addressed?
Has there been a process to evaluate and prioritise areas for improvement? How was the prioritisation conducted? How were the criteria for prioritisation defined?
Has a vision/strategy/concept note been developed to address this area of challenge? How has it been endorsed by the organisation?
Relevance
Effectiveness
Design Have all requirements been collected to define what needs to be changed to deliver the outcome and the expected impact?
Have blueprints been developed?
Has a roadmap been produced, defining broad objectives, timeline, responsibilities, KPIs, budget and reporting structure?
What has been the process for approving this roadmap?
Effectiveness
Construct How is the design/general concept translated into details? Is it comprehensive?
How have all relevant stakeholders been engaged in the process?
How are the new ways of operating tested prior to roll-out?
Has a detailed Implementation plan been developed prior to roll out?
Has a specific body/team been set-up to manage and oversee the implementation?
Effectiveness
Efficiency
Implement Which of the activities part of the Implementation plan have started?
Has sufficient training been provided to impacted stakeholders?
How are corrective actions, where required, determined, discussed and agreed?
How is the transition into Business as Usual planned?
Efficiency
Sustainability
Operate and review
How is the knowledge transferred and how is the hand-over performed from reform to business as usual?
Has ownership of the new way of working been transferred?
Has a post-implementation review been completed?
Sustainability
Impact
The completion of each of these stages is a milestone usually materialised by a sign-off on specific deliverables
and outputs, these outputs becoming the input into the activities taking place in the next stage. In order for the
outcome of a specific change initiative to be achieved, a sequence of input/activity/outputs relating to each
stage of the transformation is required.
Also best practice transformation seeks to have a phased delivery of benefits (outcomes), with a focus on
securing early success to build trust and engagement. This means that the component initiatives of a change
programme will move through the five stages at different pace. A challenge then becomes ensuring
dependencies between initiatives are managed. Finally, experience shows that most programs fail at the
construct and implementation phases.
WHO Reform Stage 2 Evaluation - Final report Appendices
PwC 12
2.1.4. Enablers for a transformation programme
Creating sustainable change requires more than simply focusing on discrete part of the transformation. Firstly change initiatives need to be setup, directed, managed and controlled if they are to deliver their intended benefits through disciplined programme management arrangements in the form of a Programme Management Office (PMO) or equivalent that enables the change initiative to be effectively managed and controlled.
Secondly change management and communication is critical in turning the potential failure of a change initiative into success, accelerating the delivery of benefits, increasing "stickiness" of changes and ensuring their sustainability. This is achieved through the application of a number of change management and communication activities and techniques.
These cross cutting activities span the whole life cycle of the transformation.
The resulting questions for the evaluation are as follows:
Domain Evaluation questions Criteria
Programme Management
Engaged stakeholders
Have stakeholders been identified, assessed and are they managed on an on-going basis throughout the reform process to ensure a high level of commitment?
Are they appropriately informed to contribute to the success of the programme?
Effectiveness
Clear scope Is the scope of the programme well defined, complete, communicated and agreed?
Does the scope support the objectives of the WHO High-level implementation strategy?
Effectiveness
Managed risks and opportunities
Are effective risk identification processes in place and are the key risks mitigated and opportunities taken?
Effectiveness
Delivery-enabling plans
Are the plans in place realistic, achievable, understood and bought into by key stakeholders executing the reform and those affected by it?
Efficiency
Effectiveness
Focused benefits management
Has a realistic business case been developed subjected to an appropriate level of challenge?
Are benefits clearly defined, owned and tracked for program success?
Efficiency
Effectiveness
Sustainability
High performing teams
Is the programme team highly motivated with the right blend of skills and personalities?
Are team members given enough time to deliver on the reform initiatives in addition to their current workload?
Does the organisation support the team to deliver?
Efficiency
Effectiveness
Smart financing Is the budget and associated policies, processes and reporting standards adequate for effective cost estimation and programme financial management?
Efficiency
Integrated suppliers
Have the right partners been selected?
What is the approach for ensuring suppliers understand the reform and help deliver change?
Effectiveness
WHO Reform Stage 2 Evaluation - Final report Appendices
PwC 13
Active quality management
Has an agreed quality plan been developed based on appropriate standards to monitor and evaluate the quality of the work performed?
Effectiveness
Embedded lifecycle assurance and learning
Has a clear assurance plan been defined which outlines the nature, timing and extent of planned assurance, quality reviews and embeds learning?
Effectiveness
Agile change control
What change control arrangements are in place to manage changes to the scope, requirements, deliverables and intended benefits that impact the project baseline, time or cost?
Efficiency
Effectiveness
Strong governance and reporting
What are the governance arrangements supporting the reform agenda?
How are outputs reported and is reporting targeting all stakeholders to the reform?
Effectiveness
Change Management
Business case, leadership, alignment and action
Does a change vision exist?
Is there agreement on the objectives of the reform?
Is the case for reform actively pushed by leaders?
How are leadership activities executed and monitored?
Relevance
Culture and behaviours
Does the WHO promote a culture conducive to reform?
Sustainability
Communication & stakeholders management
Has a change readiness assessment been carried out?
Is the communication delivered adequate and containing the right type of messages for different stakeholders of the WHO reform?
Are stakeholders identified, mapped and is there a stakeholder management plan?
Has a communication and engagement plan been defined? How is it executed and reported against?
Effectiveness
Sustainability
Change network and change management knowledge transfer
Has a change agent network been identified?
Are change agent network activities taking place on an on-going basis?
How is the change agent network supported?
Effectiveness
Sustainability
Clarity of working practices and organisational design
Has an organisational impact assessment been carried out for the reform overall and for reform initiatives specifically?
Is there clarity on the working practices, process documentation, organisational design and clear policies to ensure WHO staff understand the purpose of the reform and their role?
Effectiveness
Efficiency
Tools and guidance
Are systems, tools and guidance adapted to support the reform? Efficiency
Alignment of reward processes
Are personal objectives aligned?
How are benefits of the reform demonstrated?
How is the informal and formal reward process managed?
Sustainability
Demand from management
What are the mechanisms in place to coach champions and managers?
Does management regularly report on performance?
Sustainability
Training design and delivery
Are trainings set-up to ensure acquisition and transfer of knowledge and new skills?
Sustainability
Monitoring How is the success of the reform defined? Efficiency
WHO Reform Stage 2 Evaluation - Final report Appendices
PwC 14
How is it assessed?
How were KPIs defined?
What is the process and mechanism in place for monitoring and reporting?
How does the Reform Team respond to findings?
Sustainability Have learnings from pilots (initial implementations) been applied to full implementation?
Are processes, policies and rules being changed where necessary to support the reform?
Sustainability
2.2. Execution approach
2.2.1. Overall approach
In line with the Terms of Reference for the evaluation, we have tailored a structured and systematic approach that lays down the criteria for evaluation, instruments for delivering on the evaluation, and ultimately a socialisation approach culminating with the final report to be presented at the WHO Executive Board 134th Session in January 2014.
A ‘Mobilise’ Phase will allow us to agree with you on the scope, approach, evaluation framework and project management to quickly get started with the assignment. The product of the ‘Mobilise’ Phase consists of this Inception Report and has been developed on the basis of limited consultations with WHO stakeholders to gather initial views on the areas of focus of the evaluation and expectations with regards notably to the spirit in which work should be conducted.
An ‘Evaluate’ Phase will consist of the core of our work. We will review the entire reform process engaged by the WHO from the adequacy of its Implementation plan, to the execution of the activities under the Implementation plan. Section 2.2.3 below provides a detailed explanation of how we intend to conduct the evaluation and against which evaluation standards the reform will be evaluated. Section 3 outlines which instruments (desk review, interviews, observations and staff survey) will be used to support this analysis.
A ‘Synthesize and Report’ Phase will consist of consolidating the findings gathered in the previous phase, conducting a gap analysis, drawing conclusions and recommendations and delivering on the preliminary and final evaluation report. This phase also relates to socialisation of findings and presentation of the final report.
The overall approach is presented in the next page in Figure 2.
WHO Reform Stage 2 Evaluation - Final report Appendices
PwC 15
A
sses
sE
valu
ate
Synt
hesi
sean
d R
epor
t
Conduct a gap analysis
Assess WHO reform and current status
Confirm scope, objectives and approach for the evaluation
Conduct key stakeholder interviews
Review current practices
Draft reports
Present at Board Meeting
Agree and validate on the project plan
Mob
ilise
Implementation strategy• Quality and linkage to objectives
• Feasibility of timelines
• Resourcing, role and responsibilities
•Change management and program
delivery
Assess Phase• Conclusions from Stage One report
• Gaps in reform areas
• Solutions to fill gaps
Review plan execution• Operating elements
• Activities in progress and outputs
• Challenges and barriers to change
• Monitoring and Performance system
• Inception report• Confirmed scope and evaluation framework
• Preliminary findings and conclusions
• Preliminary report
• Final report
•Presentation to Executive Board
DeliverablesApproach
Ju
ly 20
13
Analysedocumenta-tion and data
Draw conclu-sion from findings
Discuss prelimi-nary results with EMG
Conduct a survey to assess change readiness
Incorpo-rate feedback from EMG
Synthesize and draw conclusion
Develop recommendations
Final reports
Ja
nu
ary 2
014
A substantial number of outputs of the reform are expected to be generated between July and December 2013,
i.e. during the duration of the evaluation. In order to provide timely feedback to the organisation despite a long
lead time to the final evaluation report presentation, our approach draws on real-time evaluation (RTE),
an approach in which a primary objective is to provide feedback in a participatory way in real time (i.e. during
the evaluation fieldwork) to those executing and managing the initiative under evaluation.
The aim is to improve the programme’s design and performance. In order to achieve this, we will contemplate frequent interactions with the Reform team, IOS, the internal Steering Committee for the evaluation and the Evaluation Management Group (EMG). Where deficiencies against evaluation criteria are identified, the evaluators will provide recommendations, tools or methods throughout the assignment that can support unlocking bottlenecks and addressing identified weaknesses.
2.2.2. The Mobilise Phase
This phase has consisted of engaging with the WHO Project Manager for this evaluation, David Webb, Director of IOS; Ian Smith, Advisor to the Director General Office and head of the Reform Team and -given the tight timeline to produce the inception report- a limited number of Member States who were present in Geneva for the Financing Dialogue on 24 June 2013 and available to meet.
The objectives of the meetings were to:
Get an initial understanding of the structure of the reform and a high level understanding of progress made so far since the start of the reform process;
Obtain a clearer picture of the activities in progress and delivered to date;
Figure 2: Overall approach
WHO Reform Stage 2 Evaluation - Final report Appendices
PwC 16
Identify specific expectations with respect to the spirit in which the evaluation work should be conducted.
We also conducted an initial collection of data and documentation relevant to the reform agenda. A desk review tool has been set-up and will continue to be populated as the assignment advances. It is presented in Annex A.1 of this inception report.
The backbone for the discussions was the ‘WHO reform story’, a synoptic one-pager document, developed by the Reform Team, which summarises the history of the reform process, its vision, objectives, priority areas and outcomes. A snapshot of the 12 initiatives is displayed for reference in Figure 3 below.
Figure 3: WHO Reform Story
WHO Reform Stage 2 Evaluation - Final report Appendices
PwC 17
The initial discussions with IOS and the Reform Team have allowed us to validate the objectives, expectations, scope, timelines and milestones for this assignment.
The outcome of this short mobilisation phase is contained in this inception report:
A conceptual framework that defines our evaluation standards and guiding questions. This is presented in Section 2.1.
A refined methodology to execute against the terms of reference of the evaluation. This is presented in section 2.2 below.
A data and information collection approach, including documents to be reviewed, interviews to be carried out, meetings to attend as observers and a staff survey to be conducted. This is presented in Section 3 below.
A schedule for the work execution. This is presented in Section 4.
Clarified roles and responsibilities for the evaluation. This is presented in Section 5. This report is now shared with the Evaluation Management Group (EMG) for feedback and endorsement during the week of the 15th July 2013.
2.2.3. The Evaluate Phase
Based on the consultation of the ‘Mobilise’ Phase and the conceptual framework presented in Section 2.1, we have refined the approach suggested in PwC’s initial proposal. It presents the details of how the evaluation will be executed and the tools or accelerators that will enable us to conclude on each step defined. The methodological approach will be broken down into two steps corresponding to the two objectives of the review:
Assess the status of WHO reform, and
Evaluate whether conditions for success of the reform are met. These two steps will drive the ‘Evaluate’ Phase and will be described in the below section. Each of the reform outcome defined in the Implementation plan will be evaluated against evaluation standards (derived from the Transformation dimensions and stages) considered essential to produce sustainable change within an organisation. We will also consider international best practices in the relevant reform areas, i.e. governance, oversight, monitoring and evaluation etc. that will act as standards of quality for each reform initiative. The evaluation will be guided by the meta-evaluation and overarching questions listed below, and as presented
in Section 3.2.2 of the Terms of Reference for this evaluation.
Principal evaluation question: - Is the WHO properly prepared to implement the reform agenda? Overarching questions: - Are the necessary conditions present to ensure an appropriate level of organizational preparedness
(institutional framework: structures, roles, instruments, procedures and guidelines, support systems for change process, communication, incentives)?
- What has been the progress made by the WHO in implementing the reform since 2011 and what has been the effectiveness, efficiency and sustainability of the measures undertaken?
- What have been the barriers to change and key bottlenecks that have slowed down the process of reform? Where issues have arisen have they been understood and tackled through appropriate actions?
WHO Reform Stage 2 Evaluation - Final report Appendices
PwC 18
2.2.3.1. Assess the Status of WHO reform
Assess status of action taken on Stage 1 recommendations
At this stage, we will review the recommendations issued by the Stage 1 Evaluation and assess if and how they have been incorporated into the Implementation plan. We will:
Review the process of prioritising the Stage 1 recommendations and assessing their feasibility of implementation;
Compare the versions of the Implementation plan and log the inclusions/exclusions of the Stage 1 recommendations;
Advise on Stage 1 recommendations to be considered by the Reform Team for presentation to the Member States.
This will be achieved through:
Desktop review (WHO meeting notes, progress updates, Implementation plans);
Interviews with the WHO Reform Team;
Interviews with Business Owners and for each reform output.
Assess progress made to date on the WHO reform Implementation plan
This stage will consist of assessing the progress made in each of the 12+1 reform initiatives against the expected
outcomes, outputs and deliverables presented in the Implementation plan. We will:
Review each outcome, output and deliverable and define at which stage of the transformation process
each reform initiative sits (i.e. Strategize & Assess, Design, Construct, Implement, Operate & Review)
presented in Section 2.1.3;
Having defined the level of maturity of the transformation, assess if the relevant tools and mechanisms
are in place for achieving the intended targets;
Assess the interdependencies of the reform initiatives affecting their timely execution;
Define gaps and bottlenecks in the effective implementation while also highlighting good practices;
Recommend international best practices (in the area of oversight, monitoring and evaluation, human
resources, accountability and risk management and communication) most relevant to the reform
initiatives under review to unlock bottlenecks.
This will be achieved through:
Desktop review;
Interviews with the WHO Reform Team;
Interviews with Business Owners and for each reform output.
Following preliminary and limited discussions with the Reform Team on the status of each reform output, we have created an initial status of the progress made so far towards reaching the objectives of the Implementation plan. In a nutshell, this tool was developed considering three analytical elements:
The reform agenda was broken down by reform initiative, outcome and related outputs;
Each reform output was assessed against each transformation stage as described in the conceptual framework in Section 2.1.3 to understand its level of advancement;
A progress status was identified per stage and per output and rated 1) completed, 2) in progress or 3) initiated.
This tool will be the accelerator supporting the analysis at this stage.
WHO Reform Stage 2 Evaluation - Final report Appendices
PwC 19
In the tool presented in Figure 4, each reform output has been assessed against the definitions of the stages of advancement of a reform process and developed into a wheel similar to the one the ‘WHO Reform story’. This initial assessment will be challenged and reviewed with a broader set of stakeholders during the course of the evaluation. Our future findings will allow us to update and refine this ‘WHO wheel’. Our initial findings show us that levels of advancement in terms of execution vary from one reform initiative to the other and within a reform initiative the status of progress of each outcome and deliverables also varies. Based on interviews and high level document review and through preliminary consultation with the Reform Team, it shows that most progress has been achieved in the programmatic initiative (1.1) while the initiatives linked to stakeholder engagement (2.5), streamlining national reporting (2.4) and providing effective support to Members States (3.1) are at the initial stages of the reform. From our initial consultations, advancing on the financing and the human resources initiatives of the reform are considered important for the success of the WHO reform.
Assess & Strategise
Stage
Design
Construct
Implement
Operate & Review
Not initiated
In progress
Completed
Status
Legend
L&D
Conflict of interestManagerial accountability,
transparency and risk
management
Figure 4: WHO wheel and preliminary status on reform
WHO Reform Stage 2 Evaluation - Final report Appendices
PwC 20
This stage of the evaluation will aim to answer key questions presented in Section 3.2.2 of the Terms of
Reference for this evaluation which include:
Does the Implementation plan adequately address all the elements of delivering on effective governance To
what degree have appropriate strategies, including resource mobilisation and allocation, and corporate strategies such as human resources management and gender mainstreaming, been identified to implement the reform proposals?
Assess the completeness of the WHO reform Implementation plan At this stage, we will review the adequacy and comprehensiveness of the results chain and theory of change presented in the Implementation plan and assess if this is adequately translated into reform operational plans. To achieve this we will:
Assess if all dimensions of a transformation, i.e. Strategy, Structure, Processes, People and Technology have been addressed by the Implementation plan (see Section 2.1.2) and what type of outputs are defined per dimension;
Assess the robustness of the results chain and assumptions underlying the theory of change- this will be done by reviewing the causal linkages between the inputs, outputs, outcomes and deliverables and contextual/external factors which may affect the achievement of the targets. We will notably verify if outputs are specified for all stages of transformation;
Assess the translation of the Implementation plan into operational plans for each reform initiative;
Assess if the completeness of the inputs (consider all relevant dimensions) and are adequate to achieve outputs in the set timeframe;
Identify gaps in the theory of change and in the operational plans;
Assess the robustness of the monitoring framework and accountability systems in place to ensure reporting on the results of the Implementation plan.
This will be achieved through:
Desktop review;
Interviews with the WHO Reform Team;
Interviews with Business Owners and for each reform output;
Interviews with WHO Senior Management Team and Member States.
This stage of the evaluation will aim to answer key questions presented in Section 3.2.2 of the Terms of
Reference for this evaluation which include:
Does the Implementation plan adequately address all the elements of delivering on effective governance
and change- strategy, structure, process, people, technology? Is it detailed and comprehensive enough to allow for adequate implementation? Is a new design proposed for one or all of the elements? If so, how adequate and feasible is the new design in terms of execution and timelines?
Are indicators and timelines sufficient, feasible and appropriate to effectively measure (monitor and evaluate) performance and report on the desired outcomes of the reform process?
2.2.3.2. Evaluate whether conditions for success of the reform are met
This second step of the Evaluate Phase is linked to the second objective of the evaluation and contemplates the
enablers of a transformation programme. Firstly, we will assess the current capacity of the Reform Team and
other actors in charge of the reform and review how they currently manage the reform agenda. Concurrently,
we will assess if the elements of successful change management are present. This aspect of the evaluation will
be supported by the elements and questions presented as part of Section 2.1.4.
WHO Reform Stage 2 Evaluation - Final report Appendices
PwC 21
Assess programme management
At this stage, we will
Evaluate the program management component of the reform, as presented in Section 2.1.4;
Review the project management and excel tools in use by the Reform Team for monitoring reform results;
Assess the adequacy of the stakeholder management strategy, risk management process, reporting/ monitoring frameworks and financial management;
Conduct a gap analysis and identify the critical enablers that are missing for the reform to have in depth and transformative results
This will be achieved through:
Review of project management tools;
Interviews with the WHO Reform Team;
Interviews with Business Owners and for each reform output;
Assess change management
At this stage, we will:
Evaluate if the change management practices currently in place are supported by the adequate balance of leadership, communication, availability of tools, trainings and monitoring of change, as presented in Section 2.1.4;
Review the structure of the communication strategy for reform and its implementation;
Assess WHO’s culture alignment with the change vision;
Review the content of change leadership and network activities and assess their adequacy;
Conduct a gap analysis and identify the critical enablers that are missing for the reform to have in depth and transformative results
This will be achieved through:
Desktop review;
Interviews with the WHO Reform Team;
Interviews with Business Owners and for each reform output;
Interviews with WHO Senior Management Team and Member States;
Representatives from Regions and Country offices;
Staff survey.
Assess staff perception and readiness for change
In order to cross-check interview and assess whether conditions for successful change implementation are
present, we complement the above assessment with a change readiness assessment focusing on staff feedback
and perception. This will allow us to test how effective management approach to change has been to date.
The instrument to achieve this will be an electronic survey sent to all staff. It is described in Section 3.4 below.
This stage of the evaluation will also contemplate key questions presented in Section 3.2.2 of the Terms of
Reference for this evaluation including:
What is the efficacy of internal and external instruments and modalities to implement the reform proposals, including related risk management plans and risk mitigation actions?
To what degree is there coherence in the governance mechanisms to inform, monitor and evaluate the implementation of the WHO reform package? To which extent are these mechanisms and processes adequate to prioritise the various components of the reform proposal?
WHO Reform Stage 2 Evaluation - Final report Appendices
PwC 22
What are the specific strategies to address change management? How well are the pertinent issues addressed?
2.2.4. The Synthesize and Report Phase
This phase of the evaluation relates to the finalisation of the evaluation preliminary and final reports,
their dissemination and the broader socialisation of the evaluation findings and recommendations.
The following is planned:
Drafting of the preliminary report;
Discussion of findings and recommendations with WHO Steering Committee and WHO Senior
Management Team to socialise preliminary findings, test conclusions and adjust as required prior to
submitting the preliminary evaluation report to the EMG;
Distribution of preliminary report to the EMG on 2 September 2013 for review, feedback and guidance;
Update of report based on the above EMG feedback received by 30 September 2013 and finalisation;
Distribution of the final report to the EMG and PBAC in October 2013;
Preparation and finalisation of summary presentation of findings;
Presentation to the Audit Committee IEOAC at the November session;
Presentation to PBAC at their January meeting;
Report of the PBAC to the Executive Board in January 2014.
Ad-hoc sessions will be organised with the Reform Team and Business owners as questions arise. We will have
regular contact with the Project Manager, David Webb, the Reform Team, the Steering Committee, the EMG
and senior WHO stakeholders throughout the course of the assignment.
Close proximity with the WHO will ensure that our evaluation results meet the needs of the Executive Board.
WHO Reform Stage 2 Evaluation - Final report Appendices
PwC 23
3. Data collection instruments
Four primary means of data collection will be used to collect data pertaining to the terms of reference of the evaluation:
Desk review
Interviews with key stakeholders
Observation
Staff survey on the reform Details are provided in the sections below.
3.1. Desk review
We will collect data through the review of key documentation and secondary evidence on the WHO reform process such as:
The Independent evaluation report: Stage One, External Auditor WHO
WHO reform documentation including Board minutes
Any project documentation linked to the reform agenda
Reports from the UN Joint Inspection Unit including the ‘Decentralization of Organisations within the United Nations System- Part III: the World Health Organisation’ and the ‘Review of management and administration in the WHO’
Programme and Budget for the Biennium
Detailed organisational charts
Internal strategies including HR policies
The list of current available documents can be found in Annex A.1.
We will use a desk review tool to be populated as the assignment advances, in collaboration with the WHO
counterparts. All documents will be made available on the PwC secure workspace that is specially created by the
Independent Evaluation Team for sharing documents with the WHO focal points.
3.2. Interviews
All interviews will be conducted by a minimum of two members of the Independent Evaluation Team, thus ensuring adequate capturing and validation of findings. Data from interviews and meetings will be uploaded into a database and mapped against each area of the reform in order to structure evidence effectively for analysis purposes. Table 1 below presents the list of interviewees.
WHO Reform Stage 2 Evaluation - Final report Appendices
PwC 24
Table 1: List of interviewees
High-level WHO Secretariat HQ
Margaret Chan, Director General
Anarfi (AB) Asamoa-Baah, Deputy Director-General, HQ/ODG Office of the Director-General
Andrew Cassels, Director (Strategy), HQ/ODG Office of the Director-General
Hans Troedsson, Executive Director, HQ/ODG Office of the Director-General
Bersabel Ephrem, Senior Adviser to the ADG/GMG, HQ/GMA GMG ADGO Office of the Assistant DG
Dr. Mohamed Jama, Assistant Director-General, HQ/GMA GMG ADGO Office of the Assistant DG
Regional
Dr Luis G. Sambo, Regional Director, AFRO
Dr Carissa Etienne, Regional Director, AMRO
Ms Zsuzsanna Jacab, Regional Director, EURO
Dr Samlee Plianbangchang, Regional Director, SEARO
Dr. Shin, Regional Director, WPRO
Dr. Ala Alwan, Regional Director, EMRO National:
TBD
Business Owners:
Najeeb Al Shorbaji, Director, HQ/KMS Knowledge Management and Sharing
Gian Luca Burci, Legal Counsel, HQ/LEG Office of the Legal Counsel
Christy Feig, Director, HQ/DCO Department of Communications
Nicholas Jeffreys, Comptroller & Director, HQ/FNM Department of Finance
Daniel Lopez-Acuna, Advisor to Director-General, HQ/ODG Office of the Director-General
Elil Renganathan, Director, HQ/PRP Planning Resource Coordination and Perf Monitoring
Mari-Andree Romisch, Director, HQ/CCO Country Collaboration
Cecilia Rose-Oduyemi, HQ/GBS Dept for Governing Bodies and External Relations
Mahen Sandrasagren, Coordinator/ acting Director, HQ/HRQ HQ HR Management
Gaudenz Silberschmidt, Senior Advisor to the Office of the Director-General, HQ/ODG Office of the Director-Genera
Ian Smith, Advisor to Director-General, HQ/ODG Office of the Director-General
David Webb, Director HQ/IOS Office of Internal Oversight Services
Additional informants of reform initiatives will take place on an as-needed basis
Member States
Gabrielle Jacob, First Secretary (Health), Permanent Mission of Ireland
Peter Mamacos, Multilateral Branch Chief, US Department of Health and Human Services
Colin Mciff, Health Attaché, US Permanent Mission, Geneva
Cath Patterson, Minister Counsellor (Health), Australian Permanent Mission to the UN, Geneva
Magali Girod, Policy Adviser, , UK Mission to the United Nations in Geneva
John Stuppel, Health and UN Cross Systems Team Leader, DFID, UK
Mr B. Kuemmel, Adviser, Federal Ministry of Health, Germany
Elissa Golberg, Ambassador Canada
Onoda Tomoko, Second Secretary Japan
Jan Knutson, Ambassador, Sweden
Sverre Berg Lutnaes, Counsellor, Norway
Maria Luisa Escorel De Moraes, Minister Counsellor, Brazil
Alexey Kulikov, Third Secretary, Russia
Anastacia Osundwa, Kenya
Other Geneva-based Members States representing various groupings- TBD
EMG
Ms Jane Halton, Australia (Chair)
Ms Precious Matsoso, South Africa
Dra Zelibeth Valverde, Panama
Prof. Mohammad Hossein Nicknam, Islamic Republic of Iran
Prof. Ogtay Shiraliyev, Azerbaijan
Prof. Pe Thet Khin, Myanmar
Dr Sukai Prom-Jackson, External, Member (Inspector, Joint Inspection Unit)
Steering Committee
Matshidiso Moeti, Director, WHO, AFRO
Jon Kim Andrus, Deputy Director, WHO, AMRO
Samir Ben Yahmed, Director, WHO, EMRO
Imre Hollo, Director, WHO, EURO
Herbert Tennakoon, Assistant Regional Director, WHO, SEARO
Tieru Han, Director, WHO, WPRO
WHO Reform Stage 2 Evaluation - Final report Appendices
PwC 25
Interviews will be conducted in a phased approach, starting with Business Owners, EMG and Steering Committee members and WHO HQ and Regional Directors. Member States representatives will be interviewed throughout the interview period pending on availabilities.
3.3. Observation
Another mean for data collection will consist of attending specific WHO meetings as observers. The following meetings are considered as important avenues for better understanding the context of the reform agenda and governance structure.
Financing Dialogue, Taking steps towards a fully-funded programme budget 2014-2015, 24 June 2013.
The Independent Expert Oversight Advisory Committee 10th meeting, Sessions on WHO reform and Financing Dialogue, 3 July 2013.
Regional Director’s Program Management (DPM) network meeting, Sessions on national reporting, 9 July 2013.
3.4. Survey
After drawing preliminary conclusions from the interviews described above, the Independent Evaluation Team recommends undertaking a Change Readiness Assessment which covers key elements of change management and will aim to test our conclusions with a wider audience using an electronic survey. This will allow us to test how staff perceives the reform process, how projects are heading, and if they feel adequately equipped to implement them. The survey will be an opportunity to reach out to WHO regional and country offices. By using a secure survey platform, we will administer the survey on an anonymous basis to all WHO staff for a
period of three weeks. Results will be collected by the Independent Evaluation Team. We will develop survey
questions/statements that can be easily rated, from a scale of 1 to 5, for the purpose of the analyses.
Appendix A-2 presents an initial list of questions that we intend to ask, against the following change
management themes:
Awareness of WHO reform;
Impact of the reform on the person;
Impact of the reform on the Workplace;
Communication on the reform;
Engagement with the reform process;
Leadership;
Evidence of Impact of the Reform;
Organisational Climate.
WHO Reform Stage 2 Evaluation - Final report Appendices
PwC 26
3.5. Methodology Limitations
In defining the conceptual framework and execution methodology, the following limitations have been
identified and mitigated:
Domain Limitation Mitigation
Conceptual Framework
The conceptual framework, although logically sound, cannot be used as straight jacket – it needs to be fined tuned to every situation
Initial framework has been fined-tuned to WHO specific context
Evaluators will assess whether there are elements of the framework that do not apply to a specific reform initiative
Execution approach
Moving picture: the evaluation is taking place as reform progresses. Situation may be changed between the initial interviews and the final evaluation
Be clear that we are assessing the reform as it stand early July 2013. Should elements change, these will be mentioned in the final report.
WHO Management to also issue their own report on progress since the evaluation for the January PBAC meeting
Timing: the evaluation is planned to be performed during July and August 2013, a period when most northern hemisphere stakeholders are likely to be away and may not be available to the Independent Evaluation Team. This could create biases.
Early communication to all stakeholders to be interviewed
Extension of staff survey period over July and August
Timing: reform is mostly in the design/implementation stages, which do not allow to evaluate the overall impact of the Reform
Clearly exclude assessment of impact from evaluation scope
Consider Stage 3 evaluation in 3-4 years
Data collection instruments
Desktop review could provide a false sense of comfort that reform is happening because documented
Challenge the status of reform in interviews. Cross check with result of staff survey
Interviews could be instrumentalised to provide a biased picture of current state (positive or negative)
Clear assessment criteria
Cross-check responses from various groups to the same questions
Cross-check with other instruments
Survey may not be statistically meaningful and only provides “perception”
Ensuring adequate sampling and response rate through reminders
Focus the survey on the very perception (“I feel”, “I think”), as opposed to looking for factual elements
Ensure translation in WHO working languages
Ensure anonymity
Observation. The presence of the observer may result in the events being observed, not being conducted as they would otherwise be. Also given the timeframe we can only observe events taking place in the July-August period.
Cross-check with other instruments
Be discrete during events under observation
WHO Reform Stage 2 Evaluation - Final report Appendices
PwC 27
4. Project Management Plan
As described in the Gantt chart, the table below sets out the expected deliverables, milestones and feedback
expected by the EMG.
Deliverables Deadlines Inception report 8 July 2013 EMG feedback on inception report 15 July 2013 Preliminary report 2 September 2013 Feedback on preliminary report 30 September 2013 Final Evaluation report 9 October 2013 Summary PowerPoint presentation 20 December 2013 Presentation to 134th session of the Executive Board 20 January 2013 (TBC)
We have planned for monthly EMG meetings which we suggest should include discussion of initial findings,
approach and method, survey, key stakeholder involvement, reporting and any other important project
elements. We assume that EMG feedback on the inception report and preliminary report will be provided by 15
July 2013 and 30 September 2013 respectively. We also understand that the date proposed for the 134th session
of the Executive Board is 20 January 2013, date at which we will provide our presentation of the evaluation
findings. The IOS Secretariat will be arranging and scheduling meetings.
We have planned bi-monthly meetings with the Steering Committee. These meetings will cover reviews of
evaluation tools and protocols, socialising findings and offer technical guidance. The IOS Secretariat will be
arranging and scheduling meetings.
We have planned bi-weekly meetings or phone calls with IOS to discuss updates on the assignment and resolve
any operational issue.
We have planned for regular interactions with the Reform Team to ensure timely scheduling of interviews with
key informants.
The Gantt chart below presents the detailed workplan that will guide the execution of the Evaluation work.
WHO Reform Stage 2 Evaluation - Final report Appendices
PwC 28
2013 2014
Planning (# weeks) 24/06 01/07 08/07 15/07 22/07 29/07 05/08 12/08 19/08 26/08 02/09 09/09 16/09 23/09 30/09 07/10 14/10 21/10 16/12 23/12 30/12 06/01 13/01 20/01
Mobilise 2 weeks
1. Collect information 1 week
1.1 Collect data
2. Assess WHO reform status (nb Excel model and notes) 1 week
2.1 Confirm scope and objectives
2.2 Conduct initial interviews with key stakeholders
2.3 Analyse progress made towards 12 reform areas
2.4 Review key documentation
3. Reporting 1 week
3.1 Define detailed approach to evaluation
3.2 Produce inception report
Evaluate 6 weeks
4. Conduct key stakeholders interviews 4 weeks
4.1 Plan and organize interviews
4.2 Conduct interviews
5. Desk review 1 week
5.1 Collect data
5.2 Review key documentation
6. Conduct survey 4 weeks
6.1 Define scope
6.2 Plan survey
6.3 Roll-out survey
6.4 Analyse results
Synthesize & Report 12 weeks
7. Data analysis 4 weeks
7.1 Conduct gap analysis
7.2 Synthesize findings and results from survey
8. Preliminary report 3 weeks
8.1 Report writing
9. Inclusion of EMG feedback 2 weeks
10. Final report 2 week
10.1 Report writing
11. Socialisation & Presentation at Board meeting 3 weeks
11.1 Presentation writing
11.2 Socialisation
11.3 Presentation at Board meeting
TOTAL 20 weeks
Key milestones
Deliverables
Deadlines
Resource management
EMG
Steering Committee
IOS
Reform Team
Final Pres at
EB
20.01.2014
Inception
report
EMG
feedback
15.07.2013
Preliminary
report
02.09.2013
EMG
feedback
30.09.2013
Final report
09.10.2013
Final Pres
20.12.201308.07.2013
Key risk associated with availability of stakeholders during the months of July-August 2013
WHO Reform Stage 2 Evaluation - Final report Appendices
PwC 29
WHA
MemberStates
EB
DGO IOSEvaluation
Management Group
Steering
Committee
WHO
Secretariat
Oversight
of Stage 2 Evaluation
Engagement Partner
Project
Director
Independent
Evaluation Team
Governance
Reform
TeamDG/DDG
Regional
Directors(6)
RegionalPolicy Group
Business
Owners
Consultant
SME
5. Roles and responsibilities
5.1. Project management structure
Gill Sivyer is the responsible Partner for this evaluation. She is at the disposal of the EMG and the IOS Project
Manager for any remarks or suggestions that they may have and will ensure the quality of our services and the
supervision of the Independent Evaluation Team. She will also take part in key meetings.
The main PwC contact for any project work on a day to day basis is Antoine Berthaut, Project Director.
Antoine Berthaut is supported by:
Josephine Pallandt, Senior Manager, who will be responsible for governance aspects of the evaluation;
Sarah Des Rosiers, Senior Consultant, who has a background in public health, and who will work under the guidance of Antoine and Josephine to conduct the analysis.
The team has adequate gender balance and cultural diversity representation (American, French, Dutch,
Canado-Haitian).
Additional consultants from PwC will be mobilised on an as needed-basis (e.g. to address specific technical
questions to complement the core team described above). An external Subcontractor maybe involved as Subject
Matter Expert, prior to approval by WHO.
The below figure demonstrates the linkages between the Independent Evaluation Team and the WHO entities
to be involved in the Stage 2 Evaluation as well as the planned interactions between both entities.
Figure 5: WHO Stage 2 reform organigram
WHO Reform Stage 2 Evaluation - Final report Appendices
PwC 30
5.2. Role of WHO
The below table presents the roles and responsibilities of the WHO counterparts described above and sets out
who will be involved with either the day-to-day management of the Independent Evaluation Team, the
oversight of the execution of the evaluation or the facilitation of the key informant interviews throughout the
course of the evaluation.
WHO role Nature of involvement Estimated workload
IOS/Project Manager Mr David Webb, Director, IOS Dr Maria Santamaria, Coordinator Evaluation, IOS
Day to day oversight of the Independent Evaluation Team and responsible for all aspects concerning contracting
Facilitation of access to interviewees and documentation
Focal point for progress updates and provide input into evaluation approach
Focal point for providing technical oversight over the assignment and focal point for progress updates
Advising of key informants and facilitating introductions
Facilitate, organise and participate in EMG and steering committee meetings
2 days/week
Reform Team Ian Smith, Advisor to the Office of the Director-General Gaudenz Silberschmidt, Senior Advisor to the Office of the Director-General Ms Linda Larsson (WHO Reform documentation) Ms Michelle Dean (logistics arrangements)
Provide documentation on the WHO reform
Advise on list of key informants
Provide logistical arrangements involved in the organisation of interviews and focused group meetings
Provide ad-hoc explanations related to the WHO reform
Participate in assessment of WHO reform status
Support definition of survey participants
1-2 days/week
Evaluation Management Group Ms Jane Halton, Australia (Chair) Ms Precious Matsoso, South Africa Dra Zelibeth Valverde, Panama Prof. Mohammad Hossein Nicknam, Islamic Republic of Iran Prof. Ogtay Shiraliyev, Azerbaijan Prof. Pe Thet Khin, Myanmar Dr Sukai Prom-Jackson, External Member (Inspector, Joint Inspection Unit)
Direct the evaluation work and make key decisions on the evaluation work
Ensure the project's scope aligns with the Terms of Reference for the evaluation and represent Member States interests
Provide direction and guidance as work progresses to ensure objectives are met and risks mitigated
Provide technical guidance on the evaluation approach and quality enhancement
Validate findings and discuss recommendations
Validate and approve key deliverables
Ensure quality and timeliness in decision-making
0.5 day every 2 weeks
Steering committee Dr Anarfi Asamoa-Baah, DDG Hans Troedsson, Executive Director, Office of the Director-General Ian Smith, Reform Team David Webb, IOS Mohamed Jama, Assistant Director General, General Management 6 Regional Nominees
Follow-up on the execution of the evaluation and act as a sounding board
Guide and provide feedback on data collection and tools
Address technical questions
Review initial findings and facilitate the socialisation of findings
Ensure transfer of knowledge of the evaluation work within the WHO
0.5 day/week
WHO Reform Stage 2 Evaluation - Final report Appendices
PwC 31
5.3. Quality assurance
Quality assurance is integrated throughout the evaluation and is based on the four eyes principle. The following checkpoints will be built into the key phases of the evaluation:
Mobilise: Review of evaluation terms of reference by Gill Sivyer and the respective subject matter expert.
Evaluate: - review of the detailed work programme to ensure appropriate scope and depth of coverage - review of workpapers at completion of fieldwork to ensure all necessary steps have been completed
and documented - review of issues raised to ensure consistency of evaluation, justification and relevance prior to
discussion and validation with evaluees.
Reporting: Review of the written report by both Gill Sivyer and PwC internal quality assurance to ensure the evaluation meets the requirements of WHO and to ensure that all issues are fact based and recommendations are practical.
Closure: At the conclusion of the evaluation a formal satisfaction review will be undertaken to ensure that
lessons learned are identified and built into subsequent reviews of a similar nature."
WHO Reform Stage 2 Evaluation - Final report Appendices
PwC 32
Appendix A. - Data collection
A.1. Desk review tool
The list of documents available in this table has been provided largely by the Reform Team and will be
populated during the course of the evaluation as we enter into the ‘Evaluate’ Phase.
Reform Agenda Documentation
Programmatic reform
Programmatic priorities Draft twelfth general programme of work, SIXTY-SIXTH WORLD HEALTH ASSEMBLY,19 April PROPOSED PROGRAMME BUDGET 2014–2015, SIXTY-SIXTH WORLD HEALTH ASSEMBLY,19 April Twelfth General Programme of Work, 2014–2019, SIXTY-SIXTH WORLD HEALTH ASSEMBLY, 24 May 2013 Programme budget 2014–2015, SIXTY-SIXTH WORLD HEALTH ASSEMBLY, 24 May 2013
Governance reform
Oversight by the governing bodies
Governing Body minutes of meetings and other relevant gatherings and discussions
Revised terms of reference for the Programme, Budget and Administration Committee of the Executive Board, EB131.R2, 28 May 2012
Harmonisation and alignment of governance processes
Project governance documentation, governance Principles , project pans and charters, project agreements, project minutes
Enhanced strategic decision-making by governing bodies
Agenda’s and minutes of relevant governing body meetings, founding documents, charters.
Streamlined reporting of and communication with Member States
Current reports and communications with Member States about the reform initiatives, relevant outputs and tools, Reform team & other key stakeholder meeting minutes
Strengthened effective engagement with other stakeholders
Current engagement initiatives /plans / terms of references, minutes of relevant external stakeholders meetings, outputs and tools used to date
Managerial reform
Effective technical and policy support for all Member States
Review of management, administration and decentralisation in the World Health Organization, report by the Joint Inspection Unit, EB132/5 Add.6, 4 January 2013 Report of the Taskforce on the roles and functions of the three levels of WHO
Staffing matched to needs at all levels of the Organisation
Amendments to the Staff Regulations and Staff Rules1, EB132/40, 11 January 2013
Financing and resource allocation aligned with priorities
Amendments to the Financial Regulations and Financial Rules, A66/57, 20 May 2013 WHO reform, Financing of WHO, A66/48, 13 May 2013 WHO reform, Financing of WHO, A66/48, 20 May 2013
Managerial accountability, transparency and risk management
WHO Reform Stage 2 Evaluation - Final report Appendices
PwC 33
Strengthened culture of evaluation
Formal evaluation policy, B131
Improved strategic communications
Change management
ADG meeting notes ADG Meeting notes, 2012/2013
Communications Change at WHO, Newsletter on Reform, November 2012 and April 2013 WHO reform Engagement and Communication Strategy WHO reform story
GPG GPG meeting and retreat notes, 212/2013
IEOAC IEOAC Annual Report 2012 IEOAC Agenda, Feb, July, Nov 2012, March 2013 IEOAC, July-November 2012 PBAC IEOAC 7th session final report, 31 July 2012 IEOAC, Report of the Sixth Meeting of the Independent Expert Oversight Advisory, February 2013
Project Management Tool Reform tool
Reporting to MS WHO reform High-level implementation plan and report, 20 May 2013 Implementation of WHO reform, 2012, High-level implementation plan, January 2013
Other documentation
External evaluation of the WHO reform
WHO reform, Independent evaluation report: stage one, 18 May 2012 GAO report, World Health Organization, 2012
UN agencies reform
Best practices in reform areas
WHO Reform Stage 2 Evaluation - Final report Appendices
PwC 34
A.2. Staff survey questionnaire (initial list of questions to
be reviewed)
For each of the statements below, please indicate whether you:
1) Strongly disagree;
2) Disagree;
3) Neither agree nor disagree;
4) Agree;
5) Strongly agree.
Themes # Survey questions
Awareness
1 I am aware of the WHO Reform
2 I understand the vision, objectives and planned outcomes of the WHO Reform
3 I understand clearly what our organisation will be like after the WHO Reform
4 If our organisation does not make this change, we will be affected negatively in the long term
5 I am confident that the WHO Reform will be implemented successfully
6 I feel there is a clear action plan for implementing the changes associated with the WHO Reform
Personal
7 I will personally benefit from the WHO Reform
8 I am motivated to implement reform initiatives in my workplace
9 I understand how the WHO Reform will impact my work activities
10 I understand my role and what I can do in my job to make the Reform a success
11 I understand the skills and knowledge that I will need to demonstrate in order to be successful in the new organisation
12 I am confident I will receive the training I need in order to effectively do my job
13 I understand the benefits that the initiative will bring to my work area
Workplace
14 I think there is a manageable level of change in our work unit
15 My Workplace has embraced the changes
16 There is sufficient cross functional co-operation for this change to succeed
WHO Reform Stage 2 Evaluation - Final report Appendices
PwC 35
Communication
17 I feel well informed and kept up to date on relevant WHO Reform initiatives
18 I know where to find information about the WHO Reform
19 I usually learn about organisational changes through formal communication channels (e.g., newsletters, presentations, etc.) rather than through informal communication channels (e.g., contacts, “grapevine”, etc.)
20 There are adequate ways for me to inquire, contribute or voice my concerns about to the Reform
21 Communication on reform between HQ and the regions is effective
22 Through which modes of communication would you prefer receiving information about the Reform? [list to be provided]
Engagement 23 I feel involved in the proposed changes
24 I have received timely and adequate guidance and support from my superiors to help me drive reform
Leadership
25 Senior Management considers the impact of change on the field
26 I have participated in discussion with one or more of my supervisors about the WHO Reform
27 Leaders, superiors and managers have the change skills necessary for this WHO Reform
28 My superiors support the WHO Reform and are positive about its arrival
29 My superiors are adopting new processes and contributing to the WHO Reform
30 I think that adequate support across the WHO and resources will be provided to accomplish the WHO Reform
Evidence of Impact
31 I have seen evidence of reform initiatives being implemented
32 I have seen positive results from the WHO reform
Climate 33 The climate for change is positive and supportive in the organisation
WHO Reform Stage 2 Evaluation - Final report Appendices
PwC 36
Appendix B. - Preliminary and Final report
B.1. Outline of the preliminary and final report
Table of Content
1. Executive Summary
2. Introduction
3. Scope of Evaluation and Questions
4. Evaluation Areas
4.1 Assess the WHO reform
4.1.1 Key findings
4.1.1.1 Action taken on Stage 1 Recommendations
4.1.1.2 Assess progress made to date on the WHO reform Implementation plan
4.1.1.3 Assess the completeness of the WHO reform Implementation plan
4.1.2 Challenges and Opportunities
4.1.3 Gap analysis
4.1.4 Recommendations
4.2 Evaluate whether conditions for success of the reform are met
4.2.1 Key findings (Programme and Change Management)
4.2.2 Results of survey
4.2.3 Challenges and Opportunities
4.2.4 Gap analysis
4.2.5 Recommendations
5. Suggested Action Plan
6. Conclusion and Way forward
7. Appendix- Assessment against evaluation questions
WHO Reform Stage 2 Evaluation - Final report Appendices
PwC 37
Appendix C. - UNEG Quality Checklist
C.1. UNEG Quality Checklist for evaluation Terms of
reference and Inception Reports
Evaluation Title: Stage 2 Evaluation of WHO Reform
Commissioning Office: Executive Board
1. Evaluation Purpose
1.0 The Terms Of Reference specifies the purpose of the evaluation and how it will be used.
Section 1.3
1.1 The TOR references the mandate for the conduct of the evaluation. Section 1.2
1.2 The purpose of the evaluation identified in the TOR clearly states why the evaluation is being done, including justification for why it is being done at this time.
Section 1.3
1.3 The TOR identifies the primary and secondary audiences for the evaluation and how the evaluation will be useful.
Section 1.3 Introduction
2. Evaluation Objectives2
2.0 The Terms Of Reference includes clearly defined, relevant and feasible objectives. Section 1.3
2.1 The evaluation objective(s) clearly follow from the overall purpose of the evaluation. Section 1.3
2.2 The TOR evaluation objectives are realistic and achievable, in light of the information that can be collected in the context of the undertaking.
Section 1.3
3. Evaluation Context
3.0 The Terms Of Reference includes sufficient and relevant contextual information. Section 1.2
3.1 The TOR adequately describes the particular political, programmatic and governance environment in which the evaluation will be taking place. For example, the most relevant aspects of the economic, social and political context are described.
Section 1.2
3.2 The TOR adequately describes the most relevant programmatic and/or thematic aspects relevant to the evaluation.
Section 1.2
4. Evaluation Scope
4.0 The Terms of Reference includes the scope of the evaluation. Section 1.3
4.1 The TOR explicitly and clearly defines what will and will not be covered, including, for example, the timeframe, phase in the project and/or geographical area to be covered by the evaluation.
Section 1.3 Introduction
4.2 The scope of the evaluation is adequate to meet the stated evaluation objective(s). Yes.
4.3 The scope of the evaluation is feasible given resources and time considerations. Yes – providing stakeholders availability for interviews.
5. Evaluation Criteria
5.0 The Terms of Reference specifies the criteria that will be utilised to guide the evaluation.
Section 2.1, notably 2.1.1
5.1 The TOR specifies the evaluation criteria against which the subject to be evaluated will be assessed, including, for example, relevance, efficiency, effectiveness, impact and/or sustainability.
Section 2.1, notably 2.1.1
5.2 The TOR spells out any additional criteria of relevance to the particular type of evaluation being undertaken, such as evaluations of development, humanitarian
Section 2.1, notably 2.1.2,
WHO Reform Stage 2 Evaluation - Final report Appendices
PwC 38
response, and normative programmes. 2.1.3 and 2.1.4
5.3 The scope of the evaluation is feasible given resources and time considerations. Yes – providing stakeholders availability for interviews
6. Tailored Evaluation Questions
6.0 The Terms of Reference includes a comprehensive and tailored set of evaluation questions within the framework of the evaluation criteria.
Section 2.1.2, 2.1.3 and 2.1.4
6.1 The TOR contains a set of evaluation questions that are directly related to both the objectives of the evaluation and the criteria against which the subject will be assessed.
Section 2.1.2, 2.1.3 and 2.1.4
6.2 The set of evaluation questions adds further detail to the objectives and contributes to further defining the scope.
Yes.
6.3 The set of evaluation questions is comprehensive enough that they raise the most pertinent evaluation questions, while at the same time being concise enough to provide users with a clear overview of the evaluation’s objectives.
Yes.
6.4 Factoring in the information that will be collected and the context of the evaluation, evidence backed answers to the set of evaluation questions is achievable.
Section 2.2.2 and 3
7. Methodology
7.0 The Terms of Reference specifies the methods for data collection and analysis, including information on the overall methodological design
Section 4 (data collection) and section 2 (Conceptual Framework)
7.1 The TOR contains a clear and accessible methodological plan. Preferably, a standalone section that is clearly delineated from other information contained in the TOR.
Section 2, 3 and 4
7.2 The TOR states the overall methodological approach and design for the evaluation. Section 2, 3 and 4
7.3 The data collection and analysis methods in the TOR are sufficiently rigorous to assess the subject of the evaluation and ensure a complete, fair and unbiased assessment.
Section 4
7.4 The evaluation methodology includes multiple methods (triangulation); preferably with analysis of both quantitative and qualitative data and with a range of stakeholders covered by the data collection methods.
Yes. Refer Section 4
7.5 Logical and explicit linkages are provided between data sources, data collection methods and analysis methods. For example, sampling plans are included.
Section 4
7.6 The evaluation methodology takes into account the overall purpose of the evaluation, as well as the needs of the users and other stakeholders.
Section 2, 3 and 4
7.7 The evaluation methodology explicitly and clearly states the limitations of the chosen evaluation methods.
Section 3.5
7.8 The TOR specifies that the evaluation will follow UNEG norms and standards for evaluations, as well as ethical guidelines.
Section 1.3 Introduction
8. Evaluation Work Plan
8.0 The Terms of Reference includes a work plan
8.1 The TOR work plan states the outputs that will be delivered by the evaluation team, including information on the degree to which the evaluation report will be accessible to stakeholders, including the public.
Section 2.2.1, 2.2.4
8.2 The TOR work plan describes the key stages of the evaluation process and the project time line.
Section 2.2 and 4
8.3 The TOR work plan establishes clear roles and responsibilities for evaluation team Section 5
WHO Reform Stage 2 Evaluation - Final report Appendices
PwC 39
members, the commissioning organisation and other stakeholders in the evaluation process.
8.4 The TOR work plan describes the evaluation quality assurance process. Section 5.3
8.5 The TOR work plan describes the process, if any, for obtaining and incorporating evaluand comments on a draft evaluation report.
Section 2.2.4
8.6 The TOR work plan includes an evaluation project budget. N/A
9. Gender and Human Rights
9.0 The Terms of Reference specifies how a human rights and gender perspective will be incorporated in the evaluation design.
N/A
9.1 The TOR indicates both duty bearers and rights holders (particularly women and other groups subject to discrimination) as primary users of the evaluation and how they will be involved in the evaluation process.
N/A
9.2 The TOR spells out the relevant instruments or policies on human rights and gender equality that will guide the evaluation process.
N/A
9.3 The TOR includes an assessment of relevant human rights and gender equality aspects through the selection of the evaluation criteria and questions.
N/A
9.4 The TOR specifies an evaluation approach and data collection and analysis methods that are human rights based and gender sensitive and for evaluation data to be disaggregated by sex, ethnicity, age, disability, etc.
N/A
9.5 The TOR defines the level of expertise needed among the evaluation team on gender equality and human rights and their responsibilities in this regard and calls for a gender balanced and culturally diverse team that makes use of national/regional evaluation expertise.
Section 5.1
WHO Reform Stage 2 Evaluation - Final report Appendices
PwC 40
Appendix B- Addendum to Inception Report
With offices in Aarau, Basel, Berne, Chur, Geneva, Lausanne, Lugano, Lucerne, Neuchâtel, Sitten, St. Gallen, Thun, Winterthur, Zug and Zurich, PricewaterhouseCoopers SA is a provider of auditing services and tax, legal and business consultancy services. PricewaterhouseCoopers AG is a member of a global network of companies that are legally independent of one another; the network is represented in some 150 countries throughout the world.
PricewaterhouseCoopers SA Avenue Giuseppe-Motta 50 P.O Box 2895 1211 Geneva 2 Phone +41 58 792 91 00 Fax +41 58 792 91 10 www.pwc.ch
Office 7121 Bid Ref: IOS Stage two Attn: Mr David Webb, Director, Office of Internal Oversight Services World Health Organization 20, Avenue Appia CH-1211 Geneva 27 Switzerland Geneva, 21 August 2013 Dear Sir / Madam, We thank the IOS Secretariat for facilitating the teleconference with the Evaluation Management Group (EMG) that took place on 8 August 2013. The aim of the teleconference was to discuss EMG’s feedback on the Inception report submitted to the WHO on 8 July 2013 and to obtain EMG guidance on specific items including 1) list of stakeholders, 2) the survey and quantitative approach and 3) the timeline for the assignment. The purpose of this Addendum to the Inception report is to present refinement to our detailed approach to conducting the Stage 2 Evaluation of the WHO reform following feedback received from the EMG. We remain at your disposal for any further information or clarification on our inception report and look forward to hearing from you. Yours faithfully, PricewaterhouseCoopers SA
Gill Sivyer Antoine Berthaut
Engagement Partner - Geneva Engagement Director–Geneva
Undertaking regarding taking into account WHO’s specific context
WHO Reform Stage 2 Evaluation - Final report Appendices 42
1. Undertaking regarding taking into account WHO’s specific context
1.1. EMG Feedback
Following submission of the inception report on the 8 July 2013, the members of the EMG provided their
written feedback to the content of the report, which can be summarised under the main headings as follows:
How to leverage previous WHO evaluations to inform the stage two evaluation
What should be the extent of stakeholder consultation
How this evaluation will prepare the ground for future evaluations of the reform process
Given the complexity of the evaluation subject, is there sufficient capacity in the evaluation team to address all activities to the level demanded by the evaluation.
To respond to the above mentioned feedback, the below approaches were proposed by Independent Evaluation Team (IET) and discussed with the EMG during the teleconference on 8 August 2013.
While the above approaches were generally agreed to and validated by the EMG, the EMG members noted that
it would beneficial for the IET to contextualise the evaluation approach to the content of the WHO reform,
notably the evaluation questions, and to demonstrate how WHO’s current functioning’s will be impacted by the
reform initiatives rather than relying only on a generic transformation model. EMG members requested that
the IET emphasised what needed to be done to demonstrate that this reform would be more successful than
past initiatives.
4
Leveraging previous evaluations of the WHO to inform the Stage 2 evaluation (South Africa)
Assess the level of implementation of Stage 1 recommendations
Previous WHO evaluation
Feedback Evaluation suggestion
1
List of stakeholders is limited to high-level seniormanagement at HQ and regional level (South Africa)Considering both internal and external conditions thatare enablers of reform (JIU)
Broaden stakeholder consultation tocountry level (see next agenda item)
Extend stakeholder consultation toglobal health actors (see next agendaitem)
Extent of stakeholders consultation2
Making explicit how this evaluation will prepare theground for the next phase evaluation (outcome/impact) (JIU)
Include recommendations on the futuremonitoring and evaluation of the reformin evaluation report
Future evaluation of WHO reform3
Capacity put in place by PWC to carry out all activities with level of depth demanded by evaluation as opposed to a performance assessment model (JIU)
Revise timeline to address level of depththrough additional interviews
4 Capacity to cover depth
Undertaking regarding taking into account WHO’s specific context
WHO Reform Stage 2 Evaluation - Final report Appendices 43
Finally, EMG emphasised the need for the IET to address the position of WHO in a changing global health
architecture and evolving financial environment and to consider how external factors will be affecting the
success of the reform.
1.2. Independent Evaluation Team Undertaking
In response to the EMG, the Independent Evaluation Team (IET) first reiterates that in selecting the evaluation
questions in the Inception Report, the IET already ensured the relevance of these questions, although of a
generic nature, to WHO context. Building further on this conceptual framework, and to ensure the specific
context of WHO reform is taken into account, the IET have also complemented the set of questions with specific
ones outlined below.
The table summarises the IET undertaking to address the above mentioned points.
Domain of the evaluation framework
Specific factor Evaluation question How this will be taken into account in the evaluation
External to WHO
Strategy, structure
WHO position in the global health architecture
Has progress been made by WHO in clarifying its role in a changing global health architecture? How does reform takes this changing role into account?
Interviews with external key informants on global health
Literature review addressing WHO’s positioning and role in the global health architecture
Review of alignment of WHO’s vision with the 2014-15 Programme budget
Strategy Financial crisis – funding for Health and UN agencies
What is the impact of financial constraints on the implementation of the reform?
Funding inequalities across the three levels of the organisation will be assessed and the question of resources for reform will be addressed in interviews
Specific focus will be placed on financing and result-based management initiatives
Strategy UN Post 2015 development goals and UN-wide organisation
How is WHO positioning itself in the broader UN system and what leadership is it demonstrating in the definition of the post 2015 UN development goals?
Desk review of General Programme of Work
Interviews with Secretariat’s senior management
Interviews with external key informants
Structure Changing nature of world politics
How is WHO taking into account emerging global governance arrangements and the evolution of the balance of power between geographical areas?
Interviews with Member States, Secretariat and key informants
Internal to WHO
Change management, programme management
Past reform initiatives
Have learnings and recommendations issued by previous evaluations of WHO reforms in the 90s’ been factored into the definition of the current reform program and its implementation?
Review of past reforms and their outcomes based on literature review (Chatham House report, prior JIU reports)
Undertaking regarding taking into account WHO’s specific context
WHO Reform Stage 2 Evaluation - Final report Appendices 44
Domain of the evaluation framework
Specific factor Evaluation question How this will be taken into account in the evaluation
All domains Relevance of the evaluation questions and conceptual framework to WHO’s specific context
What is the content of each reform initiative and are those initiatives effectively helping to reshape the organisation?
Interviews with the Reform Team, Business Owners and regional level senior management to discuss the substance of the reform initiatives.
Desk review of relevant documentation discussed with interviewees in order to validate and support findings on the status and content of the reform
Structure, programme and change management
Specific WHO governance arrangement in regions
What is the level of alignment of governance arrangements and processes across regions?
Review of regional governing body documentation, reports and resolutions
Interviews with WHO Regional Offices and Chairs of Regional Committees
Complement and updates to the approach and instruments
WHO Reform Stage 2 Evaluation - Final report Appendices 45
2. Complement and updates to the approach and instruments
2.1. Engagement with a selection of Members States
2.1.1. Selection of Member States
The feedback from the EMG in relation to the selection of stakeholders was an absolute requirement of a
balanced representation of the various stakeholders’ groups to ensure credible and valid findings. However, it
was also recognized that the options available are limited due to time constraints for conducting the interviews
and consultations; and due to the practicalities of conducting this evaluation through the summer vacation
period.
Selection of Member States should be representative of all countries composing the World Health Assembly and
not necessarily of those who are more vocal or supporters of the reform process. The selection of Member States
should ensure representation of North/South, East/West, BRICS and equal representation of developing
countries. Random sampling of Member States is the recommended methodology for selection. The EMG
concluded that the IET should propose a list of potential informants for validation.
In order to realign with the above feedback, two (2) countries have been randomly selected for
each of the six (6) WHO regions, with a total of 12 countries selected.
The selection aimed for balance in the following criteria:
Income level based on World Bank country classification1
BRICs vs non BRICS
Status of renewal of the Country Cooperation Strategy2
Fragile countries based on World Bank classification3
The selection excluded countries which are solicited in other ways throughout the evaluation:
Evaluation Management Group countries (i.e. Australia, Azerbaijan Islamic Republic of Iran, Myanmar,
Panama, South Africa);
Member States from the Geneva group that will be convened through a focus group discussion by the IET;
The country affiliation of Chairs of Regional Committees;
Countries the IET interviewed during the Mobilise phase of the evaluation (United States, United Kingdom,
Ireland, Brazil, Norway, Canada).
Based on the above, the random selection resulted in the following list of countries presented in the next page.
1 http://data.worldbank.org/about/country-classifications/country-and-lending-groups#Low_income 2 http://www.who.int/countryfocus/cooperation_strategy/listofccs/en/ and WHO country offices websites 3 http://siteresources.worldbank.org/EXTLICUS/Resources/511777-1269623894864/HarmonizedlistoffragilestatesFY14.pdf
Complement and updates to the approach and instruments
WHO Reform Stage 2 Evaluation - Final report Appendices 46
Table 1. List of Member States with WHO representation to be consulted
Regions Country selection
AFRO Kenya
Cameroon
AMRO Argentina
Mexico
EMRO Morocco
Egypt
EURO Russia
Turkey
SEARO India
Thailand
WPRO China
Solomon Islands
Changes suggested by the EMG are highlighted in bold.
For each of the 12 countries, the IET will endeavour to interview country representatives from the following
organisations:
Ministry of Health (priority 1)
Ministry of Foreign affairs (priority 2)
Geneva mission (priority 3)
The IET relies on the IOS to provide contacts and facilitate introductions.
In addition to the above selection of Member States, interviews will be conducted with:
the current Chairs of all WHO Regional Committees and the country affiliations to the following Member
States are presented as follows:
o Regional Committee AFRO: Dr José Vieira Van-Dúnem, Angola
o Regional Committee EURO: Dr Joseph R. Cassar, Malta
o Directing Council PAHO: Dr. Franklin Vergara Jaén, Panama
o Regional Committee EMRO: H.E. Mr Bahar Idris Abu Garda, Sudan
o Regional Committee SEARO: H.E. Dr Nafsiah Mboi, Minister for Health, Indonesia
o Regional Committee WPRO: Dr Nguyen Thi Kim Tien, Vietnam
all Member States forming the Geneva Group will be convened to attend a joint meeting organised and
facilitated by the WHO. The Geneva Group is composed of the following Member States: Australia,
Belgium, Canada, France, Germany, Japan, Italy, Mexico, Netherlands, Russia, Republic of Korea, Spain,
Sweden, Switzerland, United Kingdom, United States
The focus of the interviews with Member States will be to address the reform priorities, communication on
reform, governance arrangements and impact of the reform in their countries.
Complement and updates to the approach and instruments
WHO Reform Stage 2 Evaluation - Final report Appendices 47
2.2. Selection of non-state actors
For the purpose of this evaluation, the group of non-state actors is broken down into three main categories:
NGOs in official relation with the WHO
Private donors
Key informants on global health
2.2.1. NGOs in official relation with the WHO
Of the 194 NGOs in official relation with the WHO, six (6) NGOs have been randomly selected with the view to
ensure representation from:
Implementing NGO
Professional association
Disease or illness focused organisation
Pharmaceutical focused organisation
The resulting selection resulted in the following NGOs being sampled:
African Medical and Research Foundation
International Federation of Pharmaceutical Manufacturers and Associations
International Council of Nurses
World Medical Association
Medicus Mundi International – International Organisation for Cooperation in Health Care
OXFAM
Changes suggested by the EMG are highlighted in bold.
The focus of the interviews will be to understand NGOs perception of the WHO reform, in particular with
respect to WHO’s position in the global health architecture, its role as a convener on global health challenges
and its relationship with civil society actors.
2.2.2. Private donors
Two (2) private donor representatives have been selected on the basis of their financial support to WHO and
their recent involvement in the Financing Dialogue, June 2013.
The selection includes representatives of the following philanthropies:
Bill and Melinda Gates Foundation
Wellcome Trust
Changes suggested by the EMG are highlighted in bold.
The focus of the interviews will be to understand the private sector’s perception of WHO and its relevance in the
changing global health architecture while also aiming to assess the efforts deployed at creating a space for
dialogue between WHO and the private sector.
2.2.3. Key informants on global health
The evaluation will consider the voices of five (5) external informants on global health, from academia and
government alike, some of whom have been involved in the reform directly. These will include in particular:
Complement and updates to the approach and instruments
WHO Reform Stage 2 Evaluation - Final report Appendices 48
Professor Thomas Zeltner, former Special Envoy on WHO Financing Dialogue and current Special Envoy on Engagement with non-state actors
Dr. Sigrun Mogedal, former AIDS Ambassador of Norway
Mr Bezancon, Advisor on FAO reform and author of a 2005 report on The Global Fund’s replenishment
Dr. Ilona Kickbusch, Global Health Diplomacy, Graduate Institute of International and Development Studies
Charles Clift, Chatham House
The focus of the interviews will be to obtain their views on the positioning of the WHO in the evolving global
health architecture and the role it should play in the post-2015 development agenda.
2.3. Engagement with a selection of WHO Representatives at country level
The feedback from the EMG in relation to Member States was that in order to avoid bias, the selection of WHO
representatives (WRs) should be performed randomly, and should not rely on interviewee suggestions.
To that effect, the IET suggests that the selection of WRs reflect the random selection of 12 Member States
selected randomly above (see Table 1.). This will also allow triangulating results from interviews with WRs with
those of interviews with Member States.
The focus of the interviews will be on the regional and country level implementation of the reform agenda and
on the level of engagement of the WRs in the process.
2.4. Engagement with a random selection of WHO staff
Feedback from the EMG emphasised that in order to avoid selection bias and to capture internal staff
perceptions as to how far the reform message has penetrated the organisation, the methodology for selecting
internal WHO staff should use a random selection of staff identified from the phone directory.
Six (6) WHO staff will be randomly selected across all three levels of the organisation, and identified through
the phone directory as mentioned above.
The focus of the interviews will be to better understand how the reform is directly impacting staff in their day-
to-day work.
Quantitative Survey approach
WHO Reform Stage 2 Evaluation - Final report Appendices 49
3. Quantitative Survey approach
3.1. EMG feedback
The EMG members signalled that the current questionnaire, as designed, did not capture the necessary information from staff required to answer the question “How reform is translating at their level and what impact it will have on their work”. Following discussions on the purpose and need for the questionnaire the participants agreed on the need to redesign the questionnaire for an all staff survey. The revised questionnaire would:
a) focus on the various domains of the reform; b) include more personal details on the respondent, such as length of service with WHO; c) assess the extent to which the reform changed what the respondent do/working approach in the last 12
months; d) explore the respondent’s perception on how the reform will change WHO in the future.
It was agreed that the IET would re-submit the re-designed questionnaire to the EMG for final clearance before implementation.
3.2. Revised Objectives for quantitative survey
In line with the request of the EMG, the survey questionnaire has been reworked to address the above mentioned objectives. An updated version is presented in Appendix A for EMG feedback. The survey contains four sections. Section 1 collects information on the respondents. Section2 addresses how communication on reform is perceived by respondents. It seeks to understand how staff receives information on reform and how efficient and relevant the existing communication channels are in the eyes of the respondent. Section 3 addresses the respondent engagement with the reform process. It aims at providing a temperature gauge on the engagement of staff in the reform process. Questions seek to assess their level of engagement, from understanding the objectives of the reform, to concretely inputting into reform initiatives and being supportive of the reform process. Section 4 seeks to quantify respondent perceptions of WHO work practices in those areas of reform that the IET deemed to impact staff and work practices the most based on initial interviews with Business Owners:
Overall perception of changes in working practices
Alignment, synergy and collaboration across the organisation
Engagement with other stakeholders
Human Resources
Programmatic priorities
Finances
Accountability, transparency and risk management
Knowledge management
Note that no question will be asked on the topic of “Evaluation” given that a detailed survey has been performed in July 2013 by the Secretariat the results of which will be reviewed by the IET once made available. This section also seeks to understand if staff believes that the WHO reform will have a meaningful impact on improving work practices in the above mentioned areas.
Quantitative Survey approach
WHO Reform Stage 2 Evaluation - Final report Appendices 50
3.3. Addressing challenges encountered in constructing the quantitative survey
An early finding of the investigation to date relates to the weakness of the result-chain or theory of change of the reform. In the absence of a clear articulation of the exact and ultimate impact each reform initiative is aiming to have on work practices, building a survey to assess whether these changes are happening is not trivial. In this context, the IET had to:
Identify which areas of reform or work practices are most likely to impact staff, that staff apply or can observe;
Understand through desk review and interviews with Business Owners the goal and objectives of these initiatives;
Identify some of the missing indicators that would bring evidence of change and success of these reform initiatives;
Check if these indicators have been measured in previous surveys, in which case the same questions could be reused to see if progress has been achieved;
Where no previous question can measure the change in work practices, create new questions to provide an initial baseline.
Asking staff what specific reform related improvements they are observing makes an invalid assumption that staff understand the exact content of each reform initiative and can link these initiatives to change in their daily work. Staff cannot and do not necessarily need to be expected to make the difference between changes or improvements in work practices that are specifically linked to reform initiatives from changes or improvements that may be due to initiatives not related to reform. In this context the IET team took the view that it is better to seek to measure with absolute certainty if changes in work practices are happening than to seek to determine if these changes are unequivocally linked to the reform programme. The premise is that if changes are not seen to be happening on the ground, then the logical conclusion is that reform has yet to address the barriers that need to be addressed and implement the scope of required activities to effect change. The IET also sought to complement the approach by prompting staff on whether they believe that work underway as part of reform will have an impact on these work areas.
3.4. Note on re-use of some of the 2012 JIU survey questions
In reviewing the JIU ‘Review of Management, administration and decentralization in the WHO’ reports, the IET noted that the JIU submitted a survey to staff mid 2012 which contained a number of questions on work practices in the various areas of reform. These can be used as proxies to see if changes are being observed. In building section 4 of the evaluation survey, we leveraged a number of these previously used questions. These are identified by an asterisk (*) in the survey. By comparing the results with the results of the mid-2012 JIU, the IET will be able to identify if respondent’s perception in the various areas of reform have changed in the last year. In cases where no relevant JIU questions were available, e.g. in the area of engagement with non-state actors or risk management, the IET created new questions. These will provide an initial baseline and allow to gauge staff’s perception of WHO’s current performance in these areas.
3.5. Organisation of survey
Timing
The survey will span over a period of three (3) weeks from 2 September to 23 September 2013.
Communication
Quantitative Survey approach
WHO Reform Stage 2 Evaluation - Final report Appendices 51
Communication on the survey will be done through four (4) avenues and supported by the IOS Secretariat and
the WHO Communications Department:
1. The survey will be sent to all staff through a formal initial email communication by the IOS Secretariat
on 2 September 2013 informing them of the purpose of the evaluation and objectives of the survey;
2. A message will be posted on the WHO intranet site on the survey and Stage 2 Evaluation, supported by
the Communications Department;
3. IOS will request Assistant Director Generals and Regional Directors to send a communication to their
staff encouraging them to take the survey;
4. Two reminders will be sent by email to all WHO staff on 9 and 16 September 2013.
Platform
The survey will be administered online. The survey will be anonymous and access to detailed answers and
results will be limited to the IET.
Updated Project Management Plan
WHO Reform Stage 2 Evaluation - Final report Appendices 52
4. Updated Project Management Plan
As described in the Gantt chart, the table below sets out the expected deliverables, milestones and feedback
expected by the EMG.
Deliverables Deadlines Inception report 8 July 2013 EMG feedback on inception report 19 July 2013 Preliminary report 14 October 2013 Feedback on preliminary report 29 October 2013 Final Evaluation report 11 November 2013 Summary PowerPoint presentation 20 December 2013 Presentation to 134th session of the Executive Board 17 January 2013
Below is a detail of the EMG check points throughout the assignment:
9 July: Receive inception report
19 July: Provide feedback on inception report
08 August: EMG call on inception report
21 August: Receive survey and list of key stakeholders
28 August: Provide feedback on survey and list of key stakeholders
16 September: Receive key findings in draft form
19 September: EMG debriefing call (key findings)
03 October: EMG debriefing call (TBD)
14 October: Receive preliminary report
29 October: Provide feedback on preliminary report
4 November: EMG call on preliminary report
11 November: Receive final report
20 December: Receive final presentation
The Gantt chart below presents the updated detailed workplan that will guide the execution of the Evaluation
work.
Updated Project Management Plan
WHO Reform Stage 2 Evaluation - Final report Appendices 53
T ask Ju neP lanning
( # o f
w e e ks ) 2 4 .0 6 1.0 7 8 .0 7 15 .0 7 2 2 .0 7 2 9 .0 7 5 .0 8 12 .0 8 19 .0 8 2 6 .0 8 2 .0 9 9 .0 9 16 .0 9 2 3 .0 9 3 0 .0 9 7 .10 14 .10 2 1.10 2 8 .10 4 .11 11.11 18 .11 2 5 .11 2 .12 9 .12 16 .12 2 3 .12 3 0 .12 6 .0 1 13 .0 1 2 0 .0 1
Mobilise 6 we e k s
Collect in for m a tion 2 weeks
A ssess WHO r efor m sta tu s 2 weeks
Pr odu ce in cept ion r epor t (IR) 2 weeks
Su bm it in cept ion r epor t
EMG r ev iew of in cept ion r epor t 2 weeks
Receiv e EMG feedba ck on in cept ion r epor t
Hold EMG ca ll on in cept ion r epor t
Evaluate 12 we e k s
Con du ct key sta keh older s in ter v iew s
HQ 9 weeks
Reg ion s 7 weeks
Cou n tr ies 6 weeks
Mem ber Sta tes 10 weeks
Con du ct desk r ev iew 7 weeks
Con du ct su r v ey 7 weeks
Synthesize & Report 2 2 we e k s
Con du ct da ta a n a ly sis 8 weeks
Dr a ft Pr elim in a r y r epor t (PR) 5 weeks
Su bm it pr elim in a r y r epor t
EMG r ev iew of pr elim in a r y r epor t 2 weeks
Receiv e EMG feedba ck on pr elim in a r y r epor t
Hold EMG ca ll on pr elim in a r y r epor t
In cor por a te EMG feedba ck 2 weeks
Su bm it fin a l r epor t
Socia lise fin din g s 6 weeks
Pr epa r e fin a l pow er poin t pr esen ta t ion 2 weeks
Su bm it fin a l pow er poin t pr esen ta t ion
Pr esen t a t Ex ecu tiv e Boa r d
Key milestones
Ja n u a ry 2014Ju ly A u gu st Sept em ber Oct ober Nov em ber Decem ber
Receive EMG
feedback19 July
2013
Submit Inception
report
8 July
2013
Submit final presen-
tation20 Dec
2013
Present at Executive
Board17 Jan 2014
Hold EMG call
Week 8 August
2013
Submit final
report11 Nov
2013
Hold EMG
call 4 Nov
2013
Submit preliminary
report14 Oct
2013
ReceiveEMG
feedback29 Oct
2013
Roles and responsibilities
WHO Reform Stage 2 Evaluation - Final report Appendices 54
5. Roles and responsibilities
5.1. Update to description and role of WHO
The EMG raised some concern regarding the governance arrangements for the oversight of the evaluation
process. In particular, the EMG sought clarification on its role in relation to the one of the “Internal Steering
Committee”. The feedback of the EMG on the role of the Steering Committee was that the title of this body is
misleading because it is not its role to provide guidance and advisory support on the evaluation.
The IOS Secretariat confirmed that the internal committee was formed in response to the previous EMG’s
specific request for a WHO representation to institutionalise the evaluation; as well as the IET’s request for a
“sounding board” to facilitate internal communication within WHO so that it could benefit from the experience
of the external consultants in conducting such an evaluation. It was agreed that the internal Steering Committee
will be renamed to better reflect its purpose and avoid the risk of confusion with the EMG’s role. The table
below presents the updated name and composition of the ‘WHO Stage 2 Evaluation Committee’.
WHO role Nature of involvement Estimated workload
WHO Stage 2 Evaluation Committee Hans Troedsson, Executive Director, Office of the Director-General Ian Smith, Reform Team David Webb, IOS Mohamed Jama, Assistant Director General, General Management 6 Regional Nominees
Follow-up on the execution of the evaluation and act as a sounding board
Guide and provide feedback on data collection and tools
Consider initial findings and facilitate the socialisation of findings
Ensure transfer of knowledge of the evaluation work within the WHO
Three planned teleconferences:
Review execution phase plans
Consider preliminary report key findings
Consider recommendations and lessons learned (final report)
Other roles and responsibilities remain unchanged.
WHO Reform Stage 2 Evaluation - Final report Appendices 55
Appendix D. - Data collection-Updated Staff survey questionnaire for feedback from EMG
WHO Reform survey
The Executive Board at its special session in November 2011 requested a two stage independent evaluation of
the WHO reform. The modalities for Stage 2 of the evaluation were endorsed by the Executive Board at its 132nd
session in January 2013 to assess the readiness of the WHO to fully implement the reform agenda and to
identify the necessary conditions for success.
As part of this work, the Independent Evaluation Team from PwC is interested in knowing more about what
staff’s involvement in the reform and what effect it is having on your day to day work.
The survey should only take about 15 minutes of your time and your answers will be completely anonymous.
The findings of the evaluation, including the results of this survey, will be presented at the PBAC session prior
to the 134th Executive Board session in January 2014.
We encourage you to participate by answering in your own capacity and sharing with us your personal
perception.
We thank you in advance for your collaboration to ensure this process is successful and beneficial to all.
The Independent Evaluation Team
WHO Reform Stage 2 Evaluation - Final report Appendices 56
1. About you This survey is anonymous; however we would appreciate if you could provide answers on your employment at WHO. This will allow us to perform a more meaningful analysis. 1.1 My post is located at:
Region of employment Tick the corresponding box(es)
HQ
AMRO/PAHO Region
AFRO Region
EURO Region
EMRO Region
WPRO Region
SEARO Region
Regional office
Country office
Other
If other, please specify
1.2 My duty station is:
Duty station Tick the corresponding box
Pick from list of all duty WHO stations
1.3 I have been working at WHO for:
Length of service Tick the corresponding box
0-2 years
3-5 years
6-10 years
More than 10 years
1.4 I am hired as:
Contract type Tick the corresponding box
International Professional staff
National Professional officer
General Service staff
Other
1.5 My appointment is:
Contract type Tick the corresponding box
Continuing
Fixed term
Temporary
1.6 My grade/category is:
Grade Tick the corresponding box
D1-D2
P5-P6
P3-P4
P1-P2
G
National Professional officer
Other
1.7 My age group is
Age group Tick the corresponding box
WHO Reform Stage 2 Evaluation - Final report Appendices 57
18-29
30-39
40-49
50-62
Older than 62
National Professional officer
Other
1.8 My gender is:
Gender Tick the corresponding box
Female
Male
1.9 The area I primarily work in is (alphabetical order):
Work area Tick the corresponding box
Communicable diseases
Corporate services/enabling functions
Health systems
Non-communicable Diseases
Preparedness, surveillance and response
Promoting health through the life‐course
2. Communication on the WHO reform This section seeks to understand how you are informed on the WHO reform. 2.1 For each of the statements below please tick the answer that best corresponds to your views:
Statement Don’t
know/ no
opinion
Strongly
disagree Disagree
Neither
agree or
disagree
Agree Strongly
agree
I am aware of the WHO reform
I have been well informed by management of the
WHO reform process*
I believe that management is communicating
adequately with staff about important staff issues
relating to reform (e.g. HR policies)
I believe communication on reform is timely
I believe communication on reform is efficient
I believe communication on reform is adapted to my
needs
There are adequate ways for me to inquire about the
WHO reform programme
WHO Reform Stage 2 Evaluation - Final report Appendices 58
2.2 I am primarily informed about the WHO reform through (ranked alphabetically):
Channels Tick the corresponding box, more
than one choice possible Available Governing Body documentation
Informally by other colleagues
Updates in management meetings
My supervisor
Participation in reform related projects or task forces
The Change@WHO Newsletter
The Staff Association
Townhall presentations
WHO Intranet and internet pages on Reform
Other
If other, please precise
2.3 I would prefer receiving information on WHO reform through:
Please select up to 3 sources by order of importance
Preferred Channels 1st
favourite
2nd
favourite
3rd
favourite
Available Governing Body documentation
Informally by other colleagues
Updates in management meetings
My supervisor
Participation in reform related projects or task forces
The Change@WHO Newsletter
The Staff Association
Townhall presentations
WHO Intranet and internet pages on Reform
Other
If other, please precise
3. Your engagement in the WHO reform In this section, we seek to understand your involvement and perception of the WHO reform 3.1 For each of the statements below please tick the answer that best corresponds to your views:
Statement Don’t
know/ no
opinion
Strongly
disagree Disagree
Neither
agree or
disagree
Agree Strongly
agree
I believe I understand the vision, objectives and
planned outcomes of the WHO reform
I believe the WHO reform will be important to make
the organisation more ‘fit for purpose’ in the future
I believe the WHO reform will have a positive impact
on health outcomes and national health systems
I understand what is expected from me to make the
WHO reform successful
I believe I can personally contribute to the
achievement of the WHO reform objectives
I have participated in taskforces or working groups
specifically set-up in my
Cluster/Department/Regional office/Country office
to address the WHO reform
WHO Reform Stage 2 Evaluation - Final report Appendices 59
4. Evolution of work practices This section lists a number of work areas related to reform and seek to understand how reform is impacting your work practices. Note: statements with an asterisk (*) indicate questions you may have previously been asked in the JIU 2012 survey as part of the ‘Review of Management, administration and decentralization in the WHO’. We are asking these questions again to identify if any change has occurred in the last year. 4.1 For each of the statements below please tick the answer that best corresponds to your views:
Statement
Don’t
know/ no
opinion
Strongly
disagree Disagree
Neither
agree or
disagree
Agree Strongly
agree
Overall change
In the last 12 months, I have seen changes in my day
to day work linked to the WHO reform
Alignment, synergy and collaboration across
the organisation
I have a clear perspective on the division of
responsibilities in my area of work between HQ, the
regional offices, and the country offices*
I think there is sufficient coordination and
cooperation within clusters/divisions/units within
the following entities:*
WHO Headquarters
Regional offices
Country offices
I think the level of coordination and cooperation
between the following WHO entities is adequate*:
WHO Headquarters and regional offices
Regional offices and country office
WHO Headquarters and country offices
There is adequate alignment between WHO’s country
presence and country needs and priorities
Regional and country offices are adequately staffed to
play their role with Member States
I think the work underway on reform will improve
the effectiveness of the organisation*
Engagement with other stakeholders
I think the WHO will be better at convening health stakeholders after the reform
Guidance on engaging with non-state actors to help improve the public health outcomes of my programme is readily available
I think the role of WHO, among other actors working
on global health, in my programme area is clearly
defined
I think WHO has the available capacities to coordinate, convene and direct the various actors in global health
I think WHO’s role in global health governance will be strengthened through the reform process
Human Resources
I think the recruitment process is fair and
transparent in WHO*
I think I will move to another duty station in the next
3 years
Training received is adapted to my career
development needs*
My training is linked to my individual performance
appraisal system in line with WHO priorities*
I think the system of administration of justice in
WHO Reform Stage 2 Evaluation - Final report Appendices 60
WHO ensures an adequate and fair treatment of staff
complaints*
I think the work underway on reform will improve
the human resource capacity of the organisation to
deliver against its mandate
Programmatic priorities
WHO sets out clear priorities for its work
I am familiar with the concept of RBM and its
implementation in WHO*
I think WHO activities are measured with clearly
defined performance indicators*
I think the WHO reform will improve the focus of the
organisation in my programme area.
Finances
The allocation of funds across WHO is transparent
I am seeing more alignment between WHO priorities
and how resources are allocated across the
organisation
I conduct resource mobilisation activities (if any) as
part of a coordinated whole of WHO approach
There is an adequate level of coordination and
cooperation in resource mobilisation activities across
the different levels of the organisation
My unit receives necessary resources in order to meet
its goals*
I think the WHO reform will enhance the ability of
my programme area to have more predictable and
sustainable financing
Accountability, transparency and risk
management
I think managers are held accountable for the
authority delegated to them*
I have a clear idea of what are my tasks and
responsibilities and how I will be held accountable
for those*
Our policies and procedures for managing conflict of
interest are effective
The key risks in the unit I work for have been
identified and documented
My unit has defined mechanisms to mitigate these
major risks facing the achievement of my units
objectives
My superior actively supports and drives the use of
risk management in our work
I think work underway as part of the reform on
accountability, transparency and risk management
will build Member State’s trust in WHO
Knowledge management
Information-sharing in my unit happens in support
to my work*
The aims and scope of knowledge management in
WHO are clear to me*
I can regularly benefit from knowledge management
in my area of work*
4.2 In your view, what are top three challenges and barriers that need to be addressed for the reform to be a success? Do you have any suggestions on how to address this? (open question)
Free text form
WHO Reform Stage 2 Evaluation - Final report Appendices 61
Appendix C- Evaluation questions
The Terms of Reference for the ‘Independent evaluation of the WHO reform: stage two’ proposes a set of
questions to be addressed by the evaluation, and which are set out in the table below. These are linked to the
key issues the evaluation should address, as approved by the EB during its 132nd session4 (EB 132/5 Add 7). The
IET has supplemented those questions with a number of more detailed related questions, as set out in this
Appendix, which have guided the evaluation and interview process. These are set-out according to the two key
issues agreed to by the EB, (i) the status of action taken on the recommendations of stage one and (ii) the
modalities of implementing the reform proposal and the sufficiency of the change management strategy.
The proposed questions are presented in bold, while the detailed evaluation questions are presented in italic.
(i) Status of action taken on the recommendations of stage one.
(a)What is the status of implementation of the recommendation relating to the reform process compared with the expected (planned) progress and achievements?
(b)Have there been unintended effects in the reform process and how have these impacted the overall result of WHO reform?
(c)What factors have been identified (from progress to date) that will enhance moving forward with the reform effectively?
(ii) Modalities of implementing the reform proposal and the sufficiency of the change management strategy including:
(d) To which degree is there coherence in the governance mechanisms to inform, monitor and evaluate the implementation of the WHO reform package? To which extent are these mechanisms and processes adequate to prioritize the various components of the reform proposal?
Has there been a process to evaluate and prioritise areas for improvement? How was the prioritisation conducted? How were the criteria for prioritisation defined?
Inception report
Have governance bodies been set-up to advise/act as sounding board on this particular area? What has been the representation of these governance bodies?
Inception report
What has been the process for approving this roadmap? Inception report
How are the expectations of WHO Secretariat and Member States managed? Inception report
How are results reported and how is the quality of the deliverables monitored? Inception report
Has a realistic business case been developed subjected to an appropriate level of challenge?
Inception report
Has an agreed quality plan been developed based on appropriate standards to monitor and evaluate the quality of the work performed?
Inception report
4 WHO, Modalities for the independent evaluation of the WHO reform: stage two, Report by the Director-General, EB132/5 Add.7, 11 January 2013.
WHO Reform Stage 2 Evaluation - Final report Appendices 62
What are the governance arrangements supporting the reform agenda? Inception report
How are leadership activities executed and monitored? Inception report
Does the WHO promote a culture conducive to reform? Inception report
Has a change readiness assessment been carried out? Inception report
Is the communication delivered adequate and containing the right type of messages for different stakeholders of the WHO reform?
Inception report
Are stakeholders identified, mapped and is there a stakeholder management plan? Inception report
Has a communication and engagement plan defined? How is it executed and reported against?
Inception report
Has a change agent network been identified? Are change agent network activities taking place on an on-going basis?
Inception report
(e)What are the specific strategies to address change management? How well are the pertinent issues addressed?
Who has been involved in developing the implementation plan? Inception report
How have relevant stakeholders been engaged in the process? Inception report
How is the knowledge transferred and how is the hand-over performed from reform to business as usual?
Inception report
Have stakeholders been identified, assessed and are they managed on an on-going basis throughout the reform process to ensure high level of commitment?
Inception report
Are they appropriately informed to contribute to the success of the programme? Inception report
Has the nature of stakeholders been considered, their roles and relationship to one another defined?
Inception report
What change control arrangements are in place to manage changes to scope, requirements, deliverables and intended benefits that impact the project baseline, time or cost?
Inception report
Does a change vision exist? Inception report
Is there agreement on the objectives of the reform? Inception report
Is the case for reform actively pushed by leaders? Inception report
How are benefits of the reform demonstrated? Inception report
How is the informal and formal reward process managed? Inception report
What are the mechanisms in place to coach champions and managers? Inception report
(f)Have the significant barriers, challenges, systematic constraints, and risks for the reform been identified (including engagement of staff in the process)? How relevant and adequate are the proposed approaches for dealing with these barriers, challenges, systematic constraints,
WHO Reform Stage 2 Evaluation - Final report Appendices 63
and risks in relation to achieving the outcomes expected from the reform process? How efficient are the approaches to address them to ensure sustainability?
Are effective risk identification processes in place and are the key risks mitigated and opportunities taken?
Inception report
Have learnings from pilots (initial implementations) been applied to full implementation?
Inception report
Has progress been made by WHO in clarifying its role in a changing global health architecture? How does reform takes this changing role into account?
Addendum
Have learnings and recommendations issued by previous evaluations of WHO reforms in the 90s’ been factored into the definition of the current reform program and its implementation?
Addendum
(g) What is the efficacy of internal and external instruments and modalities to implement the reform proposals, including related risk management plans and risk mitigation actions?
How are performance issues addressed? Inception report
Has a clear assurance plan been defined which outlines the nature, timing and extent of planned assurance, quality reviews and embeds learning?
Inception report
Are personal objectives aligned? Inception report
How does the Reform Team respond to findings? Inception report
What is the level of alignment of governance arrangements and processes across regions?
Addendum
(h)Are indicators and timelines sufficient, feasible and appropriate to effectively measure (monitor and evaluate) performance and report on the desired outcomes of the reform process?
Have has a roadmap been produced, defining broad objectives, timeline, responsibilities, KPIs, budget and reporting structure?
Inception report
Has a detailed implementation plan been developed that integrates new ways of working?
Inception report
Which of the activities part of the implementation plan have started? Inception report
Has a post-implementation review been completed? Inception report
Is the scope of the programme well defined, complete, communicated and agreed? Inception report
Does the scope support the objectives of the WHO High-level implementation strategy? Inception report
Are the plans in place realistic, achievable, understood and bought into by key stakeholders executing the reform and those affected by it?
Inception report
How are outputs reported and is reporting targeting all stakeholders to the reform? Inception report
Does management regularly report on performance? Inception report
How is the success of the reform defined? Inception report
How is it assessed? Inception report
WHO Reform Stage 2 Evaluation - Final report Appendices 64
How were KPIs defined? Inception report
What is the process and mechanism in place for monitoring and reporting? Inception report
(i) To what degree have appropriate strategies, including resource mobilization and allocation, and corporate strategies such as human resources management and gender mainstreaming, been identified to implement the reform proposals?
What are for you the strategic drivers of the reform? What has been the process to identify and agree on the major challenges?
Inception report
Was the current state of affairs analysed and the baseline created? Inception report
Has a strategy/concept note been developed to address this area of challenge? Inception report
Has an approach been designed to address the area for improvement? Has a proposal been developed?
Inception report
Is the programme team highly motivated with the right blend of skills and personalities? Inception report
Is the budget and associated policies, processes and reporting standards adequate for effective cost estimation and programme financial management?
Inception report
What are WHO’s core competencies? Inception report
What are the processes for formulating strategy? What are the processes for measuring and managing the execution of strategy?
Inception report
Does the current structure of the client support the overall strategy? Inception report
Has the strategic plan been translated into coherent sub-strategies for each of the 12+1 initiatives under review?
Inception report
How well are the sub-strategies for IT, operations, finance, and risk management aligned to the overall strategy?
Inception report
What is the impact of financial constraints on the implementation of the reform? Addendum
How is WHO positioning itself in the broader UN system and what leadership is it demonstrating in the definition of the post 2015 UN development goals?
Addendum
How is WHO taking into account emerging global governance arrangements and the evolution of the balance of power between geographical areas?
Addendum
What is the content of each reform initiative and are those initiatives effectively helping to reshape the organisation?
Addendum
(j)Are the necessary conditions present to ensure an appropriate level of organizational preparedness (institutional framework: structures, roles, instruments, procedures and guidelines, support systems for change process, communication, incentives)
Has a specific body/team been set-up to manage and oversee the implementation? Inception report
Has an organisational impact assessment been carried out for the reform overall and for reform initiatives specifically?
Inception report
Is there clarity on the working practices, process documentation, organisational design and clear policies to ensure WHO staff understand the purpose of the reform and their role?
Inception report
Are systems, tools and guidance adapted to support the reform? Inception report
WHO Reform Stage 2 Evaluation - Final report Appendices 65
Are trainings set-up to ensure acquisition and transfer of knowledge and new skills? Inception report
Are processes, policies and rules are being changed where necessary to support the reform?
Inception report
Does the strategy call for centralised control and commonality of process? Inception report
Have factors relating to internal governance, reporting lines between governing bodies, accountabilities, organizational structure, geographical footprint and relationships with external stakeholders been adequately considered?
Inception report
Which processes could be candidates for outsourcing? If processes are currently outsourced, what has been the WHO's experience?
Inception report
Are processes consistent and standard across the organisation, at the HQ, regional and country level?
Inception report
What does the process documentation tell us about WHO’s model? Inception report
Is there a clear end-to-end process view? Inception report
Is process performance measured? Inception report
What processes are internal and which have been outsourced? Inception report
Have internal controls and standardization of processes and data been considered? Inception report
Are tools and guidelines readily available and incorporated into the day to day work of WHO staff for greater transparency?
Inception report
Are governance mechanisms and accountability documented and clear? Does the WHO have a mechanism for whistleblowing?
Inception report
How do decisions get made? What is the span of control and reporting? Where within the WHO are the power brokers that influence change?
Inception report
Do leaders own change efforts, or are they delegated to others? Inception report
What is the DNA of the culture? Is it consistent across the WHO? Inception report
Are organisational structures, culture, and individual behaviours aligned with and supportive of strategy?
Inception report
Are the skill-sets of staff adequate? Inception report
Has a performance management system and incentives been considered? Inception report
Have agents of change within the organisation been identified? Inception report
What processes are in place to connect the IT function to the organisation to support the reform?
Inception report
Does the IT strategy appear to be aligned with the implementation plan? Is technology appropriately funded in the context of the reform?
Inception report
Has technology such as data management and IT infrastructure, in order to support the direction of change?
Inception report
WHO Reform Stage 2 Evaluation - Final report Appendices 66
Have all five dimensions been considered in the reform implementation strategy? Inception report
WHO Reform Stage 2 Evaluation - Final report Appendices 67
Appendix D- List of interviews
Name Position Stakeholder category
Al Shorbaji, Najeeb Director, HQ/KMS Knowledge Management and Sharing WHO HQ
Alwan, Ala Regional Director, EMRO WHO Regional Office
Andrus, Jon Kim Deputy Director, AMRO/PAHO WHO Regional Office
Arnold, Dianne DAF, SEARO WHO Regional Office
Asamoa-Baah, Anarfi Deputy Director-General, HQ/ODG Office of the Director-General
WHO HQ
Aylward, Bruce Assistant Director-General - Polio, Emergencies and Country Collaboration
WHO HQ
Balladelli, Pier Paolo WHO Representative, Argentina WHO Country Office
Bekedam, Hendrik Jan WHO Representative, Egypt WHO Country Office
Belghiti Alaoui, Abdelali Secretary of Health, Ministry of Health, Morocco Member States
Ben Yahmed, Samir DPM, EMRO WHO Regional Office
Benton, David Chief Executive Officer, International Council of Nurses NGO
Birmingham, Maureen WHO Representative, Mexico WHO Country Office
Branca, Francesco Director, Department of Nutrition for Health and Development (NHD)
WHO HQ
Burci, Gian Luca Director, Legal Counsel, HQ/LEG Office of the Legal Counsel WHO HQ
Cassels, Andrew Director (Strategy), HQ/ODG Office of the Director-General WHO HQ
Chan, Margaret Director General WHO HQ
Chestnov, Oleg Assistant Director-General - Family, Women's and Children's Health
WHO HQ
Deynout, Chantal Management Officer, NMH WHO HQ
Emiroglu, Nedret Deputy Director, Communicable diseases, health security, EURO
WHO Regional Office
Ephrem, Bersabel Senior Adviser HQ/GMA GMG ADGO Office of the Assistant DG
WHO HQ
Escorel De Moraes, Maria Luisa
Minister Counsellor, Permanent Mission of Brazil Member States
Etienne, Carissa Regional Director, AMRO/PAHO WHO Regional Office
Feig, Christy Director, HQ/DCO Department of Communications
WHO HQ
WHO Reform Stage 2 Evaluation - Final report Appendices 68
Name Position Stakeholder category
Fleischl, Juliet WHO Representative, Solomon Islands WHO Country Office
Fukuda, Keiji Assistant Director-General - Health Security and Environment
WHO HQ
Garrett, Laurie Journalist and writer Global Health Advisor
Girod, Magali Policy Adviser, Specialised Agencies and Humanitarian Team, UK Mission to the United Nations in Geneva
Member States
Golberg, Elissa Ambassador and Permanent Representative, Permanent Mission of Canada
Member States
Guerma, Teguest Director, AMREF NGO
Guner, Oner DG for Foreign Affairs & EU Member States
Halton, Jane Secretary of the Department of Health and Ageing, Department of Health, Australia
Evaluation Management Group
Han, Tieru DPM, WPRO WHO Regional Office
Hollo, Imre DAF, EURO WHO Regional Office
Hossein Nicknam, Mohammed
Acting Minister for International Affairs, International Affairs Department, Ministry of Health and Medical Education, Iran
Evaluation Management Group
Jacab, Zsuzsanna Regional Director, EURO WHO Regional Office
Jacob, Gabrielle First Secretary (Health), Permanent Mission of Ireland Member States
Jama, Mohamed Assistant Director-General, HQ/GMA GMG ADGO Office of the Assistant DG
WHO HQ
Jee, Youngmee Scientist, WP/RGO/DCC/EPI WHO Regional Office
Jeffreys, Nicholas Comptroller & Director, HQ/FNM Department of Finance WHO HQ
Kickbush, Ilona Director of the Global Health Programme at the Graduate Institute of International and Development Studies, Geneva
Global Health Advisor
Kieny, Marie-Paule Assistant Director-General - Health Systems and Innovation WHO HQ
Kloiber, Omar Director, World Medical Association NGO
Kobza, Jeffrey DAF, WPRO WHO Regional Office
Landry, Steve Deputy Director, Vaccine Delivery, Bill & Melinda Gates Foundation
Donors
Lopez-Acuna, Daniel Advisor to Director-General, HQ/ODG Office of the Director-General
WHO HQ
Lutnaes, Sverre Berg Counsellor, Permanent Mission of Norway Member States
Maina, Emmanuel Chief of the Division of Cooperation of the Ministry of Public Health, Cameroon
Member States
Mamacos, Peter Multilateral Branch Chief, US Department of Health and Human Services
Member States
WHO Reform Stage 2 Evaluation - Final report Appendices 69
Name Position Stakeholder category
Mandlhate, Custodia WHO Representative, Kenya WHO Country Office
Mason, Elizabeth Mary Acting in the absence of Flavia Bustreo, Assistant Director-General - Family, Women's and Children's Health
WHO HQ
Matsoso, Precious Director-General of Health, Ministry of Health, South Africa Evaluation Management Group
Mciff, Colin Health Attaché, Permanent Mission of the United States Member States
Menabde, Nata WHO Representative, India WHO Country Office
Mercado, Julio Counsellor, Permanent Mission in Geneva, Argentina Member States
Migliorini, Luigi WHO Representative, Russia WHO Country Office
Moeti, Matshidiso Deputy Regional Director (acting DPM), AFRO WHO Regional Office
Mogedal, Sigrun Former Norwegian AIDS Ambassador Global Health Advisor
Mourain-Schut, Françoise
Senior Legal Officer, HQ/DGO/DGD/LEG/IAL WHO HQ
Nakatani, Hiroki Assistant Director-General - HIV/AIDS, TB, Malaria and Neglected Tropical Diseases
WHO HQ
Ndiaye, Charlotte WHO Representative, Cameroon WHO Country Office
Ndihokubwayo Jean, Bosco
Regional Advisor, AF/RGO/HSS/BLT WHO Regional Office
Nzang, Andres Acting DAF, AFRO WHO Regional Office
Oduyemi, Cecilia Rose HQ/GBS Dept for Governing Bodies and External Relations WHO HQ
Oedo, Cecilia Finance assistant, HQ/HSS/Essential Medicines and Health Products (EMP) Department
WHO HQ
Osundwa, Anastacia Third Secretary, Permanent Mission of Kenya Member States
Ottiglio, Mario Director, IFPMA NGO
Padilla, Liliana Counsellor, Permanent Mission of Mexico Member States
Patterson, Cath Minister Counsellor (Health), Australian Permanent Mission to the UN, Geneva
Member States
Pendergast, Scott PRP Planning Resource Coordination and Performance Monitoring
WHO HQ
Peruga, Armando Acting in the absence of Dr Douglas Bettcher, Director, Prevention of Noncommunicable Diseases (PND)
WHO HQ
Plianbangchang, Samlee Regional Director, SEARO WHO Regional Office
Profili, Maria Cristina WHO Representative, Turkey WHO Country Office
Quiroz Ávila, Alberto Chief, Secretariat of External Relations, Ministry of Foreign Affairs, Mexico
Member States
WHO Reform Stage 2 Evaluation - Final report Appendices 70
Name Position Stakeholder category
Raviglione, Mario Director of the Global TB Programme WHO HQ
Ren, Minghui Director-General, Department of International Cooperation, National Health and Family Planning Commission, Ministry of China
Member States
Renganathan, Elil Director, HQ/PRP Planning Resource Coordination and Perf Monitoring
WHO HQ
Ruiz Vargas, Ruth Dafne Director of Multilateral Affairs, Secretariat of Health, Ministry of Health, Mexico
Member States
Sambo, Luis G. Regional Director, AFRO WHO Regional Office
Sandrasagren, Mahen Coordinator/ acting Director, HQ/HRQ HQ HR Management WHO HQ
Saphaisal, Kanita First Secretary (Health), Permanent Mission of Thailand Member States
Saunders, Philipa Consultant, OXFAM NGO
Schwartlander, Bernhard WHO Representative, China WHO Country Office
Schwartz, Thomas Director, Medicus Mundi NGO
Shin, Young-soo Regional Director, WPRO WHO Regional Office
Silberschmidt, Gaudenz Senior Advisor, HQ/ODG Office of the Director-Genera WHO HQ
Smith, Ian Advisor to Director-General, HQ/ODG Office of the Director-General
WHO HQ
Souteyrand, Yves WHO Representative, Morocco WHO Country Office
Stuppel, John Health and UN Cross Systems Team Leader, DFID Member States
Sukai, Jackson Inspector, Joint Inspection Unit of the United Nations System
Evaluation Management Group
Tegegn, Yonas WHO Representative, Thailand WHO Country Office
Tennakoon, Herbert Assistant Regional Director, SEARO WHO Regional Office
Troedsson, Hans Executive Director,HQ/ODG Office of the Director-General WHO HQ
Webb, David Director HQ/IOS Office of Internal Oversight Services WHO HQ
Wibulpolprasert, Suwit Vice Chair, International Health Policy Program Foundation (IHPF), Ministry of Public Health, Thailand
Member States
Yalanazan, Irem National Programme Officer, Mental Health and Disability, EU/ACO/TUR
WHO Country Office
Zeltner, Thomas Former Director-General of the Swiss National Health Authority and Secretary of Health of Switzerland, Special envoy to WHO
Global Health Advisor
WHO Reform Stage 2 Evaluation - Final report Appendices 71
Appendix E- List of documentation
Barry R. Bloom, WHO needs change, Nature, Vol 473, 12 May 2011.
Change at WHO, Newsletter on Reform, November 2012 and April 2013
Chatham House, the Role of the World Health organization in the International system, quoted in World Health Organization, The Third Ten Years of the World Health organization 1968-1977.
DFID. Multilateral Aid Review, Ensuring maximum value for money for UK aid through multilateral organisations, March 2011.
Garrett, Laurie. Existential Challenges to Global Health, Center on International Cooperation, New York University, August 2013.
Health In the post-2015 agenda, Report of the Global Thematic Consultation on Health, April 2013
IEOAC, Report of the Sixth Meeting of the Independent Expert Oversight Advisory, February 2013
IHP+. Progress in the International Health Partnership & Related Initiatives (IHP+), 2012 Annual Performance Report.
Ilona Kickbusch, Wolfgang Hein, and Gaudenz Silberschmidt. Addressing Global Health Governance Challenges through a New Mechanism:The Proposal for a Committee C of the World Health Assembly, Journal of law, medicine & ethics, 2010.
Joint Inspection Unit, Review of Enterprise risk management in the United Nations System, Benchmarking framework, 2010.
Joint Inspection Unit of the United Nations, Review of management, administration and decentralisation in the World Health Organization (WHO), 2012.
Ministry of Foreign Affairs of Norway, Cooperation for Health Development, WHO 's support to programmes at country level, September 1997.
Mopan Common Approach 2010: World Health Organization, January 2011.
Morrison JS. The end of the golden era of global health? Center for Strategic and International Studies; 17 April 2012.
Paul Lincoln et al, Conflicts of interest and the UN high-level meeting on non-communicable diseases, Lancet, Vol 378 November 12, 2011.
United States Government Accountability Office, United Nations, Improved Reporting and Member States’ Consensus Needed for Food and Agriculture Organization’s Reform Plan, September 2011.
United States Government Accountability Office, World Health Organization, Reform Agenda Developed, but U.S. Actions to Monitor Progress Could be Enhanced, July 2012.
WHO Accountability Compact between Assistant Director General and Dr Margaret Chan, Director General, World Health Organization, 2013.
WHO and Grayling, Stakeholder Perception Survey Global Communication Strategy Review, January 2013.
WHO Guidance Note, Operational Planning 2014-15 for Reform, Identification and inclusion of reform-specific products/services within 2014-15 operational plans, August 2013.
WHO reform, Consolidated report by the Director-General, 65th World Health Assembly, 25 April 2012.
WHO reform, Financing of WHO, A66/48, 13 May 2013
WHO reform, High-level implementation plan and report, 66th World Health Assembly, 10 May 2013.
WHO reform, High-level implementation plan and report, Report of the Programme, Budget and Administration Committee of the Executive Board to the 66thWorld Health Assembly, 20 May 2013.
WHO Reform Stage 2 Evaluation - Final report Appendices 72
WHO reform, Independent evaluation report: stage one, 65th World Health Assembly, 18 May 2012.
WHO Regional Committee for South-East Asia, Report of the sixty-fifth session, Indonesia, 5-7 September 2012.
WHO Regional Office for Europe, Rules of Procedure of the Regional Committee for Europe and of the Standing Committee of the Regional Committee for Europe, 2011.
WHO Regional Office for the Americas, A conceptual framework for the adoption of enterprise risk management (ERM) in PAHO, 2011.
WHO WPRO, Moving Forward, Making a difference’ Reform experiences of WHO Regional Office for the Western Pacific (2011–2012), Interim Report February 2012
WHO WPRO, Transforming Human Resources Management in the WHO Regional Office for the Western Pacific, 2013.
WHO, Amendments to the Financial Regulations and Financial Rules, Report of the Programme, Budget and Administration Committee of the Executive Board to the Sixty-sixth World Health Assembly, 66th World Health Assembly, 20 May 2013.
WHO, Corporate risk register, Organization-wide strategic risk management in WHO, Report by the Secretariat, Executive Board, 133rd Session, 17 May 2013.
WHO, Country Cooperation Strategy (CCS) WHO – Switzerland, 2013.
WHO, Draft twelfth general programme of work, 66th World Health Assembly,19 April 2013
WHO, Draft workplan for study on strengthening and institutionalization of Category Networks, June 2013.
WHO, Final report of the Task Force on Resource Mobilization and Management Strategies, May 2013.
WHO, Financing Dialogue, Investing in the World's Health Organization, 24 June 2013.
WHO, Financing Dialogue, WHO's budget: What is behind the numbers?, 24 June 2013.
WHO, Follow-up to the Political Declaration of the High-level Meeting of the General Assembly on the Prevention and Control of Non-communicable Diseases, 66th World Health Assembly, 27 May 2013.
WHO, Governance: options for criteria for inclusion, exclusion or deferral of items on the provisional agenda of the Executive Board, Report of the Director-General, Executive Board, 133rd session, 8 May 2013.
WHO, Human Resource Annual Report, Report of the Programme, Budget and Administration Committee of the Executive Board to the Sixty-sixth World Health Assembly, A66/60, 20 May 2013.
WHO, Human resources: annual report, Report by the Secretariat, 66th WHA, 14 May 2013.
WHO, Human resources: annual report, Report by the Secretariat, 66th World Health Assembly, 14 May 2013.
WHO, Modalities for the independent evaluation of the WHO reform: stage two, Report by the Director-General, EB132/5 Add.7, 11 January 2013.
WHO, Notes and Decisions from the Global Policy Group Retreat, 12-13 March 2013 , Chiang Mai, Thailand.
WHO, Notes and Decisions from the Global Policy Group Retreat, 28 May 2013 , Geneva.
WHO, Options to streamline the reporting of and communication with Member States,EB132/5 Add.4, 18 January 2013.
WHO, Programme budget 2014–2015, 66th World Health Assembly,19 April 2013
WHO, Real estate, Report by the Director-General, Executive Board, 132nd session, 7 December 2012.
WHO, Reforms in the WHO Western Pacific Region, Progress reports, June 2011-December 2012 and January-June 2013
WHO Reform Stage 2 Evaluation - Final report Appendices 73
WHO, Report of the regional committees to the Executive Board, Executive Board, EB 132/4, 18 January 2013.
WHO, Resolutions and decisions, 65th World Health Assembly, WHA65/2012/REC/1, 21-26 May 2012.
WHO, Review of management, administration and decentralization in the World Health Organization, report by the Joint Inspection Unit, EB132/5 Add.6, 4 January 2013
WHO, Revised terms of reference for the Programme, Budget and Administration Committee of the Executive Board, EB131.R2, 28 May 2012
WHO, Social determinants of health, Report by the Secretariat, Executive Board, 132nd session, 23 November 2012.
WHO, Streamlining of the work of the governing bodies and harmonization and alignment of the work of regional committees, Executive Board, 132nd session, 14 December 2012.
WHO, The Overarching Roles and Functions of the Three Levels of the Organization, presented and discussed at the Global Policy Group Retreat, March 2013.
WHO, WHO Country Cooperation Strategy 2008-2011 and 2012-2016, Thailand.
WHO, WHO evaluation practice handbook, OIOS.
WHO, WHO reform at country level, Country Coordination and United Nations Collaboration, October 2013.
WHO, WHO reform Programme Management Approach- Reporting, 2013
WHO, WHO reform, towards a new policy of WHO engagement with NGOs, consultation with NGOs 18 October 2012.
WHO, WHO’s arrangements for hosting health partnerships and proposals for harmonizing WHO’s work with hosted partnerships, Report by the Director-General, Executive Board, 132nd session, 23 November 2012.
WHO, WHO’s Financing Dialogue, Mission briefings, IEOAC, 16 October 2013.
WHO, WHO’s role in global health governance, Report by the Director-General, Executive Board, 132nd session, 18 January 2013.
WHO Reform Stage 2 Evaluation - Final report Appendices
PwC 74
Appendix F- Analysis of staff survey