GEOSS Monitoring and EvaluationGEOSS Monitoring and EvaluationCharles S. Baker (USA)

M&E WG Co-ChairJoint meeting of M&E WG and ET

January 20, 2010Geneva, Switzerland

1

2

Requirement for GEOSS M&E Requirement for GEOSS M&E

10-year Implementation Plan (2005):Item 7: Technical Approach, Capacity Building and Outreach:

“To monitor performance against the defined requirements and intended benefits.”

Item 5: Funding and Measuring Progress“GEO will develop performance indicators for GEOSS.”

Resolution of the Third Earth Observation Summit (2005):“Resolve to conduct a mid-term assessment of GEO by 2010.”

Cape Town Declaration (2007):“We resolve to meet again before the end of 2010 to review

progress, conduct a mid-term assessment and give further guidance on the implementation of GEOSS.”

3

At GEO-V in Bucharest, Plenary agreed toAt GEO-V in Bucharest, Plenary agreed to

• Establish M&E WG• Task ExCom with defining M&E WG TOR• Task ExCom with moving process forward• Open M&E WG to membership

4

Accomplishments to Date:Accomplishments to Date:M&E WG meetings• First meeting in March 2009• Second meeting in July 2009

M&E Briefings to ExCom• First briefing in June 2009 • Second briefing in September 2009

M&E Briefing to GEO VI Plenary• GEO-VI accepted M&E documents in Nov 2009• Nominations for Evaluation Team received

M&E Documents ProducedM&E Documents Produced1. M&E WG Terms of Reference

2. M&E Framework Document

3. Summary Plan for the First Evaluation

5

6

Terms of ReferenceTerms of Reference• Define Purpose of M&E WG• Define Objectives of M&E WG

– Develop M&E Framework– Develop Evaluation Plan

• Define Approach and Functions of M&E WG– M&E process to be “internal”, non-complex and low cost– M&E to rely extensively on existing tools and instruments– Performance indicators to be uncomplicated, yet viable– M&E to be conducted by individuals designated by GEO members– M&E WG to provide oversight of the M&E process

Monitoring and Evaluation FrameworkMonitoring and Evaluation FrameworkPrinciples1. M&E will be conducted within the perimeters of the GEO community;

2. M&E will take into account the non-binding nature of GEO;

3. M&E will make use of existing M&E tools and practices used in GEO;

4. M&E will address the full spectrum of GEOSS implementation activities;

5. M&E will use performance indicators for monitoring;

6. The M&E process will engage the user community;

7. M&E will provide feedback for improvement of future GEOSS planning and implementation;

8. Lessons learned from the conducted evaluation will inform improvements to the subsequent evaluation processes.

7

8

GEOSS Implementation Phases

Planning Evaluation Execution

Principle 7. M&E will provide feedback for improvement of future GEOSS planning and implementation

9

ActivitiesActivities Outcomes Impacts

What progress did we make towards achieving the targets?

Evaluation:

Did our activities produce intended outputs?

Monitoring:

OutputsInputs

GEOSS Logic ModelGEOSS Logic Model

Approach to MonitoringApproach to Monitoring• Monitoring will track the progress of Task completion• GEO Secretariat will perform monitoring• Monitoring will be ongoing and systematic• Monitoring will coincide with Work Plan reporting cycle• Information will be collected through Task Sheets• Task Sheet Template modified, based on M&E WG input, to:

– Identify key milestones, with expected dates for their achievement– Identify quantitative indicators of progress, where appropriate– Enable collection of information on the use of portals – Separate planning from reporting information – Link overarching Tasks to Strategic Targets

10

Approach to Monitoring (cont’d)Approach to Monitoring (cont’d)• Work Plan Progress Report

• Secretariat will score Overarching Tasks (green, yellow, red)• Progress Reports will show the score of the Sub-Tasks when

necessary to explain the score of an Overarching Task;

• Secretariat will identify any gaps in the Work Plan (i.e., critical areas requiring work but for which no task is yet underway).

• Revised GEOSS Roadmap database will provide information for assessing progress toward achieving Targets;

• Secretariat will give an overall score for each Strategic Target;• Distribution of Progress Report should be expanded to ensure that

all parties involved in proposing, executing, or reviewing corrective actions are fully informed at all stages of the process.

11

Approach to Monitoring (cont’d)Approach to Monitoring (cont’d)• M&E WG will develop performance indicators for monitoring;• Draft set of indicators will be presented to the ExCom for

approval in late spring 2010;• Upon approval of these indicators by ExCom, Secretariat will

expand Progress Report to include a report on each of the indicators.

12

13

Approach to GEOSS EvaluationApproach to GEOSS EvaluationEstablish Evaluation Team to conduct evaluation;

New Evaluation Team will be appointed for each evaluation;

First evaluation, mid-term assessment, based on Cape Town Declaration;

Second evaluation to begin transition to regular cycle of evaluating GEOSS Targets:• One GEOSS Transverse Area Target• One GEOSS Societal Benefit Area Target• Evaluation of unplanned contributing activities;

Third and subsequent evaluations to be based on a regular cycle:• Two or three Transverse Area Targets per year• Three Societal Benefit Area Targets per year• Future ministerial statements included in the scope of evaluations;

Complete evaluation report(s) will be offered for factual review by the stakeholders.

14

GEOSS Building Blocks and SBAsGEOSS Building Blocks and SBAs

15

Summary Plan for the First EvaluationSummary Plan for the First Evaluation• First evaluation will focus primarily on progress towards realizing

priorities identified in the Cape Town Declaration; • Other aspects of GEOSS implementation will be evaluated within

the context of these priorities;• Strategic Targets will be a key reference for clarifying the

intended priorities and outcomes for GEOSS;• Evaluation Team will make extensive use of Progress Reports and

updated GEOSS Roadmap database;• Draft evaluation report will be circulated within the GEO

community for factual review;• The Evaluation Team is solely responsible for the findings,

conclusions and recommendations in the evaluation report.• ExCom will coordinate response to the recommendations.

15

Cape Town DeclarationCape Town Declaration“We confirm our common view that:• The sustained operation of terrestrial, oceanic, air-borne, and space-based observations networks is critical for

informed decision making;• Data interoperability is critical for the improvement and expansion of observational, modelling, data

assimilation, and prediction capabilities;• Continued research and development activities and coherent planning as essential for future observation

systems;• Continued cooperation and dialogue will establish GEOSS as a powerful means to support informed decision

making; • Coordination, at national, regional and global levels, continued investments, scientific and technological

advances and innovative approaches to financing will be vital for upgrading and expanding Earth observations and building the capacity of individuals, institutions and systems, particularly in developing countries;

”We support the establishment of a process with the objective to reach a consensus on the implementation of the Data Sharing Principles for GEOSS to be presented to the next GEO Ministerial Summit. The success of GEOSS will depend on a commitment by all GEO partners to work together to ensure timely, global and open access to data and products;”

”We commit to explore new ways and means for the sustained operations of the shared architectural GEOSS components and related information infrastructure;”

”We commit ourselves to working together to improve the interoperability of and access to observation and associated prediction and information systems towards the continued strengthening of GEOSS and the full realization of the 10-Year Implementation Plan.”

16

17

Evaluation TeamEvaluation Team• M&E WG will appoint Chair/Co-Chairs.

17

18

Timeline of Evaluation ActivitiesTimeline of Evaluation ActivitiesJ oint meeting of M&E WG and Evaluation Team Early J anuary

2010

Submit a draft Detailed Evaluation Plan to the M&E WG

End J anuary 2010

M&E WG approves the Detailed Evaluation Plan Mid February 2010

Evaluation Team issues Interim Report on findings Mid April 2010

Evaluation Team issues Final Evaluation Report to M&E WG

Mid J une 2010

M&E WG submits Final Evaluation Report for Executive Committee review and coordinated response to the recommendations of the Evaluation Report

3 weeks prior to ExCom summer meeting 2010

Evaluation report + Executive Committee response submission to Plenary and Summit (WG)

Mid September 2010

18

BACK UP SLIDESBACK UP SLIDES

19

20

Guidelines for M&E WGGuidelines for M&E WG

Prepare final draft ToR• Recognizing that M&E should be within GEO perimeter• Recognizing that M&E should involve user community• Recognizing that M&E should be independent of Committees

Develop M&E Framework• Observe the need for efficiency, simplicity and optimal use of resources• Focus initially on monitoring, making use of available tools and instruments• Develop a simple procedure for evaluation• Recommend improvements to Task Sheets as the key monitoring tool• Review and consolidate GEOSS Roadmap• Develop parameters, criteria and indicators for evaluation

21

Guidelines for M&E WG cont’d.Guidelines for M&E WG cont’d.

Develop Plan for First Evaluation• Recognizing particular nature of GEO• Recognizing the pilot nature of evaluation• Recognizing the need to inform improvements in M&E framework• Recognizing challenging schedule

Evaluation Questions What Should be Observed

(Evaluation Criteria)

Information Required

To what extent do the Strategic Targets (and the associated outcomes) adequately reflect the priorities and commitments in the Cape Town Declaration?

Strategic Targets include all significant aspects of the priorities and commitments in the Cape Town Declaration.

Cape Town DeclarationStrategic Targets document Evidence supporting the interpretation of the above documents.

To what extent are the Work Plan Tasks reflective of the actions required to achieve the Strategic Targets?

All Tasks are necessary to the achievement of the Strategic Targets. Processes are in place to identify gaps in the current Work Plan where additional Tasks are required to achieve Strategic Targets and to solicit offers to fill these gaps.

Strategic Targets documentGEOSS RoadmapCompleted Task Sheets (current and past versions) Evidence regarding the relationship of Tasks to Targets. Evidence regarding work planning and gap analysis processes.

To what extent are the Tasks being implemented as intended?

Statement of targets and milestones in GEO planning documents in a form that permits objective verification against actual performance Actual performance reflects substantial achievement of performance expectations

Completed Task Sheets (current and past versions). Work Plan Progress Reports and other information regarding the status of all Tasks. Information concerning the actual conduct of work planning, Task management, and reporting processes

22

To what extent are expected outcomes of GEOSS being realized?

Clearly stated expected results that inform work planning and performance monitoring Achievement of near-term outcomes in some domains, although likely of limited scope

Statement of expected outcomes for GEOSS in a form that depicts the logical relationships between GEOSS Tasks and the expected outcomes at various levels and/or time scales and between the expected outcomes and the ultimate intended impacts of GEOSS Evidence that statements of GEOSS outcomes influence the selection of Tasks Information regarding processes for the monitoring and assessment of outcomes related to GEOSS Evidence regarding the achievement of near-term GEOSS outcomes

Are there significant unplanned benefits or unintended negative consequences of GEOSS?

Documentation of unplanned benefits of GEOSS Absence of any significant negative unintended consequences.

Information concerning the existence and magnitude of unintended consequences of participation in GEO, GEO Tasks or of GEOSS operations

23