The Roles of Users in Enhancing Utility of Evaluation

With Special Reference to the International Evaluationon the Funding and Management Performance of the National Natural Science Foundation of China

Prof. Yonghe Zheng, Deputy Director, Bureau of Planning, NSFCProf. Zhaoying Chen, Deputy Director, NCSTE

Outline

Brief overview of the national evaluation effort in China

NSFC case: International Evaluation on NSFC’s Funding and Management Performance

The Roles of Users in Enhancing Utility of Evaluation

1. Brief overview of the national evaluation effort in China

Currently, to establish an evaluation system within the public administration system in China has become an urgent need in Chinese governments.

The whole – of - government performance evaluation system has not established in China.

MoF is the leading body

Ministry of Finance (MOF) is the leading body of the performance evaluation of public sectors in China.

There are already some pilot activities of performance evaluation, which carried out by line ministries or local governments in China.

2. NSFC case: International Evaluation on NSFC’s Funding and Management Performance

The evaluation is commissioned by MOF and NSFC together. The State Council attached importance to this evaluation.

The NSFC case is the most influential evaluation in public sector in China up to now and represents good practice from China.

6

Dual Objectives of the Evaluation

6

■ Accountability Provide an independent assessment of the overall performance of NSFC’s funding and management during the past 25 years

■ Lesson learning Improve the NSFC’s funding and management performance; redefine NSFC’s strategic role within the NIS of China

7

What the evaluation focuses on and what the evaluation does not attempt to do

7

The evaluation does

■Focus on NSFC’s funding and management performance over 25 years

■Focus on NSFC’s overall performance as a whole

The evaluation does NOT

Assess NSFC from the agency’s internal perspective

Assess individual projects supported by NSFC

Evaluation Approach and Principals

Modality: domestic preparation followed by international evaluation

Approach: Evidence-based Principals: Independent, Objective,

Fairness

The scope of the Evaluation will cover four dimensions:

(1) NSFC’s strategic positioning,

(2) funding performance,

(3) management performance,

(4) impact on China’s S&T system.

Evaluation Issues

Issue 1. NSFC’s strategic positioning in the national innovation system of China

Issue 2. Funding strategyIssue 3. Contributions to original innovationIssue 4. Promoting the balanced development of scientific

disciplinesIssue 5. Fostering innovative talentsIssue 6. Supporting the national demands and challenges Issue 7. Modality of funding managementIssue 8. Funding instrumentsIssue 9. Peer Review SystemIssue 10. Impact of NSFC’s funding

11

Implementation of the Evaluation

11

Phase Major Evaluation Activities

Phase12010. 2 － 2010. 6

Inception and evaluation design

Phase22010. 7 － 2010.12

Prepare evaluation evidence report The IEC first meeting in Beijing

Phase32011. 1 － 2011. 6

Complete the Evaluation Report

Phase42011 年 7 － 10 月

Use and dissemination of the evaluation results

12

Use of the Evaluation

12

■NSFC leadership understands why the organization needs evaluation

■The use of evaluation has been considered to be key issues to be addressed in the evaluation design from the very beginning and also made systematic arrangements

■During the implementation of the evaluation, some findings have already been put into application.

■There will be a series of follow-up activities to ensure that evaluation is used appropriately

3. User’s Role in Enhancing Utility of Evaluation

NSFC: Inception and evaluation design Ensure that the evaluation is targeted to

the demands of users, respond to current concerns and debates;

Discuss with NCSTE team about the conception framework of the evaluation; Identify the 10 key issues and evaluation questions;

Promote IEC and NCSTE team to understand NSFC’s funding and management system;

NSFC: Evaluation Implementation Provide coordination and support

requested by IEC and NCSTE team; Prepare a volume of case-based evidence

for the evaluation; Assure that NCSTE can access to all

relevant information and select the target group for interview or focus group meeting;

Judge the credibility of the message in the evaluation report

3. User’s Role in Enhancing Utility of Evaluation

NSFC: Response and Dissemination Take management response to the

evaluation report at the draft stage Distribute the evaluation report to line

ministries, the National People's Congress, the Chinese People's Political Consultative Conference

Arranging dissemination events of the evaluation.

3. User’s Role in Enhancing Utility of Evaluation

NSFC: Follow-up NSFC leading level and director level:

One week seminar to discuss the main results of the evaluation; identify possible action in NSFC strategy and funding management;

NSFC staff: improving conditions for evaluation-based learning in NSFC;

Release the report to the public

3. User’s Role in Enhancing Utility of Evaluation

Primary users should really understand why they need evaluation;

The use and dissemination of evaluation should be addressed from the beginning and also built into the work plan of the evaluation.

Lesson Learned

Thanks！