Designing Evaluation Plans for Science & Technology Projects to … · 2020. 6. 25. · Factors outside the project’s control that can influence the ability of the project to achieve

Copyright © American Association for the Advancement of Science 6/25/2020 1

Designing Evaluation Plans for Science & Technology Projects to Strengthen Implementation and Outcomes

Maysaa Alobaidi, Ph.D. Senior Program Associate, Research Competitiveness ProgramAmerican Association for the Advancement of Science (AAAS)


Thank you

Hannah Weiss, M.S. Heather McInnis, Ph.D. Irene Aninye, Ph.D.

Martin Fernandez, M.S. Charles E. Dunlap, Ph.D.


About AAAS

• World’s largest general scientific membership society

• Founded in 1848

• Publisher of six international journals including Science

• More than 250 affiliated societies serving >10,000,000 people

• AAAS Research Competitiveness Program has worked for 20 years to build capacity for research communities

American Associationfor the Advancement of Science

https://promo.aaas.org/science/join/


About RCP

Review of thousands of proposals annually

Strategic assessment and data-driven evaluation of more than $1 billion in S&T programs

Short courses for students, faculty, and administrators

PDF overviewbit.ly/aaas-rcp


Announcements

• A PDF of this presentation will be available—an e-mail will be sent to everyone registered.

• An on-demand recording will be made publicly available for free—the link will be sent to everyone registered.

• Please send questions via the Q&A box. Questions are only visible to us. We will read them and answer them anonymously.


Goals

Overview

• The use of evaluation in S&T projects• Evaluation framework for S&T projects• Trends and challenges in evaluating S&T projects• Designing and developing evaluation plans for S&T

projects• Working with evaluators

Evaluation of science and technology (S&T) projects:

History of Program Evaluation

1

Industrial Revolution and Reform Movements

2Expansion of Social Programs

3

Focus on Educational Assessment

4Legislation in the U.S. Mandating Evaluation

5

Professionalization of Evaluation

6Backlash against Social Programs

7

Expansion and Integration of Evaluation

1800-19301930-1950

1950-19601960-1970

1970-1980 1990-present1980-1990

Source: Stufflebeam, D. L., Madaus, G. F., & Kellaghan, T. (2000). Evaluation models: Viewpoints on educational and human services evaluation. Boston: Kluwer Academic Publishers.


Evaluation

• A systematic assessment of an activity, project, or program to determine:

• Efficiency • Effectiveness• Impact

• Why do evaluation? • Building consensus as a basis for effective

collaboration among team members• Adhering to funding agency requirements • Informing future policies and decision

making


Paradigm Shift

• Stakeholder involvement • Learning process • Actionable recommendations • Analysis of impact and ROI


When does evaluation happen during the life cycle of an S&T Project?

Proposed Project Funded Project Planning

Project Implementation Project Closeout

Knowledge spillover, Capacity

improvementSocietal and

Economic Impacts

1 2 3 4 5 6


Evaluation of Science & Technology Projects

Proposed Project Planning of Funded Project

Project Implementation Project Closeout Knowledge spillover,

Capacity improvementSocietal and Economic

Impacts

Proposals Evaluation

Strategic Planning Monitoring Evaluation






Impacts

Evaluation of Proposals


Expert Review (Evaluation) of Proposals

• Expert peer review plays a critical role in facilitating resources allocation decision making (e.g., research funding)

• Can Scientists (as member of peer review panels) accurately evaluate the potential economic and societal impact of proposed research projects?






Impacts

Strategic Planning


Strategic Planning – Evaluation Logic Model

ImpactsLong-term Outcomes

Intermediate Outcomes

Short-term OutcomesActivities Inputs

Contextual Factors: Factors outside the project’s control that can influence the ability of the project to achieve its goals and objectives

Assumptions: Assumptions made by project leaders about how different parts of the project are supposed to work and the resulting outcomes

Actions taken to achieve

project goals

Resources available/needed to achieve project

goalsResults/benefits/changes

Products resulting

from project

Outputs


Strategic Planning-Mapping Evaluation Needs

What is being evaluated in Science & Technology Projects?• Innovative and cutting edge

research• Collaboration networks

• Multi-disciplinary

• Multi-sectoral

• Multi-jurisdictional

• Research capacity development

• Diversity and workforce development

• Technology development and transfer



Economic Impacts• Demonstrating the contribution of

research to economic development and competitiveness• Multi-level approach to evaluation

(going beyond the project level)

• Understanding the different pathways to economic impacts



Societal Benefits

• Targeting broader societal goals such as improvements to the environment, public health, security, diversity and inclusion.



International Research Collaborations

• Growth of international collaborations• Differences in research cultures• Linkages with industry


Stakeholder Mapping & Engagement


•Stakeholder A•Stakeholder B

•Stakeholder E

•Stakeholder B•Stakeholder D

•Stakeholder A•Stakeholder B•Stakeholder C

Project Theme or Objective





The evaluation plan outlines key questions to be answered at different phases of a project.

• Inputs Who will participate in the project? What infrastructure and resources will be

used to implement project activities?

• Activities What research, education, training, and

outreach and engagement activities are planned?

Designing and Developing an Evaluation Plan


• Outputs What products are expected to result from

implementing project activities? e.g., publications, technologies, patents

• Outcomes How will the outputs resulting from the

project be used? By who? e.g., uses of publications, technologies, and patents



• Impacts What ecosystem and societal benefits are

expected to result from the project's work? e.g., downstream effect on knowledge, innovation, economic growth, etc.



The plan also outlines the evaluation ‘study design’.

• Experimental design –randomized controlled trials; the ‘gold standard’ of study design, rarely used in evaluation due to cost and feasibility issues.




Quasi-experimental design-more commonly used in evaluation; e.g., pre-post comparison, comparison group design, and interrupted time series.

Observational/descriptive study design – more feasible but lacks rigor terms of inferring causality; e.g., case studies.


The plan also outlines the evaluation data collection and analysis methods.• Selection of methods depends on the

questions being asked.

• It also depends on the availability of time and resources to collect and analyze evaluation data.

• The use of different types of data and methods can strengthen the quality of evidence generated and improve the ability to reach different audiences.



• Examples of data collection and analysis methods: • Surveys• Interviews/focus groups• Social Network Analysis• Scientometrics• Historical Tracing• Econometrics







Impacts

Monitoring


Evaluating the Performance of S&T Projects

Assessment and feedback on progress relative to goals and objectives. • Leadership & Management• Communication • Collaborations/Partnerships• Context/Environment


Evaluating the Performance of S&T Projects

Adjusting evaluation to potential shifts in project goals and design

• Should the goals that were agreed upon in the beginning of the project be considered as the standard against which to measure goal achievement?

• How to account for the learning and potential changes and adaptations to project design and goals during the course of project implementation?






Impacts

Evaluation


Evaluation of the Outcomes & Impacts of S&T Projects

• Short- and Mid-Term -immediately or shortly after the end of the project

e.g., uses of publications, technologies, and patents

• Long-Term Outcomes/Impacts -past project period

e.g., downstream influence on knowledge, innovation, economic growth, and other societal benefits (e.g., health, security, and environmental benefits)


Challenges with Capturing Long-term Outcomes and Impacts

• The impacts of S&T projects can sometimes be unpredictable and take a long time to materialize

• The link between research and impact is diffuse and indirect (especially at the project level and for basic research)

• Expected impacts usually require other inputs (the S&T project is only one contributing factor)


• Ensuring the use of evaluation findings and recommendations requires: • Provision of a detailed

implementation plan • Engaging relevant stakeholders

through clear communication of findings and underlying analysis

• Tracking implementation of evaluation findings

Implementation of Evaluation Findings


• An effective strategy for the communication and dissemination of evaluation findings can enhance their use and implementation.

• Timely dissemination of evaluation findings and recommendations.

• Stakeholders should be informed of how the evaluation findings may affect them as individuals or groups.

• The communication of evaluation findings should be tailored to the specific needs of different audiences.

Communication & Dissemination of Evaluation Findings


What should evaluation reports cover?• Key findings and recommendations

from the evaluations including progress and gaps

• Relevance of findings and recommendations to the participants and to project implementation and management

• Success stories and best practices• Recommendations for improvement

Communication & Dissemination of Evaluation Findings


AAAS RCP Evaluation Approach


• Identify and engage an evaluator as early as possible (e.g., proposal development stage). If engaged later in the process, this will limit the evaluator’s ability to :• Provide guidance on relevant

evaluation questions

• Provide feedback on project design to accommodate evaluation activities

• Collect relevant data to answer evaluation questions

Working with Evaluators


Core competencies of evaluators: • Professional Practice• Methodology • Context • Planning And Management• Interpersonal


Source: American Evaluation Association (AEA): https://www.eval.org/page/competencies

https://www.eval.org/page/competencies



Guiding Principles For Evaluators: • Systematic Inquiry• Competence• Integrity/Honesty• Respect for People• Responsibilities for General and

Public Welfare

Source: American Evaluation Association (AEA): https://www.eval.org/p/cm/ld/fid=51


• Evaluations can provide critical information to allow for a more efficient and effective project implementation and management:• Evaluation can contribute to a more

informed process of program design and implementation

• Evaluation can provide an opportunity to address early challenges

• Evaluation can help identify early challenges and lessons to improve performance.

The benefits of Evaluation to S&T Projects


• Commitment from project leadership to evaluation activities and implementation of evaluation findings is critical to the success of evaluation.

Embracing a Culture of Evaluation


All graphics and images used in this presentation (excluding illustrative charts with textual content) are made with Infogram (Copyright © 2017 Prezi, Inc.)


PDF overviewbit.ly/aaas-rcp

Maysaa Alobaidi, Ph.D.Senior Program Associate Research Competitiveness ProgramAmerican Association for the Advancement of Science1200 New York Ave NWWashington DC 20005T: 202.326.6488Email: [email protected]

Documents

Designing Evaluation Plans for Science & Technology Projects to … · 2020. 6. 25. · Factors outside the project’s control that can influence the ability of the project to achieve