10/27/2014
1
NOYCE PROJECT EVALUATION
John T. Sutton, Ph.D.
RMC Research Corporation
633 17th St., Suite 2100
Denver, CO 80202
(800) 922-3636
PRESENTERS
Brandie Ward, RMC Research Corporation
Arlene Mitchell, RMC Research Corporation
Emma Espel, RMC Research Corporation
10/27/2014
2
REFLECTION QUESTIONS
What type of NOYCE project are you evaluating and what is the
greatest challenge you encounter in conducting the evaluation?
What measures or protocols do you use and for what purposes?
What methods do you use to report your findings and how useful is
that perceived?
REFLECTION QUESTIONS
Which of the Program Evaluation Standards do you feel has the
greatest application in the NOYCE project evaluation? Why?
If you were giving advice to another Noyce Project that was
thinking about how to improve or enhance their evaluation, what
would that be?
10/27/2014
3
NOYCE PROJECT EVALUATION
Brandie WardRMC Research Corporation633 17th St., Suite 2100Denver, CO(800) [email protected]
TYPE OF NOYCE PROJECT EVALUATING?
Capacity Building Grant:
University of Denver – Access in Mathematics for All (AMA)
10/27/2014
4
GREATEST CHALLENGE ENCOUNTERED IN CONDUCTING THE EVALUATION?
Small budget limits evaluator participation
Reliant on project documents
MEASURES OR PROTOCOLS USED?
Document review to monitor their communications and processes to achieve their goals for the first year.
Review of:Tasks and timelinesMeeting minutes, action items, status updatesEmailsActivity logsProject records (participant database)
10/27/2014
5
METHOD FOR REPORTING/COMMUNICATING
No report, yet
Communicate via email
Provide recommendations to the project, sharing documents, resources, and sharing a participant tracking database.
PROGRAM EVALUATION STANDARD
Utility Standard, specifically related to the subcomponents
U5 Relevant Information U6 Meaningful Processes and Products U7 Timely and Appropriate Communicating
and Reporting
10/27/2014
6
ADVICE?
Involve your evaluator at the very beginning stages of the project, if possible
Be in regular communication
Arlene P. MitchellRMC Research Corporation
Denver, CO
10/27/2014
7
Master Teachers (MTFs) 23 original MTFs plus 7 added Instructors for professional development through
UNL during summer, school year
Teaching Fellows (TFs) 14-month program to earn Masters degree Complete an entire year of interning and student
teaching in classroom with Master teacher
2 cohorts Cohort 1 (6 TFs): Completed 2nd year of teaching Cohort 2 (7 TFs): Completed 1st year of teaching
Teacher Survey (online) Quantitatively measure changes in preparedness and use of
instructional strategies, confidence and level of support for teaching and learning mathematics, and engagement in activities related to teaching and learning mathematics.
Teacher Leader Inventory (online) Adapted from the Teacher Leader Model Standards from the
Teacher Leadership Exploratory Consortium Assess quantitatively the extent to which MTFs helped with
creating a collaborative culture, assessing and using research, promoting professional learning, using assessments and data, and improving outreach and collaboration.
10/27/2014
8
Telephone interviews with randomly selected MTFs Questions about their contributions and experiences
to school, district, and UNL
Telephone interviews with district mathematics coordinators and principals Impact of the NOYCE project in their district or
contributions to their high-need building
Cohort 1 Open-ended survey about
program Open-ended survey about
1st year of teaching Teacher Survey Teacher Leader Inventory Interviews
Preparation to teach, philosophy of teaching, and intent of their instruction
Interviews with principals
Cohort 2 Open-ended survey
about program Open-ended survey
about 1st year of teaching
Teacher Survey Interviews with
principals
10/27/2014
9
Measures of central tendencies Teacher Survey: MTFs, TFs Teacher Leadership Inventory: MTFs, TFs-Cohort 1
ANOVAs Teacher Leadership Inventory for MTFs for statistically significant
changes over time Teacher Survey for TFs-Cohort 1 for statistically significant changes
over time
Cohen’s d Effect size reported for any statistically significant findings
Qualitative data Summarized using categories and themes
Utility:Credibility, attention to stakeholders, relevant information, timely and appropriate communicating and reporting
Feasibility:Project management, practical procedures, resource use
Accuracy:Justified conclusions and decisions, valid information, reliable information, explicit program and context descriptions, information management, sound designs and analyses, communication and reporting
10/27/2014
10
Importance of principal investigators Contacting MTFs and TFs to complete evaluation
pieces Communicating to adjust evaluation pieces during
project’s life
Maintaining data records for project’s life Well-organized data base to compare results over
time for impacts
Emma Espel
10/27/2014
11
What type of NOYCE project are you evaluating and what is the greatest challenge you encounter in conducting the evaluation?
What type of NOYCE project are you evaluating and what is the greatest challenge you encounter in conducting the evaluation?
Midwest Regional Robert Noyce Connections
10/27/2014
12
What type of NOYCE project are you evaluating and what is the greatest challenge you encounter in conducting the evaluation?
Goals
Increase the personal connections among Midwest Noyce project members;
What type of NOYCE project are you evaluating and what is the greatest challenge you encounter in conducting the evaluation?
Goals
Increase the personal connections among Midwest Noyce project members;
Utilize these new and/or strengthened connections to share expertise and knowledge across Midwest Noyce projects;
10/27/2014
13
What type of NOYCE project are you evaluating and what is the greatest challenge you encounter in conducting the evaluation?
Goals
Increase the personal connections among Midwest Noyce project members;
Utilize these new and/or strengthened connections to share expertise and knowledge across Midwest Noyce projects;
Support the professional development of Noyce Scholars; and
What type of NOYCE project are you evaluating and what is the greatest challenge you encounter in conducting the evaluation?
Goals
Increase the personal connections among Midwest Noyce project members;
Utilize these new and/or strengthened connections to share expertise and knowledge across Midwest Noyce projects;
Support the professional development of Noyce Scholars; and
Build a community of practice among Noyce Scholars.
10/27/2014
14
What type of NOYCE project are you evaluating and what is the greatest challenge you encounter in conducting the evaluation?
To what extent do university‐related Noyce personnel, K‐12 Noyce Teacher Scholars, and conference participants expand their connections as a result of participating in project sponsored activities?
What type of NOYCE project are you evaluating and what is the greatest challenge you encounter in conducting the evaluation?
To what extent do university‐related Noyce personnel, K‐12 Noyce Teacher Scholars, and conference participants expand their connections as a result of participating in project sponsored activities?
10/27/2014
15
What type of NOYCE project are you evaluating and what is the greatest challenge you encounter in conducting the evaluation?
To what extent do university‐related Noyce personnel, K‐12 Noyce Teacher Scholars, and conference participants expand their connections as a result of participating in project sponsored activities?
What measures or protocols do you use and for what purposes?
10/27/2014
16
What measures or protocols do you use and for what purposes?
Quantitative and Qualitative
Online surveys
Needs assessments
Conference evaluation
Activity satisfaction
What measures or protocols do you use and for what purposes?
Quantitative and Qualitative
Online surveys
Needs assessments
Conference evaluation
Activity satisfaction
Social media: LinkedIn™ Group
Track group characteristics
10/27/2014
17
What methods do you use to report your findings and how useful is that perceived?
Needs assessment report
Annual report
Image from Chatty, D., Baas, S., & Fleig, A. (2003). Participatory processes towards co‐management of natrual resources in pastoral areas of the Middle East. Food and Agriculture Organization of the United Nations.
What methods do you use to report your findings and how useful is that perceived?
Needs assessment report
Annual report
Image from Chatty, D., Baas, S., & Fleig, A. (2003). Participatory processes towards co‐management of natrual resources in pastoral areas of the Middle East. Food and Agriculture Organization of the United Nations.
10/27/2014
18
Which of the Program Evaluation Standards do you feel has the greatest application in the NOYCE project evaluation? Why?
Yarbrough, D.B., Shulha, L.M., Hopson, R.K., and Caruthers, F.A. (2011). The Program Evaluation Standards: A Guide for Evaluators and Evaluation Users, 3rd ed. Mountain View, CA: Sage.
Which of the Program Evaluation Standards do you feel has the greatest application in the NOYCE project evaluation? Why?
Utility
Yarbrough, D.B., Shulha, L.M., Hopson, R.K., and Caruthers, F.A. (2011). The Program Evaluation Standards: A Guide for Evaluators and Evaluation Users, 3rd ed. Mountain View, CA: Sage.
10/27/2014
19
Which of the Program Evaluation Standards do you feel has the greatest application in the NOYCE project evaluation? Why?
Utility
Yarbrough, D.B., Shulha, L.M., Hopson, R.K., and Caruthers, F.A. (2011). The Program Evaluation Standards: A Guide for Evaluators and Evaluation Users, 3rd ed. Mountain View, CA: Sage.
Which of the Program Evaluation Standards do you feel has the greatest application in the NOYCE project evaluation? Why?
Utility
Yarbrough, D.B., Shulha, L.M., Hopson, R.K., and Caruthers, F.A. (2011). The Program Evaluation Standards: A Guide for Evaluators and Evaluation Users, 3rd ed. Mountain View, CA: Sage.
10/27/2014
20
Which of the Program Evaluation Standards do you feel has the greatest application in the NOYCE project evaluation? Why?
Utility
Yarbrough, D.B., Shulha, L.M., Hopson, R.K., and Caruthers, F.A. (2011). The Program Evaluation Standards: A Guide for Evaluators and Evaluation Users, 3rd ed. Mountain View, CA: Sage.
Which of the Program Evaluation Standards do you feel has the greatest application in the NOYCE project evaluation? Why?
Utility
Yarbrough, D.B., Shulha, L.M., Hopson, R.K., and Caruthers, F.A. (2011). The Program Evaluation Standards: A Guide for Evaluators and Evaluation Users, 3rd ed. Mountain View, CA: Sage.
10/27/2014
21
If you were giving advice to another Noyce Project that was thinking about how to improve or enhance their evaluation, what would that be?
If you were giving advice to another Noyce Project that was thinking about how to improve or enhance their evaluation, what would that be?
Be willing to explore other options to creatively and usefully target research questions using available resources (e.g., social media as an evaluator tool)