Project Evaluation Workshop

Project Evaluation Workshop

Massachusetts Board of Higher Education

Improving Teacher Quality Higher Education Partnership Program

Hoagland-Pincus Conference Center

March 3, 2005

2

Workshop Objectives

Through an interactive process:• Develop shared understanding of minimal

expectations/standards for TQ project evaluations and reporting

• Work towards a common conceptual model of professional development as basis for common approach to program evaluation

• Identify next steps for technical assistance with evaluation plans

• Support networking among projects to facilitate sharing of evaluation approaches, measures, tools

3

Challenges

• Diversity among projects– Content areas– Target audiences– PD delivery mechanisms– Duration– Current stage of projects (new vs. ongoing)– etc., etc.

• Resource limitations

4

Agenda

• Introductions

• BHE’s Evaluation and Reporting Expectations

• State-level Data Collection

• Logic Model of Professional Development

• Planning for Project Evaluation

• Next Steps

5

Who We Are: Background

UMass Donahue Institute:

• Public service and outreach arm of the UMass President’s Office

• Broad range of services to federal, state, local public and non-profit organizations; services include applied social science research and program evaluation

• Experience evaluating large statewide education reform initiatives for the Massachusetts Department of Education and Board of Higher Education

• Experience developing and implementing evaluation plans for professional development and other educational programs and interventions at the local, regional, and state levels

6

Who We Are: TQ Evaluation Team

Eric Heller, UMass Donahue Institute Director of Research and [email protected]

Christine Lewis, Research ManagerLead Project Manager for TQ [email protected]

Jean Supel, Research ManagerCo-Project Manager for TQ [email protected]

7

Who We Are: Functions of TQ Evaluation Team

• Coordinate state-level collection of standardized project data and reports on behalf of Board of Higher Education

• Provide technical assistance to projects in support of quality project evaluation efforts

• Develop state level project report through aggregation of project data and meta-analysis of project reports

8

Who You Are: Introductions and Project Overview

• Subject Matter (e.g. Math, Language Arts, Math/Science, etc.)• Project Length• New or Existing Project• Target Population

– Single or multiple cohort(s)

– Teachers and/or Paraprofessionals; other

– # of participants anticipated

– School district(s) Involved

– Grade level(s)

• Type(s) of Professional Development Activities (e.g. Summer Institute, Graduate level course, After school group, classroom support, etc.)

• Brief Description of Activities

9

Evaluation & Reporting Expectations - 1

• Develop and Implement a Project Evaluation that addresses:

– Formative Evaluation Objectives• Provide timely feedback on project activities

• Identify strengths and weaknesses

• Identify gaps, unmet participant needs

• Support continuous improvement of content and delivery

– Summative Evaluation Objectives• Based on logic model of professional development• Document project implementation model (for replication)• Measure participation levels• Measure short-term participant outcomes (required)• Measure longer-term participant outcomes (to the extent feasible)• Measure student outcomes (to the extent feasible)

10


• Activity Tracking and Reporting

– Target Audience– # Participants – Subject– Grade Level– Duration (# hours)– Timespan– # Credits

Note: ACTIVITY data for each completed PD activity will be collected annually through BHE online system

11


• School Tracking and Reporting

– School Name– District– School Type (public/private/charter) – Poverty Level– # Participants by

• Role (teacher/paraprofessional/administrator/other)• Grade level taught

Note: SCHOOL data for all participants will be collected annually through BHE online system

12


• Participant Tracking and Reporting

– Basic standardized demographic and educational descriptive data to be collected from all participants in all activities. (See enclosed sample participant survey)

– Form may be used as is, or incorporated into customized local survey

– Electronic versions of survey to be distributed via email following workshop (PDF and Word)

– Optional: Use of Participant Code to assist in participant tracking, linking to other local evaluation data

Note: PARTICIPANT data for every participant will be collected annually through BHE online system

13


TQ Program Reporting Requirements

• Annual report (multi-year projects) describing status, progress, milestones of project activities; includes evaluation progress report, with interim findings

• Final report (all projects) summarizing project activities/milestones and final evaluation report

• Evaluation reports (interim and final) to include project objectives, evaluation questions, methodology and results

• General template for organizing evaluation report will be provided to facilitate state-level meta-analysis

• Report to be submitted electronically

14

Standard Logic Model of Professional Development

PD and Related Support Activities

Growth in Participant

Skills / Knowledge

Improved Instruction

Improved Student

Outcomes

15

Planning for Project Evaluation - 1

Logic Model Step 1 – PD and Related Support Activities

Sample Evaluation Questions:

• To what extent have project activities been implemented as planned? What implementation challenges were encountered and how were they addressed?

• Who participated and to what extent? In workshops? In follow up support activities?

• To what extent were participant expectations/needs met?

• How did participants perceive the quality of the activities?

Sample Data Sources:

• Activity and participant tracking system

• Staff interviews

• Participant feedback – surveys, interviews, focus groups

16


Logic Model Step 2 – Growth in Participant Knowledge/Skills

Sample Evaluation Questions:• To what extent do participants achieve the stated learning

objectives of PD activities?• To what extent do participants retain or deepen their

understanding of concepts learned following PD? Do follow-up/support activities lead to enhanced understanding of concepts?

• Do participants experience other benefits/outcomes (e.g., self-confidence?

Sample Data Sources:• Pre-test/post-test of PD content (short-term outcomes) • Follow-up administration of test (longer-term outcomes)• Other indicators of mastery – e.g., course assessments• Pre/post survey of attitudes, beliefs

17


Logic Model Step 3 – Improved Instruction


• To what extent do participants’ instructional practices change as a result of participation?

• To what extent do participants incorporate material gained through PD into curriculum?

• What challenges do participants encounter as they attempt to implement new skills, contents, approaches? How are challenges addressed?


• Interviews, focus groups

• Participant survey of self-reported changes

• Classroom observation

18


Logic Model Step 4 – Improved Student Outcomes


• To what extent do students of participants achieve improved outcomes related to observed changes in instructional practices? Affective (attitudinal)? Cognitive?


• Standardized assessments

• Classroom assessments

• Teacher perceptions (survey)

19

Moving Forward

Our role is to work with you and your evaluator (internal or external) to implement an evaluation plan that:

• Includes both formative and summative evaluation questions• Tracks activity, school, and participant data required for annual

reporting• Has data collection and analysis organized around a basic logic

model of professional development• Includes appropriate outcomes that measure targeted phases of the

logic model

20

Next steps . . .

• Projects complete evaluation plan summaries and submit to TQ Evaluation Team (today or via email within 1 week)

• TQ Evaluation Team reviews evaluation plans

• We will be in contact with you with feedback, questions, or possible suggestions – by phone, email, or on-site work session

• Send us updated evaluation plan summaries as you further develop and/or revise you plan (email attachment)

• Contact us with any questions or concerns

Documents

Project Evaluation Workshop