Research panel

TIMELESS LEARNINGPOLICY & PRACTICE

JD HOYEPresident National Academy Foundation

THEORY OF CHANGE

Academy Outcomes

Academy Outcomes

Academy

Students

Academy

Students

Local Partners & Advisory

Board

Local Partners & Advisory

Board

NAF Funders & Partners

NAF Funders & Partners

Increased Capacity to Implement NAF Model

Increased Capacity to Implement NAF Model

Local Education Partners

Local Education Partners

NAF Supports

NAF Supports

Student Outcomes

Student Outcomes

Community

Benefits

Community

Benefits

PANEL DISCUSSION

Research, Data and Evaluation

Moving Toward a Performance

Measurement System

Where We Have Been:

•Creating Standards for High Fidelity

•Essential Data to Collect

•Including all Members of the Network in the Discussion

•Rich Dialogue Around Performance Measurement

Where We Are Going:

•Clarifying What it Means to Be a NAF Academy

•Streamline Data Collection and Reporting

•Self-Assessment Tool revised based on feedback from Academies and additional input from experts

•Increase Opportunities For Teachers, Administrators, Parents, and Students

Performance Measurement

Performance Measurement System

• Clearly Articulated Standards & Benchmarks

• System for Collecting Data For Those Indicators & Benchmarks

• System for Analyzing Reporting, • Process for Using Indicator Data to

improve student performance

Affiliate Program

Member Academy

Leader Academy

Distinguished Academy

Components of the NAF Performance Measurement

System

Self-Assessment & Planning Tool

Student-level Data (ConnectEDU)

Stakeholders Surveys

Creating a Culture of Inquiry

Context for Performance Measurement System

• MDRC Random Assignment Study Most Definitive

• No Measurable Effects– Academic Achievement– Graduation Rates– Post-Secondary Education Outcomes

• Positive Effects On– Student Engagement– Career Related/Work-Based Learning

Experiences– Long Term Earnings (Substantial Effects)

Key Findings From Previous Research

• Increased Premium on Rigorous Evaluation and Reliable Estimates of Program Effects

• Increased Attention to Career and College Readiness and Overlap Between the Two

• Early Indicators of High School Success

• Emphasis on Data-Driven Decision-Making

Key Developments in Research and Evaluation

• Impacts on Academic Outcomes, Post-Secondary Education

• Key Drivers of Positive Impacts• Understanding Cross-Site Variation in

Impacts• NAF Specific Impacts

Need For Additional Research

• What does it mean to be a NAF Academy?• What Experiences Should Students Have?• What Outcomes Should Fully Implemented

NAF Academies Achieve?• What Data Are Required– To Evaluate NAF? – To Monitor Implementation?– To Maximize program Effectiveness?

Key Research & Evaluation Questions for NAF

Affiliate Program Member Academy Leader Academy Distinguished Academy


System





Guiding Principles:

•Clearly articulate a tangible rubric on what defines a NAF Academy

•Demonstrate emerging and best practice along the rubric

•Develop a tool that allows Academies to reflect on their performance against the rubric

•Encourage and inform an exchange of best practices and lessons learned among Academies and between NAF and the Academies

Development Process:

•Conducted three tier process:–280 academies reviewed–Variance review, 5 in each region

–Visit 15 sites

•Site visits conducted by a team of experts including MDRC, National Career Academy Coalition, and members of the Data Tracking Steering Committee to the 15 Academies that scored in the Leader/ Distinguished category or are AOEs

•Self-Assessment Tool revised based on feedback from Academies and additional input from experts

Self-Assessment and Planning Tool Overview

• Data Collection & Revision

• Professional Development• Academy Leadership

The Self-Assessment Tool

has 4 key sections

Academy Development

Advisory Board

Curriculum & Instruction

Work-Based Learning & Internships

Each Section outlines the core concepts in

the rubric…

…and drives each concept to a level of

tangible detail

• Recruitment• Length of Program

• Provides a WBL Program• Internship Completion• Students’ Career Goals

• Integration

• Instructional Practices

• Instructional Supports

• Board Membership

• Board Function

• Board Supports

• Cohort Scheduling• Academy Decision-

Making

•Cohort scheduling ensures that academy students experience NAF academy courses plus a minimum of two core subject at each grade level.

•Cohort scheduling ensures that students receive instruction in a personalized environment where every child is known by a caring adult.

•Cohort scheduling assumes a schedule where there is weekly common planning time for the academy team of at least three staff, so that integrated learning, student supports and individualized student assessment can occur.



System





Why Student Data?

Demographic At-RiskPoverty

DisciplineAttendance

Program NAF AcademyModel

ServicesSupport

LearningStudent

OutcomesGPASAT

High School Exit

Why ConnectEDU?

Why ConnectEDU?

Why ConnectEDU?



System





• Provide NAF periodic snapshots of how academies and students are

performing.

• Provide individual academies information to help inform their own growth and improvement.

• Fill in the gaps in the performance measurement system with data that can not be obtained using other data sources.

Why Do Surveys?

• Students• Teachers• Parents• Employers

Whom To Survey?

1. Focus on a student survey to measure behavior, attitudes and outcomes for which we don’t have measures in school records.

2. Select/create instrument or instruments.

3. Select a sample of academies that represent the network.

4. Survey all students in the sampled academies.

5. Conduct in-class or on-line student survey every two years to measure changes.

Surveys: Next Steps



System





• Ensure Data Quality

• Organize for collaborative work

• Build assessment literacy and skill in data interpretation

Preparation

• Create data overview

• Dig into student data

Inquiry

• Examine policy and procedures and instruction

• Plan to assess programs

• Acting and assessing

Move Beyond Data Analysis



System





Business

Research panel