Upload
shanna-carpenter
View
216
Download
0
Tags:
Embed Size (px)
Citation preview
THEORY OF CHANGE
Academy Outcomes
Academy Outcomes
Academy
Students
Academy
Students
Local Partners & Advisory
Board
Local Partners & Advisory
Board
NAF Funders & Partners
NAF Funders & Partners
Increased Capacity to Implement NAF Model
Increased Capacity to Implement NAF Model
Local Education Partners
Local Education Partners
NAF Supports
NAF Supports
Student Outcomes
Student Outcomes
Community
Benefits
Community
Benefits
Where We Have Been:
•Creating Standards for High Fidelity
•Essential Data to Collect
•Including all Members of the Network in the Discussion
•Rich Dialogue Around Performance Measurement
Where We Are Going:
•Clarifying What it Means to Be a NAF Academy
•Streamline Data Collection and Reporting
•Self-Assessment Tool revised based on feedback from Academies and additional input from experts
•Increase Opportunities For Teachers, Administrators, Parents, and Students
Performance Measurement
Performance Measurement System
• Clearly Articulated Standards & Benchmarks
• System for Collecting Data For Those Indicators & Benchmarks
• System for Analyzing Reporting, • Process for Using Indicator Data to
improve student performance
Components of the NAF Performance Measurement
System
Self-Assessment & Planning Tool
Student-level Data (ConnectEDU)
Stakeholders Surveys
• MDRC Random Assignment Study Most Definitive• No Measurable Effects
– Academic Achievement– Graduation Rates– Post-Secondary Education Outcomes
• Positive Effects On– Student Engagement– Career Related/Work-Based Learning Experiences– Long Term Earnings (Substantial Effects)
Key Findings From Previous Research
• Increased Premium on Rigorous Evaluation and Reliable Estimates of Program Effects
• Increased Attention to Career and College Readiness and Overlap Between the Two
• Early Indicators of High School Success• Emphasis on Data-Driven Decision-Making
Key Developments in Research and Evaluation
• Impacts on Academic Outcomes, Post-Secondary Education
• Key Drivers of Positive Impacts• Understanding Cross-Site Variation in
Impacts• NAF Specific Impacts
Need For Additional Research
• What does it mean to be a NAF Academy?
• What Experiences Should Students Have?• What Outcomes Should Fully
Implemented NAF Academies Achieve?• What Data Are Required– To Evaluate NAF? – To Monitor Implementation?– To Maximize program Effectiveness?
Key Research & Evaluation Questions for NAF
Components of the NAF Performance Measurement
System
Self-Assessment & Planning Tool
Student-level Data (ConnectEDU)
Stakeholders Surveys
Guiding Principles:
•Clearly articulate a tangible rubric on what defines a NAF Academy
•Demonstrate emerging and best practice along the rubric
•Develop a tool that allows Academies to reflect on their performance against the rubric
•Encourage and inform an exchange of best practices and lessons learned among Academies and between NAF and the Academies
Development Process:
•Conducted three tier process:–280 academies reviewed–Variance review, 5 in each region
–Visit 15 sites
•Site visits conducted by a team of experts including MDRC, National Career Academy Coalition, and members of the Data Tracking Steering Committee to the 15 Academies that scored in the Leader/ Distinguished category or are AOEs
•Self-Assessment Tool revised based on feedback from Academies and additional input from experts
Self-Assessment and Planning Tool Overview
• Data Collection & Revision
• Professional Development• Academy Leadership
The Self-Assessment Tool
has 4 key sections
Academy Development
Advisory Board
Curriculum & Instruction
Work-Based Learning & Internships
Each Section outlines the core concepts in
the rubric…
…and drives each concept to a level of
tangible detail
• Recruitment• Length of Program
• Provides a WBL Program• Internship Completion• Students’ Career Goals
• Integration
• Instructional Practices
• Instructional Supports
• Board Membership
• Board Function
• Board Supports
• Cohort Scheduling• Academy Decision-
Making
•Cohort scheduling ensures that academy students experience NAF academy courses plus a minimum of two core subject at each grade level.
•Cohort scheduling ensures that students receive instruction in a personalized environment where every child is known by a caring adult.
•Cohort scheduling assumes a schedule where there is weekly common planning time for the academy team of at least three staff, so that integrated learning, student supports and individualized student assessment can occur.
Components of the NAF Performance Measurement
System
Self-Assessment & Planning Tool
Student-level Data (ConnectEDU)
Stakeholders Surveys
Components of the NAF Performance Measurement
System
Self-Assessment & Planning Tool
Student-level Data (ConnectEDU)
Stakeholders Surveys
• Provide NAF periodic snapshots of how academies and students are
performing.
• Provide individual academies information to help inform their own growth and improvement.
• Fill in the gaps in the performance measurement system with data that can not be obtained using other data sources.
Why Do Surveys?
1. Focus on a student survey to measure behavior, attitudes and outcomes for which we don’t have measures in school records.
2. Select/create instrument or instruments.
3. Select a sample of academies that represent the network.
4. Survey all students in the sampled academies.
5. Conduct in-class or on-line student survey every two years to measure changes.
Surveys: Next Steps
Components of the NAF Performance Measurement
System
Self-Assessment & Planning Tool
Student-level Data (ConnectEDU)
Stakeholders Surveys
• Ensure Data Quality
• Organize for collaborative work
• Build assessment literacy and skill in data interpretation
Preparation
• Examine policy and procedures and instruction
• Plan to assess programs
• Acting and assessing
Move Beyond Data Analysis