360 Data on Scholars Skills, Attitudes, and Behaviors:

Measurement for Program Improvement

and Scholar Outcomes

Elizabeth BarrettDirector of EvaluationHigher Achievementebarrett@higherachievement.org202.567.2444

Agenda

• Higher Achievement and 360 Data• How do we analyze the data

– Collection and reporting– Analysis techniques

• Internal Use– Program improvement– Individual scholar support

• External Use– How to communicate this information effectively– 360 data as it relates to other outcomes

• Interactive Session

Higher Achievement

• Academic enrichment and support program for middle school children from underserved communities

• Rigor + Opportunities = Results• Culture of high expectations • 3 components:

– After-School Academy, Summer Academy and High School Placement

Higher Achievement

• Locations in– DC Metro (including Alexandra, VA)– Baltimore– Richmond, VA opening Summer 2011– Expanding to Pittsburgh, PA

• This summer we will be serving 700 scholars

Higher Achievement and 360 data

• Focus has been on grades and test scores

• Movement beyond these outcomes– 21st Century skills, soft skills, etc.– These skills are just as important as

hard skills• Aligning with our Culture of high

expectations– Spirit, excellence and respect– Higher Achievement’s movement

toward skills, attitudes and behaviors– High school preparation and beyond

Higher Achievement and 360 Data

• Gather data from– Scholars about themselves– Mentors or Summer Teachers about

each scholar– Parents about their child

• Collected bi-annually– 1 time for each After-School Academy

and Summer Academy

360 Survey

360 Survey

Collection and Reporting

• Customized database management system: ISIS– Electronically collects and stores data– Reporting feature allows data to be

manipulated in • Excel – qualitative responses• SPSS – quantitative analysis

Analysis Techniques

• We keep it simple– Averages – questions and categories– % Agree/Disagree– Regression Analysis

• Group Response: parent, mentor/summer teacher and scholar

• Across Academy Trends• Academy Trends

– Affiliate/Center/Individual

Example – Across Academy Trends

AFSA 08.09 SA09 AFSA 09.103.8

3.9

4

4.1

4.2

4.3

BehaviorsSkillsAttitudes

Example – Across Academy Trends

AFSA 08.09 SA09 AFSA 09.103.5

3.75

4

4.25

AXW1W4W6W7

Example – Affiliate Trends

Scholar

Parent

Mentor

Overall

3.5 3.75 4 4.25 4.5

AttitudesSkillsBehaviors

Example – Affiliate Trends

• Greatest Need1. Behavior changes2. Build confidence in public speaking

3. Increasing organization skills and preparedness

• Greatest Strength1. Attitudes/skills 2. Applies academic knowledge/skills3. Interacts respectfully with peers4. Enjoys Higher Achievement

Example – Affiliate/Center/Individual

Example – Affiliate/Center/Individual

Example - Regressions

Attitudes and skills…– Respect toward adults (A)– Confidence in academic ability (A)– Applies academic knowledge/skills (S)

….predict key behaviors– Confidence in public

speaking/presenting (B)– Interacting respectfully with peers (B)

Based on regression analyses

Internal Use – Program Improvement

• Identify and plan center activities based on strength and weakness areas (After-School Academy and Summer Academy planning)

• Comments home to families during Summer Academy (7th and 8th graders)

• Target scholars based on grades/behavior issues (6th and 7th graders)– Need for scholar to internalize

informationlead to change

Internal Use – Individualization

• Push for individualization through data– Individual scholar grade report

• Increased, decreased, maintained – Top/Bottom %

– Student Achievement Plans for struggling scholars• Goal setting, weekly progress reports

– 360 Individual Reports• Internal motivation to improve attitudes

and behaviors• Incorporate grades and test scores

External Data Use

• Identified 3 key outcomes to externally communicate:– Scholar interacts respectfully with

peers (behavior)– Scholar applies academic knowledge

and skills (skill)– Scholar confidence in academic abilities

(attitude)• Align with core model and culture

External Use - Communication

• Effective Communication– Consistency:

• Written and verbal communication across ALL staff

– Aligns with Key Messages:• Our research-based program challenges

middle school students to meet their full potential in three key areas: academics, social skills, and leadership. When students get the skills and support they need to invest in their own success, they discover that they can be scholars.

360 and Other Outcomes

• 360 data as it relates to other outcomes data– Compliments/back up other outcomes

data• Grades• Test scores• Attendance• High school placement

– Used to show impact on soft skills• Leadership• Confidence in academic ability• Social Skills

Example

• 65% showed significant improvement in math • 60% showed significant improvement in reading • Teacher surveys show:

– 73% of the students improved in homework completion

– 71% improved in overall academic performance • Parent surveys show:

– 92% are getting better grades on their report cards

• Student surveys show: – 68% feel better about their ability to read and

speak English – 81% are more comfortable solving math problems – 86% reported an improvement on homework

completion

Interactive Session• After-School Academy main components

– Study Hall– Community Meeting: all scholars and staff gather for

academic related activities– Mentoring with structured curricula in math, literature

and seminars• Scenario:

– During After-School Academy planning you need to compose an action plan for our Ward 1 Achievement Center based on:• Individual Scholar Grade Report• Center level 360 report

– Come up with actionable items that can be implemented for the whole center that addresses needs/weakness and/or celebrates successes/strengths.