Final msa nais merged presentation

INDEX Mission Skills Assessment

Mission Skills Assessment Assessing Character Traits in Middle School Students

Independent Schools Data Exchange www.indexgroups.org 617-413-8382

1


Mission Skills Assessment

Mission Skills Assessment: A Tool To Alter The Way Schools Think About Education

MSA measures six character strengths—Teamwork, Creativity, Ethics, Resilience, Curiosity, Time Management—that have proven essential for success in school and in life. The assessment gives each school a more scientific way to measure its curriculum’s success at meeting the goals outlined in its mission.

2


Who We Are

•  INDEX Elementary Schools: National benchmarking and best practices group of 28 Independent PS – 8/9 co-educational day schools with 400+ students.

•  Center for Academic and Workforce Readiness and Success, Educational Testing Services (ETS).

3


MSA Schools

Duke School (NC) Episcopal Day School (GA)

Far Hills Country Day School (NJ)

The Foote School (CT)

Grace-Saint Luke’s Episcopal School (TN)

Green Vale School (NY)

Greenwich Country Day School (CT)

The Lexington School (KY)

Marin Country Day School (CA)

New Canaan Country School (CT)

Old Trail School (OH) The Pike School (MA)

Rippowam Cisqua School (NY)

Rodeph Sholom School (NY)

Rumson Country Day School (NJ)

Saint Martin’s Episcopal School (GA)

Shady Hill School (MA)

Shore Country Day School (MA)

St. Patrick’s Episcopal School (DC)

Trinity Episcopal School (NC)

4


What is the Mission Skills Assessment

The MSA is a longitudinal assessment measuring and benchmarking student achievement and improvement in core mission skills.

•  Teamwork

•  Creativity

•  Ethics

•  Resilience

•  Curiosity

•  Time Management

5


Roadmap

• R&D • ETS • Constructs

• Cognitive Labs •  Pilot Test

•  Fall Test •  Spring Test •  Horizons • 1st Report

• Fall Test • 2nd Report

• Fall Test •  INDEX Schools • 3rd Report

6


The Assessment

•  Web-based. •  Student self-assessment

•  2 30-minute tests for approximately 60 minutes total. •  Assess once per year. •  SJT and other student-completed performance measures. •  6th, 7th, and 8th graders. •  The test is fun! There are no wrong answers.

•  Teacher-rater assessment •  Teacher rates each student individually

•  Outcome data (e.g. test scores, grades, absences, etc.) •  Institutional focus. No tracking of individual performance.

7

INDEX Mission Skills Assessment 8

Noncognitive Skills Research


What are Noncognitive skills?

•  Not traditionally “tested” by standardized assessments

•  Not new but newly important; labeled “21st century skills”

•  Essential capacities necessary for success in school and in life

•  Often taught implicitly, but can be taught explicitly

9


Attitudes & Beliefs

Social & Emotional Qualities

Habits & Processes Personality

Attitudes (evaluations)

Emotion Management

Time Management

Openness to Experience

Intelligence Theories

Test Anxiety Goal Setting Conscientiousness

Self-Efficacy Self-Regulation Test-taking Strategies

Extraversion

Self-Concept Coping with

Stress Study Skills Agreeableness

Confidence Teamwork Self-Monitoring Emotional Stability

Lipnevich, MacCann, & Roberts (2011)

10


Disposition Toward School

GPA Class Absences

Discipline Infirmary Visits

SSAT

.27 -.04 -.05 -.04

Conscientiousness .21 -.31 -.23 -.22

Teacher ratings (noncognitive)

.38 -.35 -.42 -.26

11


rho

Primary Secondary Tertiary

Conscientiousness .28 .21 .23

Openness .24 .12 .07

Agreeableness .30 .05 .06

Emotional Stability .20 .01 -.01

Extraversion .18 -.01 -.03

Cognitive ability .58 .24 .23

Poropat, A. (2009). Psychological Bulletin

Grades

12


Academic Success

Noncognitive skills correlate positively with achievement on NAEP and PISA.

Noncognitive factors predict college grades as strongly as cognitive variables do.

ETS, (2008); Poropat, (2009)

13


Life Trajectory

Early childhood interventions do not raise IQ, but improve noncognitive skills which affect education, employment, earnings, and crime.

Heckman et al. (2010)

14


Life Satisfaction

Noncognitive variables…have been demonstrated to predict happiness, health, marital satisfaction, and peer relationships.

Diener & Lucas (1999); Bogg & Roberts (2004); Watson, Hubbard, & Wiese, (2000); Jensen-Campbell et al. (2002)

15


Demographics

The benefits of higher noncognitive skills are demonstrable across IQ, socio-economic class, gender, and race.

Tough, Paul. How Children Succeed: Grit, Curiosity and the Hidden Power of Character. (2012)

16


Ethics

100%

Love of Learning

73%

Resilience

67%

Creativity

67% Teamwork

40%

Analysis of mission statements of 20 independent middle schools

17


Intellectual Interpersonal Intrapersonal

Knowledge and mastery of general principles

Appreciation for diversity

Social responsibility and citizenship

Continuous learning, intellectual interest and

curiosity

Leadership Physical and psychological health

Artistic and cultural appreciation

Interpersonal skills

Career orientation

Adaptability and life skills

Perseverance

Ethics and integrity

18

Analysis of university mission statements (Oswald et. al., 2004)


Per

cent

Rat

ing

“Ver

y Im

port

ant”

* ***

**

The Conference Board, Corporate Voices for Working Families, Partnership for 21st Century Working Skills, and the Society for Human Resource Management.

19


The Character Imperative

1) Research shows that these skills lead to positive educational and life outcomes;

2) Colleges, universities, and employers are seeking these skills in students and workers; and

3) We must demonstrate the value we add.

20


Guiding Children to Achieve their Full Potential

Students are provided with a distinct advantage if their schools excel in training noncognitive skills. Independent Schools can lead the way in designing innovative character assessments and intentional classroom strategies.

21


Constructs in MSA


Teamwork

Models of teamwork include

•  Cooperation

•  Influence

•  Conflict resolution

•  Guiding others

2nd most valued work attribute according to Are they Really Ready to Work? report

50

60

70

80

90

100

MathIT Written Com.

EthicsEnglish

Critical Thinking

Reading Comp

WritingLifelong Learning

Creativity

Leadership

Oral Com.

Teamwork

Work Ethic

In high school students, teacher reports of teamwork were correlated with grades across several subjects (Wang, MacCann, Zhuang, Liu, & Roberts, 2009)

23


Creativity

•  Originality, progressiveness, or imagination •  Dictionary.com

•  Tasks measuring creativity, “require examinees fairly quickly to think of…a series of responses fitting the requirements of the task….”

•  Carroll, 1993; p. 428

24


Creativity predicts graduate school outcomes beyond GRE verbal and mathematics scores (Frederickson & Ward, 1975; Bennett & Rock, 1995; Bennett & Rock, 1998)

Creativity

5th most valued work attribute according to Are they Really Ready to Work? report

50

60

70

80

90

100

MathIT Written Com.

EthicsEnglish

Critical Thinking

Reading Comp


Creativity

Leadership

Oral Com.

Teamwork

Work Ethic


Ethics

Defined in our study as cheating and concern for others

One of the most valued work attribute according to Are they Really Ready to Work? report

50

60

70

80

90

100

MathIT Written Com.

EthicsEnglish

Critical Thinking

Reading Comp


Creativity

Leadership

Oral Com.

Teamwork

Work Ethic

26


Resilience

•  A level of adaptability that allows individuals to survive and thrive in adverse conditions

•  Dent and Cameron (2003)

•  One way we measure resilience in the MSA is to measure core self-evaluation – or how capable, worthy, and effective a person feels.

27

INDEX Mission Skills Assessment Judge and Hurst (2007)

2002

Inc

ome

1979 Core Self Evaluation

28


Intrinsic Motivation / Love of Learning / Curiosity

•  What people will do without external inducement (Malone & Lepper, 1987)

•  In a study of 3rd – 8th graders, intrinsic motivation was positively correlated with:

•  Grades: r = .34

•  Standardized Test Scores = r = .27

•  However…

29


(Lepper, Corpus, & Iyengar, 2005)

30


Time Management

Determining one’s needs, setting goals to meet needs, and prioritizing and planning to meet goals (Lakein, 1973).

Planning Organization

Grades .21** .38**

Grades (6 months later)

.25** .42**

Liu, Rijmen, MacCann, & Roberts (2009)

31


How are we doing it?

32


Our GPS: Multi-Trait Multi-Method (MTMM) Design

“Triangulation” Measurement

Method True Resiliency

Self Ratings

Teacher Ratings

Alternative Method

True Resilience

The underlying view of measurement in the MTMM analysis is that to measure a theoretical construct, different measures, each with its own bias, are selected. Bias that is due to method effects is reduced through a triangulation process. Baron & Kashy, 1992, p. 170

Not directly observable

34


Item Types

35


Self-Report


Situational Judgment Test


Biographical Data


Creativity Performance Test


Teacher Ratings


Resilience: Representative Items

Teacher Ratings This student is easily discouraged.

Situational Judgment Test You are feeling stressed about the amount of homework that you have been given by your teacher(s). Below are some ways that you might think, feel, or act in this situation, right at the time that you feel stressed-out. Rate how often you do each activity when you feel stressed. How do you think, feel, or act when you are stressed from having too much homework to do? 1.  I take control and say to myself: I can do

this!

Resilience

Self-Report I overcome challenges and set backs.

41


What did we find?

42


Participation

•  Fall 2011 •  18 INDEX schools •  2,081 students

•  Spring 2012 (optional assessment) •  13 INDEX schools •  1,677 students

•  Fall 2012 •  20 INDEX schools •  > 2,600 students •  Data being analyzed

43


94

95

96

97

98

99

100

101

102

103

2011 2012 2011 2012 2011 2012 2011 2012 2011 2012 2011 2012

Creativity Ethics Intrinsic9Motivation Resilience Teamwork Time9Management

Average'Con

struct'Score

Construct

Average'Construct'Scores'By'Gender

Male

Female

44


The MSA Demonstrates Evidence of Reliability

45


Reliability

Construct!2011

Reliability!2012

Reliability!Time Management! 0.86! 0.86!Resilience! 0.84! 0.85!Intrinsic Motivation! 0.89! 0.90!Creativity! 0.88! 0.88!Teamwork! 0.77! 0.85!Ethics! 0.90! 0.85!

SAT Reliability

Math .91

Critical Reading .91

46


The MSA Demonstrates Evidence of Validity

47


The MSA Predicts

•  Grades

•  Teacher Ratings of Student Quality •  “How would you rate this student compared to other students in

your school?”

48


0

0.5

1

1.5

2

2.5

3

3.5

4

E - Bottom 20%

D - 20%-39% C - 40%-59% B - 60%-79% A - Top 20%

2012

Stu

dent

Qua

lity

Rat

ing

2011 Creativity

Creativity and Student Quality Rating

49

r = .56


How “Big” Are Our Correlations?

Examples from other research (Meyer et al, 2001)

Variable 1 Variable 2 Correlation Sugar Consumption Child’s Behavior .00

Aspirin Heart Attack Death .02

Parental Divorce Child Well-Being .09

Lead Exposure Child IQ .12

Ibuprofen Pain Reduction .14

Alcohol Aggressive Behavior

.23

Viagra Sexual Functioning .38

Gender Height .67

50


2 2.2 2.4 2.6 2.8

3 3.2 3.4 3.6 3.8

4

E - Bottom 20%

D - 20%-39% C - 40%-59% B - 60%-79% A - Top 20%

Spri

ng 2

012

GPA

2011 Time Management

Time Management and GPA by Grade

6

7

8

6th r = .29 7th r = .59 8th r = .58

51


The MSA Predicts

•  Student Well Being

• My#life#is#going#well#• My#life#is#just#right#• I#would#like#to#change#many#things#in#my#life#• I#wish#I#had#a#different#kind#of#life#• I#have#a#good#life#• I#have#what#I#want#in#life#• My#life#is#be;er#than#most#kids’#

52


4.1!4.2!4.3!4.4!4.5!4.6!4.7!4.8!4.9!5!

A!-!Top!20%! B!-!60%-79%! C!-!40%-59%! D!-!20%-39%! E!-!Bo7om!20%!

2012!W

ell!B

eing!

2011!Resilience!

Resilience!and!Well!Being!

r = .33

53


4.1

4.2

4.3

4.4

4.5

4.6

4.7

4.8

4.9

5

A - Top 20% B - 60%-79% C - 40%-59% D - 20%-39% E - Bottom 20%

2012

Wel

l Bei

ng

2011 Ethics

Ethics and Well Being

r = .31

54


There is initial evidence that the MSA predicts just as well as

standardized tests

55


0.50 0.46 0.45 0.44

0.38 0.31

0.16 0.15

0

0.1

0.2

0.3

0.4

0.5

0.6

Cor

rela

tion

wit

h St

uden

t Q

uali

ty

MSA and ERB Scores: Correlations with Student Quality Ratings

56


0.59 0.56 0.52 0.49

0.41

0.28 0.27 0.21

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Cor

rela

tion

wit

h G

PA

MSA and ERB Scores: Correlations with GPA

57


0.28

0.23 0.22 0.22 0.19

0.14

0.04 0.01

0

0.05

0.1

0.15

0.2

0.25

0.3

Cor

rela

tion

wit

h W

ell B

eing

MSA and ERB Scores: Correlations with Well Being

58


Current and Next Steps

•  Continual refinement of MSA assessment (adding interests)

•  Assessing underserved populations with MSA

•  Gathering interest from additional schools

59


Implications for Schools

60


Implications for Schools

•  Wide-ranging and deep:

•  Why?

•  What?

•  How?

61


Why? Mission and Vision

•  Reexamine school mission and how measured

•  Strength of mind and strength of character

•  Develop a Vision for Character

62


How?

Programs-intentional and well-designed

Evidence-based

Evaluation

High-quality training and

support

63


What? Programs

How can we teach these skills better and how will we know we are succeeding?

•  Deliberate, explicit and implicit instruction

•  Proven interventions

•  Performance Character Audit

•  Articulated, spiraled, integrated curriculum

•  Grade-level goals

•  Updated and analyzed curriculum maps

64


Programs

•  Social Emotional Learning, advisories, personal awareness, and co-curricula

•  Shared lexicon, common language-Glossary of Terms for Character

•  Best Practices-follow through, evaluation, refinement

65


What? Training and Support

•  Ongoing education–school leaders, faculty, parents

•  Shared vision, vocabulary, and resources

•  Participation in regional and national educator character summits

•  Ongoing partnerships and dialogue

66


What? Evidence-based Evaluation

•  Updated Progress Reports

•  Parent and student conferences

•  Faculty and parent education summits

•  Greater emphasis: Self-reflection and meta-cognition

•  Surveys

67


Evidence-based Evaluation

•  Integrated assessments

•  Participation in new assessments (MSA)

•  Benchmarking

•  Data for national accreditation and new criterion

68


Measuring what we value:

Criterion 13:

The Standards require a school to provide evidence of a thoughtful process, respectful of its mission, for the collection and use in school decision making of data (both external and internal) about student learning.

69


Improvement Phases

Envision Implement Evaluate and Refine

Commitment tied to mission

Needs and resources assessment

Professional Development & Partnerships

Create clear vision

Action Plans, integration, and enhancements

Communication and engagement

Build engagement

Evidence-based strategies and programs

•  What is working? •  How do we know? •  How can we improve

teaching and learning?

70


Community Responses

•  Excitement!

•  Shared understanding

•  Partnership

•  Shared commitment to studying results and asking the next questions…

71


Look to your mission, look to your alumni, look to your purpose and your school’s view of what our society needs the most: creative and innovative thinkers, people of fine judgment, leaders, collaborators, and…then select or design the instruments which will help you make these predictions.

Martin, Jonathan. Lessons from Law School Admissions: More from USC Attributes that Matter, 21K12 Blog, January 23, 2013.

Your School, Your Mission

72


Thank you! Questions? Comments?

FOR MORE INFORMATION, PLEASE CONTACT:

INDEX INDEPENDENT SCHOOL DATA EXCHANGE

[email protected] WWW.INDEXGROUPS.ORG

Chuck Baldecchi Head of School The Lexington School (KY) [email protected] Tim Bazemore Head of School New Canaan Country School (CT) [email protected] Jeremy Burrus Research Scientist ETS [email protected]

Dave Michelman Head of School Duke School (NC) [email protected] Ned Murray Head of School Episcopal Day School (GA) [email protected] Jennifer Phillips Director of Educational Advancement Far Hills Country Day School (NJ) [email protected]

Lisa Pullman Executive Director, INDEX [email protected] Rich Roberts Managing Principal Research Scientist ETS [email protected]

MSA Steering Committee:

73

Documents

Final msa nais merged presentation