Do You See What I See? - New York University · 2014-06-30 · Do You See What I See? The Impact of School Accountability on Perceptions of the School Environment Many districts are

Emilyn Ruble Whitesell

IESP Summer Seminar

June 20141

Do You See What I See?The Impact of School Accountability on

Perceptions of the School Environment

Many districts are recognizing parent, teacher, and student perceptions

as important dimensions of school quality (e.g., NYC, Chicago, LA)

Stakeholder consensus may also be an important aspect of school quality

May reflect other positive school characteristics (proxy?)

May help facilitate school improvement efforts

This is one of few studies to compare the perceptions of stakeholders

within schools

Between-group consensus

Within-group consensus

2

Introduction

1. Background and motivation

2. Data and sample

3. Consensus measures and descriptive statistics

4. Relationship between consensus and passing rates

5. Impact estimate: effect of accountability on consensus

6. Mechanism (?)

7. Conclusion and next steps

3

Outline of presentation

Why might consensus matter for schools?

May contribute to success in school reforms (e.g., distributed leadership, CSRs)

Associated with stakeholder satisfaction and performance (Griffith, 2000; O’Day, 1984)

What school factors are associated with consensus?

Consensus is lower in schools with recent principal changes (Griffith, 1999)

Student and parent consensus (within groups) is higher in schools with homogenous

student bodies and lower student turnover (Griffith, 2000)

Most research focuses on principal/teacher perceptions of school leadership:

Consensus is positively associated with more effective leadership, stronger academic

climates, higher teacher morale, lower teacher turnover, and higher student achievement

(Atwater & Yammarino, 1992; Atwater et al., 1998; Covay Minor et al., 2014; Goldring et

al., 2012; Urick, 2012) 4

What do we know about consensus?

Multiple dimensions of school climate, but high expectations and discipline

are particularly important

Long line of research on the importance of discipline and expectations,

relating these domains to student achievement

“Academic press,” effective schools research, and a tradeoff with community?:

(e.g., Coleman, Hoffer, & Kilgore, 1982; Purkey & Smith, 1983; Rosenholtz, 1985; Shouse, 1996;

Phillips, 1997)

“No excuses” charter schools, with zero-tolerance discipline and high expectations:

(e.g., Carter, 2000; Whitman, 2003; Thernstrom & Thernstrom, 2008; Wilson, 2008, Angrist et al.,

2010; Abdulkadiroglu et al., 2011; Angrist, Pathak, & Walters, 2011; Fryer, 2011; Dobbie & Fryer,

2011, 2013)

5

Focus on views of school expectations and discipline

Prior research finds school accountability increases student achievement

(e.g., Carnoy & Loeb, 2002; Hanushek & Raymond, 2005; Figlio & Rouse, 2006;

Chiang, 2009; Winters, Trivitt, & Greene, 2010; Rockoff & Turner, 2010;

Winters & Cowen, 2012)

Consensus is a potential mechanism through which accountability affects

test scores

How might accountability affect perceptions and consensus?

Information / stigma

Changes in school practices

6

Conceptual framework

NYC Learning Environment Surveys

Responses from parents, teachers, and students (only in grades 6 -12)

High coverage and nearly one million respondents per year

NYC School Progress Reports

Letter grades (A-F) assigned to schools

Grades based on student progress (60%), student performance (25%), and the school

environment (15%)

New York State School Report Cards

School factors: size, grade span, teacher turnover, student attendance

Average teacher characteristics: experience, qualifications

Average student characteristics: performance, race, poverty, educational needs 7

New York City data, 2007-2012

8

Including schools with at least 10 survey responses in

the relevant stakeholder group

0

200

400

600

800

1000

1200

1400

1600

1800

2007 2008 2009 2010 2011 2012

Total Schools

>=10 Parent responses

>=10 Teacher responses

>=10 Student responses

0

100

200

300

400

500

600

700

Ele

m

Mid

dle K8

Hig

h

Ele

m

Mid

dle K8

Hig

h

Ele

m

Mid

dle K8

Hig

h

Ele

m

Mid

dle K8

Hig

h

Total Schools >=10 Parent

responses

>=10 Teacher

responses

>=10

Student

responses

Total schools by school type and N of survey responses, 2012Count of schools by N of survey responses, 2007-2012

9

Domain Teacher Parent StudentSchool safety I am safe at my school. My child is safe at school. I am safe in my classes.

I am safe in the hallways, bathrooms,

and locker rooms.

I am safe on school property outside

my school building.

Bullying Students in my school are often threatened or bullied.

Students threaten or bully other students.

Students threaten or bully other students

at school.

Discipline Order and discipline are maintained at my school.

Discipline is enforced fairly at my child’s school.

Discipline in my school is fair.

Teacher respect for students

Adults at my school are often disrespectful to students.

School staff are disrespectful to students.

Teachers in my school treat students with

respect.

Cleanliness The school is kept clean. My child’s school is clean. My school is kept clean.

Expectations My school sets high

standards for student

learning.

My school sets high

standards for student

work in their classes.

The school has high

expectations for my child.

I need to work hard to get good

grades at my school.

My teachers expect me to continue

my education after high school.

Significant overlap in questions across stakeholder groups

Others: Gang activity, resources for conflict resolution, course / activity offerings, goal setting, academic counseling, parent communication

10

Average stakeholder perceptions, 2007

0

0.5

1

1.5

2

2.5

3

3.5

4

Safety Gangs Bullying Discipline Respect Cleanliness Offerings Expectations

Parent Teacher Student

Average stakeholder perception across all schools, 2007

11

Perceptions of school expectations over time

50%

55%

60%

65%

70%

75%

80%

85%

90%

95%

100%

2.0

2.2

2.4

2.6

2.8

3.0

3.2

3.4

3.6

3.8

4.0

2007 2008 2009 2010 2011 2012

Parents Average Rating Teachers Average Rating Students Average Rating

Parents % Favorable Teachers % Favorable Students % Favorable

Average perceptions of school expectations, across all schools

12

Perceptions of school discipline over time

50%

55%

60%

65%

70%

75%

80%

85%

90%

95%

100%

2.0

2.2

2.4

2.6

2.8

3.0

3.2

3.4

3.6

3.8

4.0

2007 2008 2009 2010 2011 2012

Parents Average Rating Teachers Average Rating Students Average Rating

Parents % Favorable Teachers % Favorable Students % Favorable

Average perceptions of school discipline, across all schools

Survey responses range from 1-4, with 4 reflecting more favorable views

Difference between average perceptions of different stakeholder groups,

within schools – “mismatch” variable

Parent-teacher difference

Parent-student difference

Teacher-student difference

In regressions, use absolute value of difference, so a larger value indicates

less consensus (greater mismatch)

Calculated separately for each domain and also averaged across domains13

Measuring between-group consensus

14

Between-group consensus varies by domain

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

Safety Gangs Bullying Discipline Respect Cleanliness Offerings Expetations

Parent-Teacher Parent-Student Teacher-Student

Average difference in average stakeholder views, within schools, 2007

Note: Consensus values are only included if a school has at least 10 respondents from the relevant stakeholder group.

Responses range from 1-4, with 4 reflecting more favorable views

Standard deviation of stakeholder perceptions within schools (variation)

Teacher variation

Parent variation

Student variation

Larger values reflect greater variation (less consensus)

Calculated separately for each domain and also averaged across domains to

create an index measure

15

Measuring within-group consensus

16

Relatively high variation in student perceptions

Average standard deviation in stakeholder views, within schools, 2007


0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Safety Gangs Bullying Discipline Respect Cleanliness Offerings Expectations

Parent Teacher Student

17

Relationship between within-group variation and

average group perceptions, 2007

12

34

Ave

rag

e s

takeh

old

er

respo

nse

0 .5 1 1.5Standard deviation in stakeholder responses

Teacher Parent Student

Discipline Expectations

12

34

Ave

rag

e s

takeh

old

er

respo

nse

s

1 1.5Standard deviation in stakeholder responses



18

Relationship between within-group variation and

standardized test passing rates0

.2.4

.6.8

1

Pro

ficie

ncy r

ate

- m

ath

.2 .4 .6 .8 1 1.2Standard deviation in stakeholder responses (index)


0.2

.4.6

.81

Pro

ficie

ncy r

ate

- E

LA

.2 .4 .6 .8 1 1.2Standard deviation in stakeholder responses (index)



19

Relationship between between-group differences and

standardized test passing rates

0.2

.4.6

.81

Pro

ficie

ncy r

ate

- E

LA

0 .2 .4 .6 .8 1Difference in average stakeholder responses (index)


0.2

.4.6

.81

Pro

ficie

ncy r

ate

- m

ath

0 .2 .4 .6 .8 1Difference in average stakeholder responese (index)



20

Estimating the relationship between

consensus and exam passing rates

(1)PASSINGst = β0 + CONSENSUSst’ β1 + Sst’β2 + Tst’β3 + Xst’ β4 + αs + φt + εst

PASSING is a passing rate (math or ELA) on standardized tests

CONSENSUS is a vector of consensus measures for school s in year t

Within-group standard deviation

Between-group difference

S – vector of school characteristics: teacher turnover rate, student attendance rate, school size

T – vector of teacher characteristics: % with fewer than 3 years experience, % with Master’s degrees

X – vector of student characteristics: poverty, race, LEP

School (αs) and year (φt ) effects

21

Relationship between consensus about school

expectations and ELA passing rate

Outcome: ELA passing rate (1) (2) (3) (4) (5) (6) (7) (8) (9)

Within-group SD

Parent -0.078*** -0.073*** -0.076***

(0.015) (0.018) (0.019)

Teacher -0.018* -0.000 0.004

(0.009) (0.011) (0.012)

Student -0.044** -0.033* -0.031

(0.019) (0.019) (0.020)


Parent-Teacher -0.009 0.001 -0.002

(0.007) (0.010) (0.011)

Parent-Student 0.002 0.008 0.016

(0.012) (0.013) (0.013)

Teacher-Student -0.015** -0.017* -0.013

(0.008) (0.010) (0.010)

Observations 8,250 8,234 4,634 4,627 4,629 8,232 4,632 4,627 4,627

R-squared 0.943 0.943 0.952 0.953 0.952 0.943 0.952 0.952 0.953

Note: All models include school and year effects. Models also control for student demographics, student attendance rate, teacher experience and education.

Standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1

22

Relationship between consensus about school

discipline and ELA passing rate

Outcome: ELA passing rate (1) (2) (3) (4) (5) (6) (7) (8) (9)

Within-group SD

Parent -0.041*** -0.050*** -0.064***

(0.014) (0.016) (0.017)

Teacher -0.011 -0.001 0.004

(0.007) (0.010) (0.010)

Student -0.017 -0.008 0.017

(0.021) (0.021) (0.023)


Parent-Teacher -0.019*** -0.013*** -0.013***

(0.004) (0.005) (0.005)

Parent-Student -0.015* -0.011 -0.022**

(0.008) (0.008) (0.009)

Teacher-Student -0.007 -0.002 0.000

(0.005) (0.006) (0.006)

Observations 8,250 8,234 4,634 4,627 4,629 8,232 4,632 4,627 4,627

R-squared 0.943 0.943 0.952 0.952 0.952 0.943 0.952 0.952 0.953

Note: All models include school and year effects. Models also control for student demographics, student attendance rate, teacher experience and education.

Standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1

23!!!aA

Effect of accountability on consensus

(2) CONSENSUSst = β0 + β1 ACCTst-1+ Sstβ2 + Tstβ3 + Xstβ4 + αs + ϴst + st

CONSENSUS is a consensus measure for school s in year t

ACCT is an accountability measure for school s in year t-1

Graded

Letter grade

Grade increase / grade decrease

School effects

School-specific time trends in lieu of year effects

24

Effect of accountability on standard deviation in

stakeholder responses about expectations, 2007-2012Parent Teacher Student

Outcome: SD in responses (1) (2) (3) (4) (5) (6) (7) (8) (9)

Graded last year -0.057*** -0.003 0.070***

(0.002) (0.004) (0.004)

Last year –A -0.062*** -0.004 0.074***

(0.003) (0.005) (0.004)

Last year – B -0.059*** -0.005 0.070***

(0.003) (0.005) (0.004)

Last year – C -0.047*** -0.001 0.065***

(0.003) (0.005) (0.005)

Last year – D -0.053*** 0.001 0.063***

(0.005) (0.008) (0.007)

Last year – F -0.044*** -0.001 0.066***

(0.007) (0.012) (0.010)

Lag grade increase 0.004* 0.008** 0.005

(0.003) (0.004) (0.004)

Lag grade decrease 0.005* 0.002 -0.006

(0.003) (0.004) (0.004)

Observations 8973 8973 8973 8304 8304 8304 5092 5092 5092

Adj. R-squared 0.665 0.667 0.634 0.620 0.620 0.620 0.572 0.573 0.524

Note: All models include school effects and school-specific time trends. Models also include student demographics (poverty, race), student attendance rate,

school enrollment, and teacher characteristics (education and experience). Schools only included if they have at least 10 responses from relevant stakeholder groups.

25

Effect of accountability on standard deviation in

stakeholder responses about discipline, 2007-2012Parent Teacher Student

Outcome: SD in responses (1) (2) (3) (4) (5) (6) (7) (8) (9)

Graded last year -0.024*** -0.011** 0.033***

(0.003) (0.005) (0.003)

Last year –A -0.026*** -0.013** 0.033***

(0.003) (0.006) (0.004)

Last year – B -0.024*** -0.014** 0.034***

(0.003) (0.006) (0.004)

Last year – C -0.019*** -0.012* 0.035***

(0.004) (0.006) (0.004)

Last year – D -0.030*** 0.003 0.028***

(0.005) (0.010) (0.006)

Last year – F -0.019** 0.005 0.029***

(0.008) (0.015) (0.009)

Lag grade increase 0.003 0.012** 0.001

(0.003) (0.005) (0.003)

Lag grade decrease -0.003 -0.000 -0.005

(0.003) (0.005) (0.003)

Observations 8973 8973 8973 8304 8304 8301 5092 5092 5092

Adj. R-squared 0.593 0.593 0.587 0.612 0.612 0.612 0.616 0.616 0.604



26

Effect of accountability on difference in average

stakeholder responses about expectations, 2007-2012Parent-Teacher Parent-Student Teacher-Student

Outcome: Difference (AV) (1) (2) (3) (4) (5) (6) (7) (8) (9)

Graded last year -0.026*** -0.124*** -0.139***

(0.005) (0.006) (0.009)

Last year –A -0.017*** -0.121*** -0.119***

(0.006) (0.007) (0.011)

Last year – B -0.025*** -0.131*** -0.139***

(0.006) (0.006) (0.010)

Last year – C -0.025*** -0.110*** -0.141***

(0.007) (0.007) (0.011)

Last year – D -0.056*** -0.133*** -0.179***

(0.010) (0.010) (0.017)

Last year – F -0.045*** -0.117*** -0.205***

(0.016) (0.016) (0.025)

Lag grade increase 0.008 0.009* 0.023***

(0.005) (0.005) (0.008)

Lag grade decrease -0.010* 0.006 -0.002

(0.005) (0.006) (0.009)

Observations 8293 8293 8293 5052 5052 5052 4638 4638 4638

Adj. R-squared 0.487 0.488 0.485 0.576 0.578 0.507 0.589 0.592 0.557



27

Effect of accountability on difference in average

stakeholder responses about discipline, 2007-2012Parent-Teacher Parent-Student Teacher-Student

Outcome: Difference (AV) (1) (2) (3) (4) (5) (6) (7) (8) (9)

Graded last year -0.104*** 0.044*** -0.001

(0.010) (0.008) (0.013)

Last year –A -0.080*** 0.055*** 0.011

(0.012) (0.010) (0.016)

Last year – B -0.100*** 0.048*** -0.014

(0.011) (0.009) (0.014)

Last year – C -0.106*** 0.027** 0.025

(0.013) (0.011) (0.016)

Last year – D -0.174*** 0.031** -0.009

(0.019) (0.016) (0.024)

Last year – F -0.194*** 0.027 -0.071**

(0.030) (0.024) (0.036)

Lag grade increase 0.018* -0.002 -0.009

(0.009) (0.008) (0.011)

Lag grade decrease 0.006 -0.003 -0.017

(0.010) (0.008) (0.012)

Observations 8293 8293 8293 5052 5052 5052 4638 4638 4638

Adj. R-squared 0.612 0.614 0.604 0.592 0.592 0.588 0.497 0.499 0.497



28

Translating the effect: percent change and effect size

Outcome: Expectations Outcome: Discipline

Native units % change Effect size Native units % change Effect size

Parent SD -0.057*** -8.77% -0.384 -0.024*** -3.52% -0.175

Teacher SD -0.003 - - -0.011** -1.54% -0.055

Student SD 0.070*** 10.69% 0.672 0.033*** 3.93% 0.321

Parent-Teacher diff -0.026*** -10.65% -0.137 -0.104*** -20.43% -0.258

Parent-Student diff -0.124*** -54.49% -0.642 0.044*** 9.12% 0.184

Teacher-Student diff -0.139*** -38.66% -0.502 -0.001 - -

To increase consensus, accountability must affect perceptions of different

stakeholders differently (in direction or magnitude)

Why might this happen – difference in salience of information / school changes?

Parent views may be more responsive to information, as they have less direct contact with the

school than teachers or students

Teacher and student views may be more responsive to school-level changes that result from

accountability pressure

The elasticity of stakeholder views with respect to accountability information

might vary for different school conditions

Views about “academic” factors may be more responsive accountability information if

stakeholders think grades reflect academic quality

29

How does accountability affect consensus?

30!!!aA

Effect of accountability on average perceptions

(3) PERCEPTIONst = β0 + β1 ACCTst-1+ Sstβ2 + Tstβ3 + Xstβ4 + αs + ϴst + st

PERCEPTION is an average perception measure (parents, teachers, or students) for school s in year t

ACCT is an accountability measure for school s in year t-1

Graded

Letter grade

Grade increase / grade decrease

School effects

School-specific time trends in lieu of year effects

31

Effect of accountability on average stakeholder

perceptions about school expectations, 2007-2012Parent Teacher Student

Outcome: Avg perception (1) (2) (3) (4) (5) (6) (7) (8) (9)

Graded last year 0.090*** 0.105*** -0.052***

(0.010) (0.015) (0.007)

Last year – A 0.095*** 0.078*** -0.057***

(0.011) (0.017) (0.008)

Last year – B 0.097*** 0.098*** -0.053***

(0.010) (0.016) (0.007)

Last year – C 0.074*** 0.117*** -0.048***

(0.012) (0.018) (0.008)

Last year – D 0.089*** 0.165*** -0.046***

(0.018) (0.027) (0.010)

Last year – F 0.058** 0.195*** -0.048***

(0.026) (0.052) (0.017)

Lag grade increase -0.002 -0.027** 0.002

(0.006) (0.011) (0.004)

Lag grade decrease 0.003 0.003 0.005

(0.007) (0.013) (0.005)

Observations 4627 4627 4627 4627 4627 4627 4627 4627 4627

Adj. R-squared 0.806 0.807 0.790 0.780 0.782 0.772 0.808 0.808 0.797


school enrollment, and teacher characteristics (education and experience). Schools only included if they have at least 10 responses from all stakeholder groups.

32

Effect of accountability on average stakeholder

perceptions about school discipline, 2007-2012Parent Teacher Student

Outcome: Avg perception (1) (2) (3) (4) (5) (6) (7) (8) (9)

Graded last year 0.087*** 0.185*** 0.048***

(0.011) (0.026) (0.012)

Last year –A 0.096*** 0.138*** 0.042***

(0.012) (0.029) (0.013)

Last year – B 0.088*** 0.180*** 0.048***

(0.012) (0.028) (0.012)

Last year – C 0.072*** 0.192*** 0.050***

(0.014) (0.031) (0.014)

Last year – D 0.090*** 0.285*** 0.067***

(0.019) (0.046) (0.019)

Last year – F 0.055** 0.350*** 0.041

(0.027) (0.092) (0.037)

Lag grade increase -0.006 -0.047*** -0.005

(0.007) (0.018) (0.009)

Lag grade decrease -0.000 -0.011 0.004

(0.008) (0.021) (0.010)

Observations 4627 4627 4627 4627 4627 4627 4627 4627 4627

Adj. R-squared 0.755 0.756 0.741 0.764 0.767 0.755 0.797 0.797 0.794


school enrollment, and teacher characteristics (education and experience). Schools only included if they have at least 10 responses from all stakeholder groups.

Stakeholder consensus is an overlooked but potentially important aspect

of school quality

In NYC, consensus measures are related to other positive school attributes

Favorable views of the environment

Math and ELA passing rates

The introduction of the letter grade school accountability system in NYC

seems to have increased consensus

Contributes to ongoing debates about measuring school quality and the

impact of accountability systems33

Discussion and policy implications

Is consensus always good? Maybe not... Interact consensus with average perceptions

Control for perceptions when estimating the relationship between consensus and outcomes

Continue to explore how accountability affects consensus – individual models Restrict to individuals who remain in school; lagged perceptions or individual FE

Consider teacher exit and principal turnover – as outcomes or controls

General modeling: What models do I believe? Worth looking at the impact of grades and grade changes, or just grading?

Estimate the impact of an F – regression discontinuity

Compare responses to letter grade and AYP information

Can I distinguish between information/stigma and changes in school practices? Hypothesis: parents respond to information, while teachers and students are more likely to respond to

changes in variables to measure school changes

Additional measures: resources (expenditures, pupil-teacher ratios); teacher perceptions of specific

practices (e.g., data use, professional development, leadership support )34

What I’m thinking about next…

[email protected]

The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305B080019 to New York

University. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.

Documents

Do You See What I See? - New York University · 2014-06-30 · Do You See What I See? The Impact of School Accountability on Perceptions of the School Environment Many districts are