Assessment, Accountability and Adaptation April 13, 2006

1

Assessment, Accountability and Adaptation

April 13, 2006

copies of this presentation can be found atwww.business.duq.edu/faculty/davies

2

Education is a product.

Economics provides tools for adequately addressing assessment, accountability, and adaptation, but to use the tools, we must first admit that education, while occupying a place of great importance, is fundamentally a product.

3

How does the market judge products, hold producers accountable, and adapt to consumers’ needs?

Assessment

AccountabilityAdaptation

4

The cycle begins with the consumer who develops a mental list of important attributes and a acceptable tradeoffs. Together, these form the basis of the consumer’s assessment of the product.

Assessment


•Attributes

•Tradeoffs

5

Consider grocery stores…

The Consumer Has a List of Important Attributes

Store is located in a safe and convenient location.

The produce is fresh.

There is a large selection of brands.

The store has a deli.

Prices are reasonable.

The Consumer Is Aware of Acceptable Tradeoffs

Convenient location is more important than fresh produce.

Lower price is more important than having a deli.

6

The consumer assesses the performance of the store by comparing the store’s attributes to the consumer’s list of important attributes and acceptable tradeoffs.

Assessment

•Attributes

•Tradeoffs

7

The consumer holds the store accountable via repeat purchase. If the store’s attributes are acceptable to consumers, the store’s sales rise. If not, the store’s sales fall.

Assessment

Accountability

•Attributes

•Tradeoffs

8

If sales fall, the store will mimic and/or improve upon attributes of stores with rising sales. If the store’s sales rise, competing stores will mimic and/or improve upon this store’s sales. This is adaptation.

Assessment


•Attributes

•Tradeoffs

9

The consumer now re-assesses the store in light of the store’s new attributes and any changes in the consumer’s attributes / tradeoffs. This is outcomes assessment.

Assessment


•Attributes

•Tradeoffs

10

In the market place, what might cause the AAA cycle to break down?

•Attributes

•Tradeoffs Assessment

Adaptation Accountability

11

Break in the Cycle #1: Consumer is unidentified

•Attributes

•Tradeoffs

Assessment


If the consumer is unidentified, questions arise:

1. What attributes are important?

2. What tradeoffs are acceptable?

With an unidentified consumer:

Appropriate assessment becomes unclear.

Source of accountability becomes unclear.

Appropriate adaptation becomes unclear.

12

Assessment


Break in the Cycle #2: Accountability not Possible

1. Repeat purchase – the consumer must have a need to purchase again so as to signal pleasure;

2. Mobility – the consumer must be able to purchase elsewhere so as to signal displeasure;

3. Short-run assessment – the consumer must be able to assess product quality prior to the repeat purchase decision.

If accountability is not possible

Appropriate adaptation becomes unclear.

Incentive for adaptation disappears.

Possibility of accountability requires:

•Attributes

•Tradeoffs

13

How does the AAA cycle play out for primary and secondary education?

Assessment

Adaptation Accountability

•Attributes

•Tradeoffs

First: Can we identify the consumer? Second: On what is assessment

based?

Third: Is accountability possible?

Fourth: What adaptations result?

14

First: Can We Identify the Consumer?

• Student?Attributes: friends, sports, extracurricular activities.

• Parents?Attributes: safety, education, sports, future opportunities.

• Colleges?Attributes: critical thinking skills, math skills, communication skills, personality, drive.

• Employers?Attributes: work ethic, communication skills, ability to function as part of a team.

• Society?Attributes: likelihood of incarceration, likelihood of becoming a welfare recipient.

15

Second: On What is Assessment Based?

• If Consumer = StudentAssessment based on the student’s social environment.

• If Consumer = ParentsAssessment based on parents’ satisfaction with perceived educational environment.

• If Consumer = CollegesAssessment based on students’ college preparation.

• If Consumer = EmployersAssessment based on students’ job skills.

• If Consumer = SocietyAssessment based on crime and poverty measures.

16

Consumer = Student

(assessment = social envrnmnt)

Consumer = Parents

(assessment = educational envrnmnt)

Consumer = Colleges

(assessment = college preparation)

Consumer = Employers

(assessment = job skills)

Consumer = Society

(assessment = crime/poverty rates)

Consumer Can Assess Quality in Short Run?

Consumer Mobility?Consumer Requires Repeat Purchase?

Yes No Yes

YesMaybeYes

Yes Yes Yes

YesMaybeYes

Yes No No

Only three definitions of “consumer” permit accountability.

Third: Is Accountability Possible?

17

How we define “the consumer” ultimately determines how education will adapt.

Definition of “consumer” determines

attributes and tradeoffs, which determine

assessment, which determines

accountability, which determines

adaptation.

Fourth: What Adaptations Result?

18

Assessment


Assessment is college preparation.

Accountability is imposed via acceptance rates.

Schools adapt by making students more attractive to colleges so as to boost acceptance rate.

Fourth: What Adaptations Result when Consumer = Colleges?

19

Assessment


Assessment is college preparation.

Accountability is imposed via acceptance rates.

Schools adapt by making students more attractive to colleges so as to boost acceptance rate.

Unintended Consequence

Schools will encourage students who are technically, rather than academically, talented to go to college rather than to prepare for the job market.

Fourth: What Adaptations Result when Consumer = Colleges?

20

Assessment


Assessment is job skills.

Accountability is imposed via employment and wage rates.

Schools adapt by making students more attractive to employers so as to boost employment and wage rates.

Fourth: What Adaptations Result when Consumer = Employers?

21

Assessment


Assessment is job skills.

Accountability is imposed via employment and wage rates.

Schools adapt by making students more attractive to employers so as to boost employment and wage rates.


Schools will encourage students who are academically, rather than technically, talented to prepare for the job market rather than to go to college.

Fourth: What Adaptations Result when Consumer = Employers?

22

Assessment


Assessment is educational environment.

Accountability is imposed via changing schools.

Schools adapt by making themselves more attractive to parents so as to boost enrollment and retention.

Fourth: What Adaptations Result when Consumer = Parents?

23

Assessment


Assessment is educational environment.

Accountability is imposed via changing schools.

Schools adapt by making themselves more attractive to parents so as to boost enrollment and retention.


Schools will develop educational environments that conform to parents’ perceptions of quality education.

Fourth: What Adaptations Result when Consumer = Parents?

24

Let’s assume that:

1. We can skip the step of identifying the consumer, and

2. Standardized Testing is the appropriate assessment tool,

What is the impact of assessment on adaptation?

Can We Assess Education Without Identifying the Consumer?

25

Assessment


Assessment is Standardized Testing.

Accountability is imposed via funding (“high-stakes testing”).

Schools adapt by making students better able to pass the test so as to boost funding.

Fourth: What Adaptations Result when Consumer = Undefined?

26

Assessment


Assessment is Standardized Testing.

Schools adapt by making students better able to pass the test so as to boost funding.


Schools will prepare students to pass the Standardized Test (“teaching to the test”).

Accountability is imposed via funding (“high-stakes testing”).


27

Let’s assume that:

1. We can skip the step of identifying the consumer, and

2. Standardized Testing is the appropriate assessment tool,

What attributes do school districts control?


28

An oft-cited important attribute is the Student-Teacher ratioThis strategic plan stresses the need for strategies aimed at keeping more teachers in the classroom.

– Strategic Plan, Texas Higher Education Coordinating Board, 2002.

Teacher recruitment is the key to the drive to reduce class size.

– Congresswoman Eddie Bernice Johnson, (D-TX), 2005.

Utah will undertake several initiatives aimed at keeping more teachers in the classroom.

– Utah Governor John Huntsman (R), 2005.

Growing numbers of students in our nation’s schools…mean that our need for teachers increases each year.

– Meeting the Challenges of Recruitment and Retention, NEA, 2005.

29

400

420

440

460

480

500

520

540

560

580

10 12 14 16 18 20 22 24

Student/Teacher Ratio, 2003

NA

EP

Sco

res,

200

3

8th Grade 4th Grade

Student-Teacher Ratio has no apparent impact on NAEP scores.

Source: National Center for Education Statistics, U.S. Department of Education

Each dot represents one state.

Reported scores are sums of state averages for math and reading.

30

400

420

440

460

480

500

520

540

560

580

100 200 300 400 500 600 700 800

Student/School Ratio, 2003

NA

EP

Sco

res,

200

3

8th Grade 4th Grade

School size has no apparent impact on NAEP scores.


31

400

420

440

460

480

500

520

540

560

580

$2,500 $3,000 $3,500 $4,000 $4,500 $5,000 $5,500 $6,000 $6,500 $7,000 $7,500

Instructional Spending per Pupil (2000$, State Cost of Living Adjusted), 2003

NA

EP

Sco

res,

200

3

8th Grade 4th Grade

Spending per Pupil has no apparent impact on NAEP scores.


32

400

420

440

460

480

500

520

540

560

580

$2,500 $3,000 $3,500 $4,000 $4,500 $5,000 $5,500 $6,000 $6,500 $7,000 $7,500


NA

EP

Sco

res,

200

3

8th Grade 4th Grade 8th Grade (1998) 4th Grade (1998)



33

400

420

440

460

480

500

520

540

560

580

$3,000 $3,500 $4,000 $4,500 $5,000 $5,500 $6,000 $6,500

Non-Instructor Spending per Pupil (2000$, State Cost of Living Adj.), 2003

NA

EP

Sco

res,

200

3

8th Grade 4th Grade



34

400

420

440

460

480

500

520

540

560

580

$40,000 $45,000 $50,000 $55,000 $60,000 $65,000

Median Family Income (2000$, State Cost of Living Adjusted), 2000

NA

EP

Sco

res,

200

3

8th Grade 4th Grade

Median Family Income has no apparent impact on NAEP scores.


35

Perhaps standardized tests are inadequate for assessing the quality of education.

What about dropout rate?

This measure comes closer to reflecting parent/student choice-to-purchase.

36

1.0%

2.0%

3.0%

4.0%

5.0%

6.0%

7.0%

8.0%

9.0%

10 12 14 16 18 20 22 24

Student/Teacher Ratio, 2000

Dro

po

ut

Rat

e (G

rad

es 9

-12)

, 20

00Student-Teacher Ratio has no apparent impact on

dropout rate.


37

1.0%

2.0%

3.0%

4.0%

5.0%

6.0%

7.0%

8.0%

9.0%

100 200 300 400 500 600 700 800

Student/School Ratio, 2000

Dro

po

ut

Rat

e (G

rad

es 9

-12)

, 20

00School Size has no apparent impact on dropout

rate.


38

1.0%

2.0%

3.0%

4.0%

5.0%

6.0%

7.0%

8.0%

9.0%

$2,500 $3,000 $3,500 $4,000 $4,500 $5,000 $5,500 $6,000 $6,500


Dro

po

ut

Rat

e (G

rad

es 9

-12)

, 20

00Spending per Pupil has no apparent impact on dropout

rate.


39

1.0%

2.0%

3.0%

4.0%

5.0%

6.0%

7.0%

8.0%

9.0%

$3,000 $3,500 $4,000 $4,500 $5,000 $5,500 $6,000

Non-Instructor Spending per Pupil (2000$, State Cost of Living Adjusted), 2000

Dro

po

ut

Rat

e (G

rad

es 9

-12)

, 20

00


Spending per Pupil has no apparent impact on dropout rate.

40

1.0%

2.0%

3.0%

4.0%

5.0%

6.0%

7.0%

8.0%

9.0%

40,000 45,000 50,000 55,000 60,000 65,000

Median Family Income (2000$, State Cost of Living Adjusted), 2000

Dro

po

ut

Rat

e (G

rad

es 9

-12)

, 20

00Median Family Income has no apparent impact on

dropout rate.


41

Does anything predict Standardized Test performance or Dropout Rate?

42

500

510

520

530

540

550

560

500 510 520 530 540 550 560

8th Grade NAEP Scores, 1998

8th

Gra

de

NA

EP

Sco

res,

200

3For 8th grade, past NAEP scores predict future NAEP

scores.


43

390

400

410

420

430

440

450

460

470

480

360 380 400 420 440 460 480


4th

Gra

de

NA

EP

Sco

res,

200

3


For 4th grade, past NAEP scores predict future NAEP scores.

44

510

515

520

525

530

535

540

545

550

555

560

420 430 440 450 460 470 480


8th

Gra

de

NA

EP

Sco

res,

200

34th grade NAEP scores predict 8th grade NAEP

scores.


45

Conclusion

Attributes that conventional wisdom suggests are appropriate have little or no impact on Standardized Test Results or Dropout Rates.

But, school performance is not random – states that perform well in the future are those that have performed well in the past.

Supposition

As most of the attributes employed by school districts are systemic, evidence suggests that educational quality may be better influenced via individual attributes.

46

Assessment, Accountability and Adaptation

April 13, 2006

copies of this presentation can be found atwww.business.duq.edu/faculty/davies

Documents

Assessment, Accountability and Adaptation April 13, 2006