The Lower Order Expectations of High-Stakes Tests:

A Four-State Analysis of Social Studies Standards and Test Alignment

Research team– Scott DeWitt, University of Cincinnati, Blue Ash College– Nancy Patterson, Bowling Green State University– Whitney Blankenship, Rhode Island College– Brooke Blevins, Baylor University– Lorrei DiCamillo, Canisius College– David Gerwin, Queens College - CUNY– Jill M. Gradwell, Buffalo State College – SUNY– John Gunn, Queens College – CUNY– Lamont Maddox, University of North Alabama– Cinthia Salinas, University of Texas – Austin– John Saye, Auburn University– Jeremy Stoddard, The College of William and Mary– Caroline C. Sullivan, Georgia State University

Issues with Conventional Testing• Assessment to both promote and measure

student achievement (NCLB, 2004).

• Multiple choice questions can range along a continuum of cognitive skills from lower to higher order, but there are no set rules for producing test items targeted to particular cognitive skill levels (Haladyna, Downing, & Rodriquez,

2002).

Alignment as Essential• Assessment does not accurately reflect

achievement when standards and tests do not match (developing understanding v. rote learning) (Black & Wiliam, 1998).

The Research Gap

While high-stakes accountability is premised on the fact that tests measure the extent to which students have met objectives set by state content standards, there are “surprisingly few empirical studies that explore the correspondence between test scores and the competencies laid out in state standards” (Reich,

2009, p. 326).

The Study

• To what extent do high-stakes tests hold students accountable for demonstrating the cognitive skills required in the corresponding state academic standards in social studies?– What is the cognitive level of the standards in each

state?– What is the cognitive level of the test items in each

state?– Across the four states, how well are the tests

aligned with the standards?

Authentic Intellectual Work(Newmann, King, & Carmichael, 2007)

CRITERIA Instruction Assignments Student Work

Construction of Knowledge

Disciplined Inquiry

Value Beyond School

Higher Order Thinking

Deep Knowledge

Substantive Conversation

Connections to the World Beyond the Classroom

Construction of Knowledge

Elaborated Written Communication

Connections to Students’ Lives

Analysis

Disciplinary Concepts

Elaborated Written Communication

Data Analysis Framework

• Revised Bloom’s Taxonomy (Anderson, et al., 2001)

– Bloom’s Taxonomy is widely used by standards writers and test developers to differentiate item expectations and intended cognitive difficulty.

• The six Taxonomy categories are grouped into Lower Order (Remember and Understand) and Higher Order (Apply, Analyze, Evaluate, Create).

Comparison by State of Standards Ratings

  NY OH TX VA

# Higher Order

14/32 24/42 64/134 30/173

% Higher Order

44 57 48 17

Findings

Norming Guidelines1. Questions with a document that do not require application of

skills beyond comprehension are Lower Order.

2. Chronology questions are Lower Order since the primary expectation is for students to remember events in sequence.

3. Questions whose answers provide only one response related in any way to the prompt are Lower Order.

4. Assume when reading the question that students do not have the content knowledge necessary to answer by remembering. Any question that provides students information that creates a possibility of figuring out the answer, e.g., through analysis, is Higher Order.

FindingsComparison of Multiple Choice Question Ratings

  NY OH TX VA

Global History & Geog

US History & Govt.

Social Studies

Social Studies

World History I

World History

II

US & VA History

# Higher Order/Total 

6/50 5/50 3/32 6/55 3/60 1/60 5/60

% Higher Order 12% 10% 9% 11% 5% 2% 8%

Comparison by State of Percentage of Higher-Order Standards to Higher-Order MC Questions

NY OH TX VA

% Higher-Order Standards

44 57 48 17

% Higher-Order MC Questions

11 9 11 5

Conclusions

• Better tools are needed to authentically assess secondary social studies students’ learning.

• The assessments do not allow students to demonstrate a level of cognitive ability matching the expectations of the standards or necessary to become productive citizens.

Conclusions

• Testing policies vary by:– Defining critical thinking as a distinct testing

criterion (OH, TX)– Excluding particular standards from the testing

process (VA, NY) *

Comparison by State of Percentage of Higher-Order Standards to Higher-Order MC Questions

NY OH TX VA

% Higher-Order Standards 44 57 48 17

% Higher-Order MC Questions 11 9 11 5

Testing policies vary by:Defining critical thinking as a distinct testing criterion (OH, TX)Excluding particular standards from the testing process (VA, NY)

Conclusions

• These high-stakes tests institutionalize lower order expectations for students.

• Students receive a message that the knowledge that matters in social studies is lower-order remembering and explaining.

• Narrowing content and focusing on teacher-centered strategies are rational reactions by teachers to their contexts.

Questions & Comments

NormingIn recent years, information about foods and food preparation around the world has had a wide audience on U. S. television and on internet websites. People who are eager to try new recipes have created a demand for ingredients not usually produced in the United States. To meet this demand, U.S. supermarkets have been stocking goods formerly unknown to U.S. buyers.

What has been the cultural effect of the trend described above?A. a decrease in religious restrictions on particular foodsB. a reduction in the numbers of people “eating out” in restaurantsC. the inclusion of additional international food choices in the U.S. dietD. people in foreign countries eating more foods produced in the United States (Ohio Department of Education, 2009a, p. 9)

Norming

The federal lands shown above were set aside as a response to concerns about:A. border conflictsB. community developmentC. environmental conservationD. property values (Texas Education Agency, 2009b, p. 31)

NormingWhich heading below best completes the partial outline below? ___________________________________

– Magna Carta– House of Burgesses– Town meetings– John Locke

A. Ideas of Social DarwinismB. Basis of British MercantilismC. Contributions to American LiteratureD. Influences on United States Constitutional Government

(The University of the State of New York, 2010c, p. 2)

Norming• In the 1940s, President Franklin D. Roosevelt made winning World War

II a priority over extending the New Deal.• In the 1950s, President Harry Truman’s focus shifted from the Fair Deal

to the Korean War.• In the 1960s, President Lyndon B. Johnson’s attention to the Great

Society gave way to preoccupation with the Vietnam War.

These presidential actions best support the conclusion thatA. Presidents prefer their role as commander in chief to that of chief legislatorB. Domestic programs are often undermined by the outbreak of warC. Presidents Roosevelt, Truman, and Johnson were not committed to their domestic initiativesD. Large domestic reform programs tend to lead nations toward involvement in foreign wars (The University of the State of New York, 2010c, p. 7)

Norming

“Identify four factors (social, political, economic and/or environmental) that contributed to the migration of large numbers of people from Europe to the United States in the late 19th century” (Ohio Department of Education, 2009a, p. 5).

The Scoring Guide provides the following as the “exemplar” response:• U. S. industrialization offered opportunities for jobs. Better

wages were available in the United States. Land was abundant in the United States. Ethnic minorities came to the United States to escape persecution (Ohio Department of Education, 2009b, p. 15).

Norming

• A prince, therefore, ought always to take counsel, but only when he wishes and not when others wish… also, on learning that any one, on any consideration, has not told him the truth, he should let his anger be felt. – Machiavelli, The Prince

Based on this quote, Machiavelli’s political writings supported the idea of A. absolute powerB. limited monarchyC. popular sovereigntyD. parliamentary government (Virginia Department of Education, 2007b, p. 34)

All of the following contributed to the trend shown in the graph EXCEPT – a. increasing industrializationB. poor worker-management relationsC. discontent with working conditionsD. a decline in the population (Virginia Department of Education, 2007a, p. 19)