This research is supported in part by three grants from the National Science Foundation: Developing a research-based learning progression for the role of carbon in environmental systems (REC 0529636), the Center for Curriculum Materials in Science (ESI-0227557) and Long-term Ecological Research in Row-crop Agriculture (DEB 0423627. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. EVIRONMENTAL LITERACY

EVIRONMENTAL LITERACY

Center for Curriculum Materials in Science (CCMS)Center for Curriculum Materials in Science (CCMS)

Developing progress variables and learning progressions for the Carbon CycleDeveloping progress variables and learning progressions for the Carbon Cycle

Karen Draney, Mark Wilson, Jinnie Choi, and Yongsang Lee, UC BerkeleyKaren Draney, Mark Wilson, Jinnie Choi, and Yongsang Lee, UC Berkeley

Progress variablesA progress variable is used to represent a cognitive theory of learning consistent with a developmental perspective. This is grounded in the principle that assessments be designed with a developmental view of student learning. The underlying purpose of assessment is to determine how students are progressing from less expert to more expert in the domain of interest, rather than limiting the use of assessment to measure competence after learning activities have been completed.With a progress variable, we seek to describe a continuum of qualitatively different levels of knowledge, from a relatively naïve to a more expert level.

Outcome space

The “outcome space” refers to the mapping of all possible responses to an assessment item to a particular level of the progress variable. The outcome space must be exhaustive (every student response must be mappable) and finite!A careful study of this mapping has engaged our group for many meetings. In order to ensure the reliability of our scoring methods, and the stability of our variable structure, we have taken sets of student responses to many of our assessment items, and had teams at both Michigan State University and and Berkeley, score them All differences were resolved through discussion (below left). In addition, we have attempted to select from our data an “exemplary” student response for each item at every scoring level of the variable(s) with which it is associated, and to annotate why it is exemplary (below right).

Items design Measurement Model

In the graphical representation to the right, an X represents a person performance on a collection of items related to Structure of Systems. The numbers on the right represent scoring levels (from the general level structure above) on individual items (e.g. 5.2 indicates a level 2 performance on item 5). Less difficult tasks and less proficient persons are toward the bottom of the page.

The use of a formal measurement model like this one to analyze our data allows us to examine our expectations for assessment empirically. We can make certain that items that we expect to be particularly easy or difficult in fact are, and we can find and examine item and person performances that are unusual or unexpected.

In this case, we expected that students would use similar levels of reasoning on a wide variety of different assessment tasks, leading to a “banding” effect in which the same scoring levels would occur together across most or all of the items. In our preliminary analysis of the Structure of Systems data, we have indeed observed this.

The BEAR Assessment SystemThe BEAR Assessment System

General Level Structure

Level 7 Quantitative Reasoningabout Uncertainty

Level 6 Quantitative Model-based Reasoning

Level 5 Qualitative Model-basedReasoning

Level 4 “School Science”Narratives

Level 3 Events Driven byHidden Mechanisms

Level 2 Sequences of EventsLevel 1 Egocentric Reasoning

about Events

Carbon Cycle progress variables•Structure of Systems•Tracing Matter•Tracing Energy

•Citizenship•Change over Time

Current Work

Future Work

To define the levels of the progress variables, we started with a description of the types of reasoning necessary to function at a high level of environmental literacy. In addition, we examined both previous literature, and written and interview-based accounts, from elementary, middle school, and high school students, to define additional levels. The lowest levels of reasoning, generally used by middle to upper elementary school students, we referred to as a lower anchor This includes a lack of awareness of various “invisible” mechanisms (including microscopic and atomic-molecular structure, large-scale structure, gases, etc.); reliance on senses rather than data; and narrative rather than model-based reasoning. As an upper anchor, we selected the highest levels of performance seen in high school students after completing relevant science units

---------------------------------------------- |2.4 | | | | | 2 |4.5 5.5 | |5.4 | | |1.4 3.4 7.4 | X| XX| X| 1 | XX| XX| XX| XXX|4.4 XX| XXXXX|1.3 XXXX| XXX| XXXXXX| 0 XXXXX| XXXXXXXXX|7.3 XXXX| XXXXXXXX| XXX| XXXXXXXX|3.3 XXXXXX|6.3 XXXXX|5.3 XXXXXXXXX|4.3 XXXXXXXXXX| -1 XXX|5.2 XXX| X|1.2 2.2 2.3 XX|3.2 X|6.2 XX| XXX| X|7.2 X| | -2 | X| | | | | | | | | -3 | | | |4.2==============================================

Students are beginning to answer with“school science”

Students are answering mostly with“hidden mechanism” answers

Students give “what happensnext” sorts of answers

These level 5 responsesare particularly easy

This level 3 response isparticularly easy

Graphical Representation of the Item level difficulties and person performance levels for Structure of Systems

Responses Exemplifying Levels of the Tracing Matter Variable

Exemplary Response Characteristics ofResponse

5. Model-basedaccounts acrossscales

The carbon is released from GrandmaJohnson's body and travels up through the soiland is used during photosynthesis by the plantto make oxygen. A primary consumer wouldeat the plant some where along the food chain,the coyote receives the carbon atom.

It Identifies reactants andgas such as oxygen relatedto chemical and physicalchanges.

4. School sciencenarratives ofprocesses

Decomposers break down GrandmaJohnson’s remains, leftover nutrients areabsorbed into the rests of a creosote bush, abird eats the fruit from the bush, the coyotecatches and eats the bird.

it correctly identifies somereactants, but faisl toidentifies gas.

3. Causalsequences ofevents withhiddenmechanisms

When Grandma Johnson's body begandecomposing, the carbon atom was releasedinto the soil under the creosote bush, andcould have gotten washed away by rain to thenearest lake, brook, or stream.

It indicates GrandmaJohnson's body iscomposed with carbonatoms. Decomposing(attention to hiddenmechanism)

2. Events basednarratives aboutmaterialsAttention

A carbon atom from Grandma Johnson'sremains sink into the ground and mixes withthe soil. Then when the soil is mixed andchurned, it rises to the top of the ground.When the coyote kills something upon that dirt,he may consume it and have some of them

it simply describes the pathof carbon atoms as naturalphenomena

• Ashes are macroscopic products and students who identify ashes should be distinguished from students who don’t.• “burning into ash” is a chemical reaction, so we can assume students identify chemical process with these responses.

• It could be a simple description about visible events.• Students might simply mention ashes without recognizing chemical reaction, so we should not assume students recognize chemical reaction with these responses.

Total 81,819

It turns to ash. The heat makes it lose weight. It loses weight because if the fire goes on it, It burns it and turns into ashes. Because wood burns into ash, takes out the water and sap. It becomes ashes. Ashes don’t weigh anything.

It turns to ash. The heat makes it lose weight. It loses weight because if the fire goes on it, It burns it and turns into ashes. Because wood burns into ash, takes out the water and sap. It becomes ashes. Ashes don’t weigh anything.

Students’ actual responsesStudents’ actual responses

TM Item scoring issue : BURNING A MATCH

Q : What happens to the wood of a match as the match burns? Why does the match lose weight as it burns?

MSU : More than Level 3MSU : More than Level 3

UCB : Level 2UCB : Level 2

Each item must represent some number of levels of one or more progress variables. The design of items, and the selection of existing items, to represent the variables, has occupied much of our time. In this process, the progress variables take form.

Grandma Johnson (GRANJOHN)Grandma Johnson had verysentimental feelings toward JohnsonCanyon, Utah, where she and herlate husband had honeymooned longago. Because of these feelings,when she died she requested to beburied under a creosote bush in thecanyon. Describe below the path of acarbon atom from GrandmaJohnson’s remains, to inside the legmuscle of a coyote. NOTE: Thecoyote does not dig up and consumeany part of Grandma Johnson’sremains.

CRD: “A carbon atom fromGrandma Johnson's remainssink into the ground andmixes with the soil. Thenwhen the soil is mixed andchurned, it rises to the top ofthe ground. When the coyotekills something upon that dirt,he may consume it and havesome of the soil come with it,which produces him withGrandma Johnson's carbonatom.”

GrandmaJohnson

Plants

Decomposers

Herbivores

Coyote

Soil

MCK: “The carbon ingrandma body isdecomposed into theground. The plants thenuse the fertile soil to useher carbon atoms. As thesoil passes it to the plant,the plant is eventuallyeaten by the coyote. Thecarbon atom then travelsto its leg.”

GrandmaJohnson

Plants

Decomposers

Herbivores

Coyote

Soil

DES: “GrandmaJohnson's remains decayand decomposers userespiration and turn itto carbon dioxide. Theplants absorb the carbondioxide. Rodents eat theplants and then thecoyote eats the rodent.”

GrandmaJohnson

Plants

Decomposers

Herbivores

Coyote

Carbon dioxide