Assessing the development of student ideas over time

Karen DraneyUniversity of California, Berkeley

What is meant by student growth?

• Change in average score over time?– Groups of students– Pretest/posttest model– T-test or ANOVA

• Maintaining steady level each year in standardized testing?– Normative definition– Staying at the same percentile level

What is meant by student growth, continued

• Meeting some defined set of criteria?– Reaching a set percentage (e.g. 80% correct) on

series of tests• Piagetian-style developmental levels?

– Strong theoretical background– Certain kinds of performances on specified sets

of tasks• Elimination of misconceptions?

– E.g. Minstrell’s work

Our approach

• Wilson’s BEAR Assessment System• 4 “building blocks”

– Progress variables – Items design– Outcome space– Measurement model

Connection to NRC Assessment Triangle

Observations

Cognition

NRCAssessment

Triangle

CognitionA model of student cognition and learning in the domain.

Observation

A set of beliefs aboutthe kinds of

observations that willprovide evidence of

studentsÕcompetencies.

Interpretation

An interpretationprocess for making

sense of theevidence.

Progress map

• The heart of the system• Big ideas, central concepts, organizing

principles…• It is critical to pick the right ones -- but not

too many (or teachers will not be able to use)

• Developmental in nature

“Developmental” can mean…Following a (perhaps) necessarily ordered development of more complex ideas:

Progress map for speed

Or, displaying increasing amounts of complexity:

5. Generation: Students use the models to generate new knowledge and to extend models. (~graduate school)

4. Construction: Students integrate scientific understanding into full working models of the domain. (~upper division)

3. Formulation: Students combine unirelational ideas, building more complex knowledge structures in the domain. (~lower division)

2. Recognition: Students begin to recognize normative scientific ideas, attaching meaning to unirelational concepts. (~high school)

1. Notions: Students bring real-world ideas, observation, logic and reasoning to explore scientific problem-solving. (~middle-school)

Or even, the order in which a curriculum is arranged:

Levels of Understanding

Buoyancy depends on thedensity of the object relative

to the density of the medium.

Lessons

12: Relative Density

Assessment Activities

Reflective Lesson @10

Reflective Lesson @7

Reflective Lesson @6

Post test

Buoyancy depends on thedensity of the object.

11: Density of Medium

10: Density of Object

Buoyancy depends on themass and volume of the object.

7: Mass and Volume

Reflective Lesson @11

Buoyancy depends on thevolume of the object.

Buoyancy depends on themass of the object.

6: Volume

Reflective Lesson @4

Pretest

4: Mass

1: Introduction

Why things sink and float:

Items design

• Each item directly linked to one or more of the progress variables

• Each item displays a clearly defined relationship to two or more levels of a progress variables

• The levels of a progress variable remain consistent across the set of items (and their responses)

• Relationship between items and progress variables is empirically tested

Outcome space

• Specifies the relationship between every class of responses to given items and the levels of the progress variables

• For every item, these must be ordered, finite, and exhaustive

• This takes work. Lots of work. Looking at student responses, and ideally, talking with students, is critical here

ChemQuery

Examples of items from our instrument:

Both of the solutions have the same molecular formulas, but butyric acid smells bad and putrid while ethyl acetate smells good and sweet. Explain why these two solutions smell differently.

C4H8O4

butyric acidC4H8O4

ethyl acetate

Level 1:Notions

Response 1: I think there could be a lot of different reasons as to why the two solutions smell differently. One could be that they're different ages, and one has gone bad or is older which changed the smell.

Response 2: Using chemistry theories, I don't have the faintest idea, but using common knowledge I will say that the producers of the ethyl products add smell to them so that you can tell them apart.

Level 2:Recognition

Response: "They smell differently b/c even though they have the same molecular formula, they have different structural formulas with different arrangements and patterns.”

Does the mass of the solution change after the two solutions are mixed and a solid forms?

Levels Score Student Responses

Notions1 “…it is solid so it will have more

mass.”

1+ “The mass may change …because there may be a chemical reaction…”

Recognition2- “No, you still have the same

amount.”

2 “No, it doesn’t change because the same amount of molecules...”

Please answer the following question. Write as much information as you need to explain your answer. Use evidence, examples and what you have learned to support your explanations.

Why do things sink and float?

Level What the Student Knows Example Responses (WTSFessay)

RD

Relative DensityStudent knows that floatingdepends on having less density thanthe medium.

ŅAn object floats when its density isless than the density of the medium.Ó

DDensityStudent knows that floatingdepends on having a small density.

“An object floats when its density issmall.”

MV

Mass and VolumeStudent knows that floatingdepends on having a small massand a large volume.

“An object floats when its mass issmall and its volume is large.”

M V

MassStudent knowsthat floatingdepends onhaving a smallmass.

Vo lumeStudent knowsthat floatingdepends onhaving a largevolume.

“An object floats when its mass issmall.”

“An object floats when its volume islarge.”

PM

Productive MisconceptionStudent thinks that floatingdepends on having a small size,heft, or amount, or that it dependson being made out of a particularmaterial.

“An object floats when it is small.”

“An object floats when it is a lightmaterial.”

UF

Unconventional FeatureStudent thinks that floatingdepends on being flat, hollow, filledwith air, or having holes.

“An object floats when it is hollow.”

“An object floats when it is flat.”

OT

Off TargetStudent does not attend to anyproperty or feature to explainfloating.

“I have no idea.”

The measurement model• Provides statistical evidence of assessment quality

(reliability, validity)• Provides assessment of student growth, even across

non-identical groups of items• Provides empirical evidence on the fit of the items &

outcome space to theory of progress variable• Provides evidence of the performance of particular

items and persons (including fit)• Provides the ability to develop a variety of graphical

summaries of student development• The key to making sure the system is working; drives

our process of revision

Dotted line indicates expected performance; blue line is actual performance

Gains from Pre- to Post-test

The process

Construct Map

Items Design

Outcome Space

Measurement Model

LBC initial constructs

5 Integration across variables

4 Bonding (ionic, covalent molecules, perhaps collections of molecules

3 Model of the atom (including elements and periodicity

2 Particulate view (definition of matter as particulate)

1 Macroscopic observationssimple

complex

build

ing

ma t

ter

Second round

5 Integratethe domain

4 PredictScientific models

3 RelatePatterns andEquations

2 RepresentScientificdefinitions

1 DescribeObservations& experience

0A

CharacterizingMatter

Matter is composed of

atoms arranged in various ways

BMeasuring

MatterMass is

used to accountfor matter

CCharacterizing

ChangeChange is

associated withrearrangements

of atoms

DQuantifying

ChangeMass is

used to keep track of change

EEvaluatingEnergies

Energy transfer used to analyze

tendency for change

FQuantizing

EnergyInteraction of

light with matter elucidates structure

Quantitative vs Qualitative

5 Integratethe domain

4PredictScientific models

3RelatePatterns andEquations

2RepresentScientificdefinitions

1DescribeObservations& experience

0

ACharacterizing

MatterMatter is

composed of atoms arranged in various ways

BMeasuring

MatterMass is

used to accountfor matter

CCharacterizing

ChangeChange is

associated withrearrangements

of atoms

DQuantifying

ChangeMass is

used to keep track of change

EEvaluatingEnergies

Energy transfer used to analyze

tendency for change

FQuantizing

EnergyInteraction of

light with matter elucidates structure

bonding &reactivityadvanced bonding models, nucleophiles, electrophiles

phase &compositionbond strengths, intermolecular attractions, polarity

properties & atomic viewsoctet rule, ionic and covalent bonds

matter with chemical symbolselements, compounds,valence electrons, periodic trends

properties of mattersolids, liquids, gases, mixtures

models & evidenceevidence about things we can’t observe directly

limitations of modelsexamining evidenceassumptions

measured amountsof matterdensity, grams permole,molarity

mass with a particulate viewatoms, isotopes, moles

amounts of mattermass, weight, volume, pressure

kinetics &changes in bondingrxn mechanisms, activation energy

products of reactionsolubilities, relative acid strengths

change &reaction typesprecipitation, acid-base, redox

change with chemical symbolsbalanced equations, phys,vs chem change

attributes of changemixing, dissolving, color change,change in form

stoichiometry & equilibriumweak acids &bases, solubility of salts, gases

amounts ofproductslimiting reagents,strong acid/base titrations, % yield

amounts ofreactants & productsreaction stoichiometry, pH

change with a conservation viewconservation of mass inchemical reactions

changes in amountchanges in mass,weight, volume

particle & energy viewsstatisical mechanics KE & temperature

degrees of changeentropy, free energy & equilibrium

energy transfer& changeenthalpy changes Hess’s law, bond breaking

heats &temperatureheat capacity,calorimetry,exo(endo)thermic

measures of energytemperature, scalesmeasures of energy

spectroscopy & structuregroup theory, transition probabilities

electronic structurequantum model,atomic & molecular orbitals, ionization energy

color with light absorptionabsorption &emission spectra

energies associatedwith lightfrequency, speed, Planck’s constant

lightproduction of light,color

Final version

5 Generation: Research

4 Construction: Examining assumptions, relating models

3 Formulation: Relating ideas and concepts, simple models

2 Recognition: Language, definitions, symbols algorithms

1 Notions: Everyday experience, logical reasoningnovice

expert

stud

e nt u

nder

sta n

ding

Focus on Student Understanding