Design Talk as a Way to Design Talk as a Way to Assess Student Learning of Assess Student Learning of

PracticesPractices

Janet L. KolodnerJanet L. Kolodner

Jennifer Holbrook, Jackie Gray, Jennifer Holbrook, Jackie Gray, Paul J. CampPaul J. Camp

College of ComputingCollege of Computing

Georgia Institute of TechnologyGeorgia Institute of Technology

Science PracticesScience Practices

Understanding a problem what might need to be Understanding a problem what might need to be investigatedinvestigated

Investigation with a purpose -- experimentation, Investigation with a purpose -- experimentation, modeling, learning from cases, ...modeling, learning from cases, ...

Informed decision making, reporting on and Informed decision making, reporting on and justifying conclusionsjustifying conclusions

Iteration towards understandingIteration towards understanding Explaining scientificallyExplaining scientifically Teamwork, collaboration across teams, giving creditTeamwork, collaboration across teams, giving credit

Design PracticesDesign Practices

Identifying criteria, constraints, problem Identifying criteria, constraints, problem specificationspecification

““Messing about” with materialsMessing about” with materials Investigation for the purpose of applicationInvestigation for the purpose of application Informed decision making, reporting on and Informed decision making, reporting on and

justifying design decisionsjustifying design decisions Tradeoff and optimizationTradeoff and optimization Iteration towards a good enough solutionIteration towards a good enough solution Explaining failures and refining solutionsExplaining failures and refining solutions Teamwork, collaboration across teams, giving creditTeamwork, collaboration across teams, giving credit

Learning by Design™Learning by Design™

A project-based inquiry approach to science A project-based inquiry approach to science education for middle schooleducation for middle school

Students learn science and technology Students learn science and technology concepts and practices in the context of concepts and practices in the context of attempting to achieve design challenges.attempting to achieve design challenges.

Highly collaborativeHighly collaborative A variety of “community rituals” are A variety of “community rituals” are

embedded in the approach to promote embedded in the approach to promote learning of science and design practices. learning of science and design practices.

Our UnitsOur Units

Physical SciencePhysical Science– Apollo 13Apollo 13 – introduction to practices of design and – introduction to practices of design and

sciencescience– Vehicles in MotionVehicles in Motion – motion and forces – motion and forces– Machines that HelpMachines that Help – simple machines and mechanical – simple machines and mechanical

advantageadvantage

Earth ScienceEarth Science– Digging InDigging In -- introduction to practices -- introduction to practices– Managing Erosion Managing Erosion –– erosion and accretion erosion and accretion– Tunneling through GeorgiaTunneling through Georgia – geology, rocks and – geology, rocks and

minerals, rock formations, underground waterminerals, rock formations, underground water

A Typical Design CycleA Typical Design Cycle Challenge is presentedChallenge is presented Students Students mess aboutmess about to generate questions for inquiry to generate questions for inquiry Investigation to address questions followed by Investigation to address questions followed by

presentation of results to classpresentation of results to class Design planningDesign planning Pin-up sessionPin-up session Construction & testingConstruction & testing Gallery walkGallery walk Additional investigation, demo, reading, discussion of Additional investigation, demo, reading, discussion of

contentcontent IterationIteration over last three steps to solution over last three steps to solution

Two LBD™ community ritualsTwo LBD™ community rituals

Gallery walks Gallery walks (explanation and presentation (explanation and presentation of results)of results)

Pin-up sessions Pin-up sessions (justification)(justification)

Ritualized public ways of participating in Ritualized public ways of participating in science and design practicesscience and design practices

Well-articulated expectationsWell-articulated expectations Repeatedly practiced and publicly discussedRepeatedly practiced and publicly discussed

Research GoalsResearch Goals

To what extent are students learning targeted To what extent are students learning targeted content, and what is responsible for content, and what is responsible for differences?differences?

To what extent are students learning targeted To what extent are students learning targeted practices, and what is responsible for practices, and what is responsible for differences?differences?

How do student conceptions and capabilities How do student conceptions and capabilities develop over time?develop over time?

What practices by teachers are most What practices by teachers are most conducive to promoting learning?conducive to promoting learning?

What’s needed to makequalitative/observational data trustworthy?

• Focu s infor medb y theoretica l perspective• Consistenc y acros s environment s an d a wa y of

managin g reliability• Carefu l planning• Analysi s informe d by expectation – s looki ng for

confirmin g and disconfirmin g evidence• We -ll controlled quantitativ e result s t o bac k up

observations• Collectio n o f qualitative dat a tha t ca n be analyzed

quantitatively using developmenta l coding rubrics• Severa l source s of dat a tha t ca n be triangulated

Assessment MethodologiesAssessment Methodologies

Content Test Pre/PostContent Test Pre/Post– multiple choice & some essaymultiple choice & some essay– what they know and development of their understandingwhat they know and development of their understanding

20 Questions20 Questions– self-assessment of capabilitiesself-assessment of capabilities

Videotaped Performance AssessmentsVideotaped Performance Assessments– design an experiment, run an experiment, analyze datadesign an experiment, run an experiment, analyze data

Videotaped Structured InterviewsVideotaped Structured Interviews– interactive design, experiment design, & performanceinteractive design, experiment design, & performance

Ethnography & Informal DataEthnography & Informal Data

Performance AssessmentsPerformance Assessments

Ask students to carry out practices in the context Ask students to carry out practices in the context of an authentic, but short performance task.of an authentic, but short performance task.

Have students work in teams so as to be able to Have students work in teams so as to be able to record their deliberations.record their deliberations.

Video deliberations.Video deliberations. Code written products and team interactions for Code written products and team interactions for

evidence of participation in practices.evidence of participation in practices. Repeat periodically throughout experimental Repeat periodically throughout experimental

period.period.

1Group Members_____________________________________________________

_____________________________________________________

Teacher ___________________________ Block/Period___________

Group Number ______ Date___________

Where the Rubber Meets the RoadGroup Problem Solving Activity

PART I: Design and Describe a Simple Experiment in 15 M inutes or Less

Race cars and large trucks have very different types of t ires. Different kinds of rubber requiredifferent amounts of force to overcome sliding friction. In this activity, your group will try tof igure out and describe a simple experiment that tests the amount of force needed to overcomesliding friction for different types of rubber and under different road conditions. You onl y have

15 minutes from right now to f inish. Everyone read the instructions together and begi n.

INSTRUCTIONS

(1) Examine the rubber block that you were given. I t has hard rubber on one side and soft

rubber on the other side.(2) Work together as a group to design and describe a simple experi ment that compares the

amounts of force needed to overcome slidi ng fr iction for both kinds of rubber.(3) Your experiment should compare sliding friction for the two types of rubber under different

conditi ons that might be found on a road surface.

(4) On the next page, write your experiment description:(a) Select a person in your group to write down the experiment description. This personshould use black pen, and should write neatly. However, all members of the group areresponsible for what is written.

(3) On the next page, write the experiment description.

(4) Do not write on the back of the sheet; raise your hand and request an extra sheet, and writethe names of your group and “page 2” on this page.

(5) Give enough detail so that someone reading what you write wi ll know:1. What you are trying to f ind out

1 Holbrook/LBD Page 1 4/7/01

PART I: Design and Describe a Simple Experiment in PART I: Design and Describe a Simple Experiment in 15 Minutes 15 Minutes Race cars and large trucks have very different types of Race cars and large trucks have very different types of tires. Different kinds of rubber require different tires. Different kinds of rubber require different amounts of force to overcome sliding friction. In this amounts of force to overcome sliding friction. In this activity, your group will try to figure out and describe a activity, your group will try to figure out and describe a simple experiment that tests the amount of force simple experiment that tests the amount of force needed to overcome sliding friction for different types needed to overcome sliding friction for different types of rubber and under different road conditions. You of rubber and under different road conditions. You have 15 minutes. Read the instructions together and have 15 minutes. Read the instructions together and begin.begin.

INSTRUCTIONSINSTRUCTIONSExamine the rubber block. It has hard rubber on one Examine the rubber block. It has hard rubber on one side and soft rubber on the other side. side and soft rubber on the other side. Work together to design and describe a simple Work together to design and describe a simple experiment that compares the amount of force needed experiment that compares the amount of force needed to overcome sliding friction for each kind of rubber.to overcome sliding friction for each kind of rubber.Your experiment should look at sliding friction for the Your experiment should look at sliding friction for the two types of rubber under several different conditions two types of rubber under several different conditions that might be found on a road surface. that might be found on a road surface. Describe your experiment on the next page.Describe your experiment on the next page.

Coding CategoriesCoding Categories

Negotiation during collaborationNegotiation during collaboration Distribution of the taskDistribution of the task Access to prior knowledgeAccess to prior knowledge Adequacy of prior knowledgeAdequacy of prior knowledge Science/design talkScience/design talk Science practice (experiment design)Science practice (experiment design) Self checksSelf checks

Scoring the Quality of Scoring the Quality of PracticesPractices

Characterizing the Characterizing the practicepractice– 1 - didn’t do it1 - didn’t do it– 2 – recognized need; 2 – recognized need;

minimal abilityminimal ability– 3 – novice level3 – novice level– 4 – intermediate 4 – intermediate

levellevel– 5 – expert level5 – expert level

Characterizing group Characterizing group behaviorbehavior– 1 – nobody did it1 – nobody did it– 2 – one person 2 – one person

recognized need and recognized need and triedtried

– 3 – half the group 3 – half the group triedtried

– 4 – done well half the 4 – done well half the timetime

– 5 – group fluency 5 – group fluency

Scoring the quality of Scoring the quality of practicespractices

A 5 -point likert scale for each coding A 5 -point likert scale for each coding category reflects a quantitative continuum.category reflects a quantitative continuum. – 1 = Not at all: no evidence of the quality to be 1 = Not at all: no evidence of the quality to be

ratedrated– 2 = Some evidence that at least one episode or 2 = Some evidence that at least one episode or

one student exhibits the quality ratedone student exhibits the quality rated– 3 = The quality is exhibited by half the group3 = The quality is exhibited by half the group– 4 = The quality is exhibited for more than half 4 = The quality is exhibited for more than half

the episodesthe episodes– 5 = The quality completely captures the nature 5 = The quality completely captures the nature

of the episodesof the episodes

Performance Assessment tasks: Coding for science practice

Jackie Gray, Paul Camp, Jennifer Holbrook, Barbara Fasse, and Janet Kolodner

Additional notes are fine and can be recorded on the coding sheet.Please note which event segment is being coded for each episode:

planning an experiment; problem set up; experimental manipulation; response

to written questions.In general, the 5 -point likert scale reflects the following quantitative

continuum. Details for each item are also included below.1 = Not at all: no evidence of the quality to be rated

2 = Some evidence that at least one episode or one student exhibits the

quality rated3 = The quality is exhibited half the time

4 = The quality is exhibited for more than half the episodes5 = The quality completely captures the nature of the episodes

Design an experiment segment:

Within an episode, the context of the group is characterized by:

Negotiations

Not at all at least one of the

members of the

group suggests acompromise about

some aspect of the

procedure

at least one of the

members of the

group suggests thatcompromise or

debate is needed

for at least half the

issues that require

it

at least two of the

members of the

group questionsseveral aspect of

the procedure and

the group makes

the needed change

Most decisions are

made about

procedure by theentire team

contributing and

decision making is

consensual

1 2 3 4 5

Prior knowledge is defined as students referring to some aspect of the curriculum unitthat relates to the current problem; referring to some aspect of a personal experience

that seems to relate to the current problem; referring to some aspect of the scienceconcept or method at issue that appears to come from previous exposure to the concept

or skill.

Students show evidence of using prior knowledge to solve the problem

Not at all at least one of the

members of the

group mentions aprior event or

concept that relates

to the problem

at least half of the

team mentions a

prior event orconcept that relates

to the problem

Several events and

concepts are

mentioned andapplied to the

problem

The group

routinely recalls

events or conceptsthat assist in their

collaborative

problem solving

1 2 3 4 5

Prior knowledge seems adequate

Not at all at least one of the

mentions of priorknowledge is

followed up on and

is useful

At least half the

mentions of priorknowledge are

appropriate to the

problem

More than one

member of thegroup mentions or

follows up on

events or concepts

that are useful

Every mention of

prior knowledge isdirectly applicable

to the problem

1 2 3 4 5

Students use science practice to decide on method/procedures

Not at all at least one of the

members of the

group suggest a

method to test at

least one variable

at least one of the

members suggest a

method and

indicates an

understanding of

fair testing

at least one of the

members suggest a

method and

indicates an

understanding of

fair testing andcontrolling for

variables

Most of the team

agrees that the

method used will

fairly test the

important variables

and their decisionswould actually be

a reasonable

experiment

1 2 3 4 5

The episodes are characterized by group self-checks on procedures

Not at all at least one of the

members of the

group questions

some aspect of theprocedure

at least one of the

members of the

group questions

some aspect of theprocedure and the

makes the needed

change

at least one of the

members of the

group questions

several aspect ofthe procedure and

the group makes

the needed change

More than one of

the members of the

group questions

several aspect ofthe procedure and

the group makes

the needed change

1 2 3 4 5

Table 1:Means and standard deviations for categories from performance assessment coding for

LBD students (typical and honors) and Comparison students (typical and honors)

Coding

category

Means (SD)

LBD Typical

Means (SD)

Comparison

Typical

Means (SD)

LBD Honors

Means (SD)

Honors

Comparison

Negotiations 2.50 (1.00) 1.50 (.58) 4.50 (.58) *** 2.67 (.58)

DistributedEffort/tasks

3.25 (.50) * 2.25 (.50) 4.00 (1.15) 3.00 (1.00)

Prior

knowledge

2.25 (.50) `1.75 (.50) 3.75 (1.50) 3.0 (.00)

Prior

Knowledgeadequate

2.75 (.96) 1.50 (.58) 3.50 (1.00) 2.67 (1.15)

Science terms

used

2.50 (1.29) 1.75 (.50) 3.50 (1.00) 2.67 (1.15)

Science

practice skills

2.75 (.96) 2.25 (.50) 4.75 (.50) *** 2.67 (.71)

Self-checks 3.00 (.82) ** 1.50 (.58) 4.25 (.50) *** 2.33 (.58)

Significance levels: * = p < .03; ** = p < .02; *** = p < .01The means are based on the likert scale: 1 - 5

Putting it all togetherPutting it all together Code videos of performance tasks so as to document Code videos of performance tasks so as to document

performance capabilities of students over time.performance capabilities of students over time. Use documentation of teacher practices to identify Use documentation of teacher practices to identify

high, medium, and low-fidelity LBD classes and the high, medium, and low-fidelity LBD classes and the practices of particular teachers.practices of particular teachers.

Analyze across high-, medium-, and low-fidelity LBD Analyze across high-, medium-, and low-fidelity LBD classrooms and between LBD and non-LBD classrooms and between LBD and non-LBD classrooms to see differences.classrooms to see differences.

Use observational data to try to explain those Use observational data to try to explain those differences in terms of the ways the classroom differences in terms of the ways the classroom practices and culture and teacher facilitation differed.practices and culture and teacher facilitation differed.

Making Observations Consistent: Making Observations Consistent: Observational Prompt ToolObservational Prompt Tool

Sections focus onSections focus on Individual, small-group, whole-class Individual, small-group, whole-class

activities -- what to look for in eachactivities -- what to look for in each LBD rituals – gallery walks, messing LBD rituals – gallery walks, messing

about, wbabout, wb what to look for in student and teacher what to look for in student and teacher

behavior when certain goals are active –behavior when certain goals are active –generating questions for inquiry, focusing generating questions for inquiry, focusing investigation, debugging, construction, …investigation, debugging, construction, …

OPT Sample PromptsOPT Sample Prompts

QuestioningQuestioning– What are teacher questions What are teacher questions

about?about?– What are student questions What are student questions

about?about?– What question types are being What question types are being

used?used?– Purpose of teacher questions?Purpose of teacher questions?

– How does teacher deal with off-How does teacher deal with off-topic questions?topic questions?

– (each question has a menu of (each question has a menu of types and a set of examples types and a set of examples associated)associated)

Gallery WalksGallery Walks – Who initiates the session?Who initiates the session?

– Who displays the artifact?Who displays the artifact?

– Who asks questions?Who asks questions?

– Who gives feedback?Who gives feedback?

– In what ways is feedback In what ways is feedback constructive?constructive?

– What comparisons are made What comparisons are made between groups or to previous between groups or to previous work of the presenting group?work of the presenting group?

– To what extent are students To what extent are students being explicit about design being explicit about design decisions, expectations, decisions, expectations, explanations?explanations?

LBD’s FoundationsLBD’s Foundations

Case-based reasoning’s model of learning from Case-based reasoning’s model of learning from experience (Kolodner, Schank, Hammond, …)experience (Kolodner, Schank, Hammond, …)

Problem-Based Learning’s model of the classroom Problem-Based Learning’s model of the classroom (Barrows, …)(Barrows, …)

Communities of Learners (Brown, Campione), Communities of Learners (Brown, Campione), Constructionism (Papert, Harel, Kafai, …), Constructionism (Papert, Harel, Kafai, …), Cognitive Apprenticeship (Collins, Brown, …), Cognitive Apprenticeship (Collins, Brown, …), architecture studio, Decision-Based Design architecture studio, Decision-Based Design (Mistree, …), transfer, development, ...(Mistree, …), transfer, development, ...

A project-based inquiry approach with more.A project-based inquiry approach with more.