Education 334XTechnology-Based Student Achievement Assessment

Electric Mysteries

Final Presentation, December 4th 2001

• Summary of Design Review and Approach

• Formative Assessment

• Storyboard

• Summative Assessment

Contents

Tradeoffs between computer-driven and hands-on assessments

Computerizing Electric Mysteries provides several logistical benefits…

• Low setup and distribution costs

• Easy administration

• Consistent equipment performance

• Little time and effort required for scoring

Putting Electric Mysteries on computer would improve its practicality as a large-scale assessment

…but computerized version must be proven effective in measuring student performance

• Lacks real world constraints

• May not accurately assess process knowledge

• Response format may not capture rich information that human observer could

Computerized version must ensure task validity and effective response format

• Anyone (I.e. non-experts) could administer the computer test

• No time needed to organize materials

• Boxes not commercially available

AdministrationAdministration

• Other than existing computers, no need for equipment purchase

• Lower per-pupil cost

• Development investment can be spread out over several spin-off applications

CostsCosts

• Human error eliminated in scoring

• Avoids problem of inconsistent equipment performance (e.g. bulb dying)

ReliabilityReliability

• Better able to map to cognitive principles (Bennett) to make “principled test”

• Avoids construct irrelevant variance due to students’ lack of practice with equipment

• Avoids equipment failure that could also compromise results

ValidityValidity

• Maintains full record of performance; hands on assessment only records last circuit interface and explanation

Response Format

Response Format Scoring SystemScoring System

• Can’t capture subtleties that human observer could

• More difficult for students to draw circuits on computer

Potential advantages of computer-driven assessments

Potential disadvantages

• Abstraction from real world constraints

• Computer familiarity could cause construct irrelevant variance

• Significant development costs

Utility of computer-driven assessments over hands-on assessments

• Computerized scoring cuts down on instructor time and makes large-scale assessment possible

• Objective evaluation

• Must have computers available for students

• Computerized version may lack capacity to make gray-area assessment distinctions

What are the essential features in a computer driven EM

• Tutorial of connecting components

• Intuitive drag and drop model

• Prompt for and record final answer, best circuit to identify box and explanation

• Need to have same reliability and validity

• Scratch paper

How reliable is EM?Would a student score the same another time?

Interrater• Very High .98 (very straight forward scoring)

Intertask• High .80-.85

– For Fifth graders, hands on (notebook and observation)

– For Eleventh graders, hands on and computer simulation

• Not High enough .61

– For Fifth graders, computer simulation

Content Validity: Is EM measuring science knowledge (doing science) or simply reading and following directions?

Cognition/ Score: Is there a clear relationship between Quality of observed Cognition (strategy use, problem representation, self-monitoring, explanation) and Measured Performance Score?

Exchangeability: Does the assessment seem to be exchangeable for direct observation? Which assessments are exchangeable?

Validity: Is EM measuring what it is supposed to be measuring?

• Formative Assessment of both adults and children

• Use of Mac Simulation for Formative Assessment

• Completion of the Storyboard

Approach to Formative Assessment

• Summary of Design Review and Approach

• Formative Assessment

• Storyboard

• Summative Assessment

Contents

Formative Assessment-- Observation and Interview

Sample

Adults (also Experts):– 1 Symbolic System Undergrad, Male, American– 2 Physics Ph.D students, Male, Chinese

• Kids: – Fifth grade, girl– Fourth grade, girl

Comprehensive (Ambitious) Information

• Adults and Kids

• Chinese and American

• Computer first, “hands on” first

• Kids exposed and not exposed to circuits

• Over-expertUse some strategies that requires more advanced but not necessary knowledge (parallel, polarity)

• ExpertUses standard strategy (brightness comparison)

• Quasi expertNeeds some aids (task, comparison strategy)

• Novice Relies on perception, uses superficial characteristics

Knowledge and Performance

“Computer” and “Hands on”

• People who do computer first have difficulty– Bulb brightness difference– The box’s function: does it light up?

• People use same strategy in both versions

• People spent less time on the second version

• Preference:

Computer (2) Hands on (2) Both (1)

Why Computer?

• Reliable– Bulbs never broken, battery never used up

• Clear– Real wires are too confusing

•More logic than mechanic– No need to worry about physical connection

Why Hands on

• “More real and more fun”

•“There is always a wall between computer and people.”

•“You cannot pick it up” (and compare their weight).

They would be equally good, if the computer …

• “Wire is not that annoying, I can cut them off easily.”

• “Brightness is more obvious.”

• “I can turn the batteries over.”

• “I can remove the box out of the circuit.”

• “If the bulb and battery can be on the desk all the time”

More Suggestion for the Current Computer Version

• Reducing Confusion “Little connection dots ” cannot be directly connected.

“Done” button makes users kicked out

“Box” is what?

Brightness cannot be figured out immediately.

Instruction should be the first page, more warming up exercise.

• Increasing FunctionMore flexibility for: Linking, Disconnecting, Turning over, Saving circuit set up.

Trade off: more flexibility and less function burden

Fun From Kids• After using the black wire for a while, "Oh, this black wire is too tired, I’m going to use the red one." • "This green wire is ugly, the yellow one is pretty, even though I know they have no difference." • "I like the real one, since I can pick them up and compare their weights." • "It is heavy, but it does not light up, it must be a dead battery or a heavy wire." • Bulb lights up, their faces light up, which is really sweet.

• Summary of Design Review and Approach

• Formative Assessment

• Storyboard

• Summative Assessment

Contents

+-

+-

NEXTNEXT

HELP

Bulbs

Wire

Batteries

Hi Mike. You have some batteries, bulbs and wires in front of you for doing some experiments.

Connect one battery, one bulb, and wires so the bulb lights.

Click Next when you are done.

EM Story BoardAfter writing his name and clicking Next, Mike sees this screen

Working Area

If Mike is unable to create the circuit, he’d receive some lessons on electricity but wouldn’t proceed with the test.

Moving fixed-shape items and wires

Students move fixed-shape items (e.g. boxes, batteries, and bulbs)by simply clicking on item to highlight it and dragging it to desired location.

Item outline becomes bold when student has highlighted item.

Students can move wires by clicking on circle at end and moving it to desired location.

+-

+-

Connectors at ends of wires and on objects appear as hollow circles when not connected.

Circles at ends of wires fill in when placed over connectors on other objects to illustrate connection has been made.

Interaction of items

Shaded circle indicates that items are in place for connection

Similarly, doubleclicking on a completed, blackened connection will un connect it and allow the objects to be moved independently

Connecting items

Students doubleclick on shaded circle to make connections, indicated by blackened circles

Blackened circle indicates that connection has been made.

HELP

Mike creates a working circuit (the back end checks it). The next screen he sees is:

NEXT

Click any time

Click when you are ready to move

forward

Instructions

Find out what is in the six mystery boxes, A, B, C, D, E, and F. They have different things inside, shown below. Two of the boxes will have the same thing. All of the others will have something different inside.

For each box, connect it in a circuit to help you figure out what is inside. You can use your bulbs, batteries and wires any way you like.

When you find out what is in a box, click the Answer box.

Click Next to start the assessment

Instructions

Find out what is in the six mystery boxes, A, B, C, D, E, and F. They have different things inside, shown below. Two of the boxes will have the same thing. All of the others will have something different inside.

For each box, connect it in a circuit to help you figure out what is inside. You can use your bulbs, batteries and wires any way you like.

When you find out what is in a box, click the Answer box.

Click Next to start the assessment

+-

+-

?

Finished

Bulbs

Wire

Batteries

Figure out what is in the mystery box labeled with a question mark “?”.

The assessment starts:

HELP

NEXT

The box has inside it either a battery or a wire. To help figure out which one is in it, connect it in a circuit with a bulb.

F

E

D

C

B

A

+-

+-

All Done

Bulbs

Wire

Batteries

Find out what is in the six mystery boxes A, B, C, D, E and F. They have five different things inside, shown below. Two of the boxes will have the same thing. All of the others will have something different inside.

For each box, connect it in a circuit to help you figure out what is inside. You can use your bulbs, batteries, and wires any way you like.

When you find out what is in a box, click on the blue answer below.

Click All Done when you are are finished with all six boxes.

HELP

NEXTF

E

D

C

B

A

Battery & Bulb

2 Batteries

Wire Nothing Bulb

+-+-

B

All Done

2 Batteries

Bulbs

Wire

Batteries

Submitting answers:

HELP

PHASE

Box B HasBattery & Bulb

2 Batteries

Wire BulbNothing

How did you know what was inside box B?

WRITE HERE

Submit Answer

Circuit Picture

When student brings a box on to work area, Answer buttons appear.

F

E

D

C

A

Construct the circuit that told you what was inside Box B, then click Circuit Picture

Clicking Circuit Picture submits the current circuit as their answer.

Clicking Circuit Picture submits the current circuit as their answer.

inside

+-+-

B

All Done

Bulbs

Wire

Batteries

Submitting answers (loop):

HELP

Battery & Bulb

2 Batteries

Wire BulbNothing

How did you know what was inside box B?

WRITE HERE.

Once student clicks DONE button, computer checks to see if choice is made and explanation given. If not student prompted to do so.

You must give answers before clicking Submit Answer. You may change your answers later by putting the mystery box back on the table.

2 Batteries

F

E

D

C

A

Circuit Picture

Construct the circuit that told you what was inside Box B, then click Circuit Picture

Submit Answer

Box B Has inside

+-+-

B

All Done

Bulbs

Wire

Batteries

Submitting answers:

HELP

Battery & Bulb

2 Batteries

Wire BulbNothing

How did you know what was inside box B?

Because the light bulb lit up with a battery and wires.

If answer is complete then answer sheet appears. Clicking Submit Answer here changes indicator and sends box back. and clears answer sheet.

2 Batteries

F

E

D

C

A

Circuit Picture

Construct the circuit that told you what was inside Box B, then click Circuit Picture

Box B Has inside

Submit Answer

F

+-

+-

All Done

2 Batteries

Bulb

Battery & Bulb

Nothing

Wire

Bulbs

Wire

Batteries

Submitting answers:

HELP

Battery & Bulb

2 Batteries

Wire BulbNothing

How did you know what was inside box F? Because the light bulb lit up with a battery and wires.

Once all boxes are complete, then finish button appears.

E

D

C

B

A

Box F Has inside

• Summary of Design Review and Approach

• Formative Assessment

• Storyboard

• Summative Assessment

Contents

Our Summative Evaluation will first focus on exchangeability with the hands-on version

H1

C1 C2

H2

rC1C2

rH1C2

rC1H2

rH1C1rH2C2

rH1H2

PROCESS – two period testing with same group of students

• Period one: students are tested using both our computerized assessment and using the hands-on assessment.

• Period two (> month later): same group of students tested using both versions.

Period 1 Period 2

Comparing correlations will determine exchangeability of versions and give explanations for discrepancies.

•Given the results of previous studies, we would expect rH1C1 to equal at least .53. A correlation lower than .53 would indicate that our assessment is less exchangeable with the hands-on version than the older computerized version of EM .

• If the correlations between the two versions in the same time period (rH1C1 and rH2C2) are significantly lower than the within-version cross-period correlations (rH1H2 and rC1C2), it would seem to indicate that the two methods give students different experiences or students view them differently

Exchangeability

Partial knowledge explanation• If all correlations are found to be about the same (likely around .53) this would support the hypothesis that students have partial knowledge and therefore their performance varies from one occasion to another regardless of method.

• If changing the methods in addition to the occasion causes correlations to drop, this would support both a partial-knowledge hypothesis and that method also causes performance to vary.

• If this study finds support for a partial-knowledge explanation, we would want to conduct a similar evaluation with experts.

Additional next steps for summative evaluation

1) Exchangeability with other methods. By seeing correlations with other measures of different kinds of knowledge, we could further assess the validity of our test.

• Grades in their science class

• Multiple choice tests on circuits or electricity

• Performance on measure of ability to conduct a scientific experiment (e.g. bugs)

Since each of these measures of performance assesses a different type of knowledge than the Electric Mysteries aims to assess, we would expect slight positive correlations.

2) Issues of utility. In addition to evaluating the exchangeability and validity of the test, we would want to study some practical considerations.

• Required resources

• Incorporation into curriculum

• Estimated administration costs and cost-effectiveness