View
218
Download
3
Tags:
Embed Size (px)
Citation preview
1/∞CRESST/UCLA
Towards Individualized Instruction Using Technology
Gregory K. W. K. Chung
Annual CRESST ConferenceLos Angeles, CA – January 22, 2007
UCLA Graduate School of Education & Information StudiesNational Center for Research on Evaluation,Standards, and Student Testing
2/∞CRESST/UCLA
Structure of Talk
• Problem statement
• Research questions
• Technical approach to diagnoses and prescription
• Study design
3/∞CRESST/UCLA
Problem Statement
4/∞CRESST/UCLA
Released NAEP Items
NAEP 1990.69 correct
Current StudyLo: .73 correctHi: .87 correct
NAEP 2003.48 correct
Current StudyLo: .48 correctHi: .69 correct
NAEP 2003.52 correct
Current StudyLo: .63 correctHi: .87 correct
.37 incorrect
.13 incorrect
.52 incorrect
.31 incorrect
.27 incorrect
.13 incorrect
.48 incorrect
.52 incorrect
.31 incorrect
5/∞CRESST/UCLA
Even Simpler Problems
Lo: .36 correctHi: .68 correct
Lo: .73 correctHi: .88 correct
Lo: .68 correctHi: .90 correct
Lo: .60 correctHi: .60 correct
Lo: .56 correctHi: .90 correct
6/∞CRESST/UCLA
Study Context
• Why pre-algebra?
• Pre-algebra provides students with the fundamental skills and knowledge that underlie algebra
• In fall 2004, 57% of CSU first-time freshmen needed remediation in mathematics (22,000 students)
• Remediation includes basic math, pre-algebra, geometry, and algebra
• By fall 2005, 3500 students did not complete remediation and were disenrolled
7/∞CRESST/UCLA
Study Context
• Tasked with developing instructional and assessment supports for pre-algebra
• Develop an approach to support rapid diagnosis and remediation of pre-algebra knowledge and skill gaps
• Develop assessment and instructional tools to support classroom instruction
• Test approach with middle school students
• Do it with technology, do it now
8/∞CRESST/UCLA
Research Questions
9/∞CRESST/UCLA
Research Questions
• What is the architecture for a diagnosis and remediation system?
• What are necessary components of such a system, how is diagnosis performed, how is remediation performed, how is the system validated, how is effectiveness measured?
10/∞CRESST/UCLA
Research Questions
• To what extent can a Bayesian network codify and capture the structure and properties of pre-algebra?
• This question addresses the use of BN for representing domain knowledge (i.e., pre-algebra) and automated reasoning.
11/∞CRESST/UCLA
Research Questions
• What are methods for developing “instructional parcels”?
• What is the structure of the student interaction, what is the format of the media delivery, what are techniques that can engage students, what are the scalability issues, what are the development bottlenecks?
12/∞CRESST/UCLA
Basic Approach
Automated Reasoning
(Bayes net of pre-algebra)
Pre-algebra pretest
adding fractions distributive
property
multiplyingfractions
Individualized instruction
Individualized posttest
13/∞CRESST/UCLA
Domain Analysis
• Specify domain structure as a Bayes net
• Specify concepts and influences among concepts
• Attach math items (“observables”) to concepts to form scales
• Develop assessment items using a step-by-step derivation of a few complex problems
• Internally coherent
• Enables precise diagnoses
14/∞CRESST/UCLA
Pre-algebra Bayes Net
definitions
operations
transformations
common errors
15/∞CRESST/UCLA
Domain Sampling
16/∞CRESST/UCLA
Sample Items
DP (OPER)
CP-ADD (CE)
AP-ADD (CE)
DP (CE)
DP (CE)
DP (CE)
DP (OPER)
DP = distributive propertyCP-ADD = commutative property of additionAP-ADD = associative property of addition
CE = common errorOPER = operation
17/∞CRESST/UCLA
Instructional Design Issues
18/∞CRESST/UCLA
Study
19/∞CRESST/UCLA
Study Design and ProcedurePretest
(1st occasion)
Assign Instructi
onal Parcels
to S
Score Tests
(at UCLA)
Generate Predictio
ns
Assign S to
Conditions
Occasion 1
Occasion 2
Computer Instruction
Posttest
Experimental
Condition
Posttest
ControlCondition
Computer Instruction
2 days
20/∞CRESST/UCLA
Expected Relations
Pretest Posttest
Pretest Posttest
Pretest Posttest
Pretest Posttest
Low
Pri
or
Kn
ow
led
ge
Hig
h P
rior
Kn
ow
led
ge
No Instruction Instruction
Score Score
Score Score
21/∞CRESST/UCLA
Results (L/M Prior Know.)
aWilcoxan signed rank test (non-parametric paired test).
Scale
Instruction No Instruction
neg. pos. tie pa neg. pos. tie pa
Adding fractions 2 19 5 <.01 1 4 0 ns
Multiplying fractions 8 20 5 .06 3 4 3 ns
Reducing fractions 2 2 2 ns 1 0 3 ns
Transformations 5 23 6 <.01 1 8 3 .02
Distributive property 8 17 5 .02 6 3 1 ns
22/∞CRESST/UCLA
Results (L/M Prior Know.)
aWilcoxan signed rank test (non-parametric paired test).
Scale
Instruction No Instruction
neg. pos. tie pa neg. pos. tie pa
Commutative property of addition
1 1 5 ns 1 1 1 ns
Commutative property of multiplication
0 5 2 .04 3 0 2 .08
Associative property of addition
3 3 3 ns 1 1 5 ns
Multiplicative inverse 5 9 5 ns 1 4 0 ns
Multiplicative identity 6 19 6 <.01 4 2 4 ns
23/∞CRESST/UCLA
Discussion
• Preliminary results promising
• Individualization of instruction and assessment tractable
• Instructional parcels appear effective but more work needed
• Next steps
• Refine items and domain sampling
• End-to-end automation
• Continue gathering validity evidence
24/∞CRESST/UCLA
Questions
25/∞CRESST/UCLA