View
1.812
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Discussion of computer assisted formative assessment projects at The Open University.
Citation preview
DMW 2008 The Open University's Institute for Educational Technology
Investigating the Pedagogical Push and Technological Pull of
Computer Assisted Formative Assessment
Denise Whitelock
The Open University, Milton Keynes
DMW Learning for Life Workshop March 2008
Behaviourist prescriptions for assessment: wolves in sheep's clothing
DMW Learning for Life Workshop March 2008
Don’t forget the human reaction
DMW Learning for Life Workshop March 2008
www.storiesabout.comwww.storiesabout.com/[email protected]
DMW 2007 The Open University's Institute for Educational Technology
Formative Assessment
Black & Wiliam (1998)
improvement 0.4 to 0.7 on standardised tests
MacFarlane-Dick (2006)
empower students as self regulated learners
Feedback and assessment – moving forward
DMW 2007 The Open University's Institute for Educational Technology
Collaborative Formative Assessment
• CSCL …. on task talk
• Predict Look and Explain
• Collaborative problem solving find a complex problem
• Tools to support distance students to communicate, find a partner
• Appropriate simulation
DMW 2007 The Open University's Institute for Educational Technology
BuddySpace
• Automatic map construction to show locations
• Automatic roster (‘buddy list’) construction using OU login authentication… so it knows which workgroup or tutorial group I’m in
• Many views: map, timelines, etc.
• Plug-ins for future enhancements
DMW 2007 The Open University's Institute for Educational Technology
BuddySpace
DMW 2007 The Open University's Institute for Educational Technology
• “State of mind”
• goals, plans, intentions
• devices
• user profiling
DMW 2007 The Open University's Institute for Educational Technology
Smart inset chosen,depending onactual data
AutomaticMapGeneration
DMW 2007 The Open University's Institute for Educational Technology
AutomaticMapGeneration
Right-click to ‘drill down’into ‘cluster’ nodes (provides scaleability)
DMW 2007 The Open University's Institute for Educational Technology
BuddyFinder
•Who else is facing a similar problem?•Where can I find complementary approaches?•Has anyone found contrary evidence?
Generally: “Find me the person who can really help me now.”
More specifically:
DMW 2007 The Open University's Institute for Educational Technology
Arbitrary keywordsat user’s discretion
BuddyFinder
DMW 2007 The Open University's Institute for Educational Technology
Easysearch
DMW 2007 The Open University's Institute for Educational Technology
Matches‘light up’
DMW 2007 The Open University's Institute for Educational Technology
InvokesGoogle’s Web Servicebehind the scenes,if requested,and auto-highlightswithin user’s own map!
BuddyFinder
DMW 2007 The Open University's Institute for Educational Technology
Finding an appropriate science application
• Complex problem solving with Global Warming
• 7 variables for manipulation
• Solar constant, CO2, H2O, Aerosol content, albedo, ice & snow, cloud cover
• Immediate effects of manipulation displayed as graphical output
DMW 2007 The Open University's Institute for Educational Technology
Global Warming
DMW, Institute of Educational Technology, September 1997DMW, Institute of Educational Technology, September 1997
DMW 2007 The Open University's Institute for Educational Technology
Global Warming
DMW 2007 The Open University's Institute for Educational Technology
Next: ‘Yoked’ apps via BuddySpace
Student A
Student B(‘yoked’, butwithout full
screen sharingrequired!)
DMW 2007 The Open University's Institute for Educational Technology
Global Warming: Simlink Presentation
DMW 2008 The Open University's Institute for Educational Technology
DMW 2007 The Open University's Institute for Educational Technology
Debate with SIMLINK
• More questioning
• Explanation became more important
• Socio emotive communication more evident
• Some couples separated for a while and then came back together
DMW 2008 The Open University's Institute for Educational Technology
What is Open Comment?
• Automated formative assessment tool
• Free text entry for students
• Automated feedback and guidance
• Open questions, divergent assessment
• For use by Arts Faculty
DMW 2008 The Open University's Institute for Educational Technology
Causal models of explanation
• First step:
• Identification of question types where students exhibit causal reasoning
• Looked for questions with:
• Salient variables
• Description of these variables
• Identification of trends
• Identification of relationship between the variables i.e. causality
DMW 2008 The Open University's Institute for Educational Technology
Three common types of question types were selected for Open Comment in History
1. Analysis of statistics, usually presented to the student as a table
2. Comprehension of a set text
3. Identifying similarities and differences for a given event
Knowledge elicitation/ethnography
• DW and SW acted as students
• Answered all three question types
• C grades
• Improved only when an “answer model” elicited
DMW 2008 The Open University's Institute for Educational Technology
DMW 2008 The Open University's Institute for Educational Technology
Open Comment components
• A Java-based feedback system
• A web service shell
• A graphical interface for testing
• A Moodle-based question type
• A forms-based editing tool
Pedagogical models: checking them out with you now!
• List the activities that tutors engage with when they are
marking a script.
• What things are they looking for e.g. error detection.
• Categorise the activities into a scheme or stages from
which you could build instructions for a robot to mark
the scripts.
DMW, IET, Managing Assessment SIG, June 2008
DMW 2008 The Open University's Institute for Educational Technology
Stages of analysis by computer of students’ free text entry for Open Comment: advice with respect to content (socio-emotional support stylised example)• STAGE 1a: DETECT ERRORS E.g. Incorrect dates,
facts. (Incorrect inferences and causality is dealt with below)
• Instead of concentrating on X, think about Y in order to answer this question Recognise effort (Dweck) and encourage to have another go
• You have done well to start answering this question but perhaps you misunderstood it. Instead of thinking about X which did not…….. Consider Y
DMW 2008 The Open University's Institute for Educational Technology
Computer analysis continued
• STAGE 2a: REVEAL FIRST OMISSION
• Consider the role of Z in your answer Praise what is correct and point out what is missing Good but now consider the role X plays in your answer
• STAGE 2b: REVEAL SECOND OMISSION
• Consider the role of P in your answer Praise what is correct and point out what is missing Yes but also consider P. Would it have produced the same result if P is neglected?
DMW 2008 The Open University's Institute for Educational Technology
Final stages of analysis
• STAGE 3:REQUEST CLARIFICATION OF KEY POINT 1
• STAGE 4:REQUEST FURTHER ANALYSIS OF KEY POINT 1(Stages 3 and 4 repeated with all the key points)
• STAGE 5:REQUEST THE INFERENCE FROM THE ANALYSIS OF KEY POINT 1 IF IT IS MISSING
• STAGE 6:REQUEST THE INFERENCE FROM THE ANALYSIS OF KEY POINT 1 IF IT IS NOT COMPLETE
• STAGE 7:CHECK THE CAUSALITY
• STAGE 8:REQUEST ALL THE CAUSAL FACTORS ARE WEIGHTED
Feedback
• 5 main categories
• Praise
• Developing/expanding
• Presentation and structure essay
• Misunderstanding
• Advice – concept linking, analysis
DMW 2008 The Open University's Institute for Educational Technology
DMW 2008 The Open University's Institute for Educational Technology
What have we learnt?
• Mixed Methodology
• Analysis tutor feedback of 200 plus TMAs
• Ethnographic study
• Rules of the Game
• Moving students into the relational zone of proximal development ( Hasse 2001)
• Knowing what to do helps you play with ideas i.e. Learn
• All applications based on pedagogical PUSH
• Open Source Tools
• New question type for Moodle
DMW 2008 The Open University's Institute for Educational Technology
No longer learning about X but reflecting about X
• Students are able to find facts similar to X
• Know how X might be disputed
• Are able to make predictions about X
• Know how to use X in an argument
• Know how far X can be pushed
• Supported with tools and strategies