Embed Size (px)
Citation preview
Meta-cognitive Education or University Challenge?
Brian WhalleyQueens University
Do we really need examinations?
orBeware External Examiners Bearing Comments
NB, as well as the slide show, there are notes and references in the Notes section.
Some things I’m going to cover
(but not necessarily the order)*• A Victorian system?
• A quick analysis of examinations at Level1• What do examinations do?• Deep Learning• Feedback in (assessment) systems• Metacognition (I’ll explain about this)• Where should assessment fit?• Criteria-based assessment• Just-in-Time Teaching, preflights• What type of assessment?• Deep Assessment (a meaningful concept?)
*Education is a highly parallel and occasional, not serial, system
Trial and error - how can we provide good learning
experiences?'You know what a learning experience is? A learning experience is one of those things that says, 'You know that thing you just did? Don't do that.’
(Douglas N Adams, 1992)
Photo: Chris Ogle
How to avoid the panic?
Do examinations provide good learning
experiences?
A Victorian system?On the basis of University Challenge (my rules)• Please answer the Qs on the folded sheet, • You have 3 minutes to do this• This is not anonymous as your answers will determine whether you get lunch – (1 course, 2 courses, 3 courses)
• I’ll collect them in, mark them and then tell you what you get for lunch – (or not, if you have not done your revision)
• But I’m not going to tell you how near you got to a better meal
• Oh, and there’s no veggie option!
Questions regarding Level 1
(semester 1):• Do you have end of semester exams?• Are these ‘essay type’?• Two questions in 2 hours?• Do these exams count towards final degree
– (and if yes, what %)
• Do you give students guidance (or practice) on what to expect in exams?
• Are any essays set which are designed? • Do you give students remarks or discussion on how well they have done post exams? – (and how is this done)?
• Do you have MCQs (or variant) rather than essays.
Assessment(some more issues)
• Do you want me to be the sole maker?• Do you want my values despite ± 10%?• Do you want a second marker’s values despite ± 10%?
• Do these values suggest you really ‘understand’ the questions?
• Should I give you marks for just trying?• What standards should I apply?• And what about criteria for pieces of work?
7 Principlesof Good Practice in Education:
1 Encourages contacts between students and tutors
2 Develops reciprocity and cooperation among students
3 Uses active learning technologies4 Gives prompt and appropriate feedback5 Emphasises time on task6 Communicates high expectations7 Respects diverse talents and ways of learning
(Chickering and Ehrmann 1996)
6 Competenciesstudents need to gain
Competence – encouragement by challenge and remarks to achieve skills levels
Confidence – promoting remarks to show themselves, and others, their achievements
Critical thinking – which is what we have been wanting all along in 'Thinking skills’, used in problem solving
Creativity – in what students do and how they do itCollaboration – bringing in team-working and ethicsCommonality – of purpose, to achieve specified (and unspecified) objectives
Curtiosity – which is more than curiosity.
Marcia Mentkowski
Mihály Csíkszentmihályi
Itiel Dror(University of Southampton, now
Cognitive Consultants International)
Control – handing appropriate responsibilities to students
Challenge – student abilitiesCommitment – student commitment to learning opportunities
What might be the best ways (note plural) todevelop these? But our usual question is not this.
Using cognitive psychology might be a help
Are these competencies, abilities and responsibilities measurable?
3 Controls on Educational Implementation
The Ten Principles of Assessment and Feedback
1. Help clarify what good performance is via aims, criteria, standards
2. Encourage ‘time and effort’ on challenging learning tasks.
3. Deliver high quality feedback information that helps learners self-correct.
4. Encourage positive motivational beliefs and self-esteem.
5. Encourage interaction and dialogue around learning (peer and teacher student)
(Nicol 2007)
10 principles continued
6. Facilitate the development of self-assessment and reflection in learning.
7. Give learners choice in assessment – content and processes.
8. Involve students in decision-making about assessment policy and practice.
9. Support the development of learning communities.
10. Help teachers adapt teaching to student needs.How do we use
exams (and coursework) to deliver these?
What actually is ‘deep learning’
• Something that is promoted by the 7P, 6C, 3C, 10P
• Experiential?•Thought-exchanging (dialogue)•Game-playing•Exemplar-exploring•Art-expressing•Problem-solving(Using the ‘200 hours’ effectively)
Something that is assessed in ways other than unseen examinations?
Assessment - what’s it for?
• Quick post-it storm (nerd-storm)– What do you think assessment is for?
•As opposed to what it is
– How do we measure (if we are to give marks)?
– Is it measurable (adequately)?In particular
If we promote Deep Learning
how do we produce Deep Assessment?
Criterion Referenced Assessment
• Provide criteria for each piece of assessed work• Avoid the ‘connoisseur’ approach (‘I know this is 57%’)
• Show students what is needed to accomplish tasks• Provide remarks with the marks• Marks and remarks go hand in hand• Give a sense of achievement• Thus, criteria need to be assignment specific and so• Link all this back to Just in Time Teaching (JITT) and After Action Reviews (AAR)– Providing proper feedback and feed-forward
External Examiners may say(after they have agreed about the marking:)
– You over-assess what does this actually mean?
– You have too many pieces of assessment– Why are there so many poor 2nd (3rd) answers in the scripts?
– You do not have enough > 60%– Your fail rate is too high ………
What they don’t (usually) ask:How do you provide feedback in this module How do students benefit from feedback?How do you assess your learning outcomes?How do you assess the skills stated for the module?
Metacognitionreflecting on one’s own knowledge base as
well as that of students
1. What is the meaning of the musical term “allegro”? A. loud, B. soft, C. quick, D. slow.
2. In Greek Myth, who is the God of wines? A. Demeter, B. Hestia, C. Artemis, D. Dionysus.
3. How many years was Margaret Thatcher Prime Minister? A. 9, B. 10, C. 11, D. 12.
4. What was Lady Chatterley’s first name? A. Eleanor, B. Constance, C. Margaret, D. Elizabeth.
5. What year was the Lockerbie disaster? A. 1986, B. 1987, C. 1988, D. 1989.
6. London’s population (2001) is approximately, in millions: A. Seven, B. Eight, C. Nine, D. Ten
In the questions about to be revealed you have to answer as you think appropriate: A, B, C, or D
BUT you will be awarded marks as follows:Correct answer 1Incorrect -2Unanswered 0So you may want to ‘pass’
However, later, you will have a further chance to gain marks
And a few more
7. What is the most common element on Earth ? A. Oxygen, B. Nitrogen, C. Iron, D. Hydrogen.
8. There is no Nobel Prize in? A. Physics, B. Chemistry, C. Biology, D. Medicine.
9. What year was John F Kennedy shot? A.1962, B. 1963, C. 1964, D. 1966.
10. What countries does Luxembourg have borders with ? A. Belgium, Holland, Germany;B. Germany, Switzerland, Belgium;
C. Holland, Belgium, France; D. Belgium, France, Germany.
11. What was the last album the Beatles recorded together? A. Let It Be, B. Rubber Soul, C. Magical Mystery Tour, D. Abbey Road
To mark1. When you have finished round one:
Have another go for those you did not answer, this time, guess, you’ll get 1 for correct and 0 for incorrect. (use a second column, different colour pen)
2. Here are the answers: mark accordingly to the two rule sets (1,0,-2) & (1,0)
3. Round 1: Negative suggests 1/3 answers were wrong - you overestimated your knowledge
4. Round 2: >25% correct suggests you should have answered more in R1 - you didn’t know you knew the answers!
MCQ or Certainty?
• Are MCQs just glorified University Challenges?
• Useful for focusing expectations rather than (or just than) knowledge per se?
• Use them as ‘preflights’ to indicate knowledge/or understanding - ie formative rather than summative.
Using metacognition• Now we have learned something about your cognitive abilities - and received some remarks about the processes
• How do we apply this to students?– Better to give MCQs before you embark on some task to show students what they know and what they need to know
– Tony Gardner-Medwin and ‘confidence-based assessment’ (rather than just MCQs)
– Students need to know the rules applied • (in general not just MCQs)
– The delayed knowledge effect (sub-conscious)
• EVEN!– Should exams be set well before the end of the semester so students can get remarks with their marks.
Rewarding the ‘experience’
• If students have problems with a troublesome concept or fail to grasp tacit knowledge do we penalise them?
• How do we provide ‘scaffolding’ in the learning experience? (The right amount of scaffolding)
• If deep learning is experiential, then how do we reward it?
• How many marks do we give for just having ‘done’ something? (Attainment)
• I don’t think an essay or MCQ scores well– Report writing is better
• Use criteria referencing• Treat exams as problem solving exercises
Towards a proper educational
feedback system• What do I mean by this?
Forget about Kolb cycles and Cowan loops* or, perhaps better, use them to give a system that works for all students (and staff)
Construct (with ‘Alignment’*) a system (for a module) that maximises learning and understanding (rather than memory recall)
To produce ‘proper’ alignment we really need more specific instructions per task/activity
And to provide remarks with the marks is use a feedback in an educational control system
Feedback and Feedforward
in a control system sense
Do ‘exams’ provide a reference input (or even output?)A MCQ might do this. Criterion Referencing might too.
Educational ‘ramp up’To raise the level of ‘attainment’We need controlled input, ramping up.
– Output, achievement, results
attainment
EeMarks, no remarks forExam events
Ese Ee Delayed marks and remarks forEssay event Ese
TTask set with CriterionReferenced assessment
Semester start Semester end
Ee endstart
‘Attainment’Ee
Delayed marks and remarks for Exam event, EeEe
Can I be more constructive?
AT
MR AT
MRTask Activity Marks Remarks
AT
MR REe
MAT
MR
Gradient = ‘learning rate’
Two Activity c/work
Two Activity c/work + exam
How do we increase the gradient?
Delays and Troublesome Knowledge & Sticking Points
• Troublesome knowledge (Land)• Tacit knowledge (Polanyi)• Delays doing activities Sticking Points
• Preflights (Just-in-Time Teaching; G Novak)– Feedforward (controlled use of advance info)
• After Action Reviews +ve Feedback or remarks
Problemspecified
Manipulation of
system
Problemsolution
CluesGuidanceHelp
Provision ofFeedback onsolutions
RulesSystemExamplesProcedures
Task Task Task
ResourcesSupport Support
Assessment
T Tacit knowledge required
S Sticking point(s) likely
TS
Fig 5.1 A temporal sequence describing a rule-based learning design, In Oliver et al. 2007. Describing ICT-based learning designs that promote quality learning outcomes
R R
What drives learningNot the assessment (itself)
– This (best) drives checking (controller in the system). It’s an indicator
Driving learning (+ve feedback)– Internal (student doing the activity)– External (comments from others, peers, tutor, manuals)
Inhibitors to learning (-ve feedback)• No remarks (‘feedback’)• Exams with no remarks (‘feedback’) before the end of the module
If we believe in these principles of ‘deep
learning’then (in general):
• Why do we set essay exams?– and seem to rely on closed exams?– and not used open book or seen?
• Why do we set exams at the end of modules?• Why do we not give ‘feedback’ on exams• Why do we believe MCQs are ‘ok’; ‘better’?• Why there are several cetls involved with active learning etc but few with ‘Assessment’?
……… and you can probably think of more
Some possible conclusions
• Students learn best by experience• Build this into assessment• Ask when, and how much, assessment is needed• Start by designing the feedback which reinforces the education
• Use the best-fitted assessment method• Do students know the criteria?• Get students (esp peer viewing/discussion) to look at criteria and then the answer(s)
• Give exams (and remarks) before the end of the module
• Fit these into the 10 Principles (etc)
Gibbs and Simpson’s (4) Conditions
Conditions under which assessment supports students’ learning11 conditions of which 4:
1. Capture sufficient study time and effort (in and out of class)
2. Are spread evenly across topics and weeks3. Lead to productive learning activity (deep learning)
4. Communicate clear and high expectations
How might we use exams?• Set questions which relate to problem solving (rather than being able to answer them by memorising)
• This might be solving an actual problem*• By providing checks or reinforcements of what has been experienced – if not experientially learned
• By using criterion referencing more– Ie by showing students what we expect* which is what (tends to) happen in e.g. physics, maths, chemistry and engineering.
Finally!
• In the light of what I have been talking about,
• Do you consider that you need to reconsider your (first year) exam process?
• Was the discussion helpful?
