Assessment fyfa cmg

Assessment- Making Thinking Visible

Cormac McGrathDirector Unit for Medical Education (CLK, LIME)

05/01/23Centre for Medical Education, Karolinska Institutet

Agenda: Assessment and MCQs

Student learning Interpolated memory tests Formative and summative assessment Feedback

Outcome based education Constructive alignment

Criterion versus norm based assessment Sketching a way forward

Constructing MCQs Validating MCQs Think aloud protocols Digitalising exams

Centre for Medical Education, Karolinska Institutet

From idea to action: a series of four workshops MCQ construction

Building a database Testing your MCQs

Face validity Testing on students

Talk-aloud protocols Scaling up

digitalising the project


Discussion

Positive experiences of assessing students’ work What concerns/ queries do you have about assessment?

Pair and Share2-3 min


Do student know what they don’t know? Dunning-Kreuger effect Students may: fail to recognize their own

lack of skill fail to recognize genuine

skill in others fail to recognize the extent

of their inadequacy recognize and

acknowledge their own lack of skill, after they are exposed to training for that skill


Student learning: What do we know?Backwash effect

May 1, 2023

6

ILOs Teaching methods Examination

Examination Study methods Student learning

What supports learning?

Interpolated memory testsstudents retained focus less likely to mind

wanderscored better on

cumulative tests Szpunar K,K, et al.,

2012 (Interpolated memory tests reduce mind wandering and improve learning of online learning)


What supports learning?

Interconnectedness Meaningfulness


Student learning: Feedback

Synthesis of 800 meta analysis, involving 80 million students Biggest effect on student learning: • Learners’ knowledge of goals/criteria• Results/feedback to the learner

• Feedback About the Task• Feedback About the Processing of the Task• Feedback About Self-Regulation

• The teacher's pedagogical skills• Classroom environment• Analyze teaching occasions together with colleagues

Hattie, Synligt lärande, rapport SKL , 2011

How do can we support student learning? Formative assessment

- Continuously runs during the teaching/ learning process - Diagnostic: gives feedback to students and teachers on

* strengths and weaknesses* difficulties* misunderstandings

- Gives an opportunity to modify/ improve


Discussion

To which degree do to you provide opportunities for interpolated “memory” tests in your teaching today?


Cormac McGrathCormac McGrath, UME 13

Outcome based education

Outcomes at different levels dictate student learning

Outcomes at programme level

Specific course goals

Teaching ocassion outcomes

Cormac McGrathCormac McGrath, UME 14

Course design and alingment

Learning outcomes

Assessment

Learning activities

Evaluation

Biggs, 2003


Progression in the choice of verbsSOLO taxonomy

IdentifyMake simple procedures

NumberDescribeListCombine

CompareContrastExplain relationshipsAnalyseRelateApply

Misses the point

TheoriseGeneraliseMake a hypothesisReflect

(Biggs & Tang 2007)

Discussion

Review your intended learning outcomes using the SOLO taxonomy.


Criterion V norm-referenced assessment Criterion-referenced

assessment: How well students have

learnt what we intended them to learn.

Criterion-referenced tests and assessments are designed to measure student performance against a fixed set of predetermined criteria or learning standards—i.e., concise, written descriptions of what students are expected to know and be able to do at a specific stage of their education.

Norm-referenced assessment: Comparing students

performances with each other, by ranking.

Compare to a cohort Relative to a group


Discussion

Which resonates most strongly with you; norm referenced or criterion referenced assessment? 2-3 min

To which extent do the assessment criteria express distinct qualitative differences?

Can you give examples? How do you share this with the students? How well do you discuss examination and

assessment/assessment criteria?



Change of focus: Traditional MCQ assessment Emphasis on product and declarative knowledge Is fair (?) Is economical Encourages repetition of large amounts of literature, which can provide

opportunity for overview of the subject

Criticism: Encourages quick answers rather than slow investigation Correctness rather than risk taking Memory rather than conceptual development Reproduction rather than creative application Little space/time for feedback Difficult to measure deep understanding of concepts and context

(Lindström, 1997)

Powerful examinations?

The design is supposed to create the possibility to assess: specific cognitive competences such as problem-solving,

including formulating questions and critical thinking, information competences, such as searching for relevant

information, making informed judgements, efficient use of information, analysing data,

communication competences, such as presenting data communicatively, both oral and written;

meta-cognitive competences such as self-reflection and self-evaluation.

Bloom and SOLO

Taxonomy Level Representative Verbs SOLO

Evaluation•Critique•Summarize

S4

Synthesis•Organize•Design

S3

Analysis•Compare•Categorize

S2

Application•Organize•Solve

S2

Comprehension•Distinguish•Match

S1

Knowledge•Identify•Label

S1


Can we achieve and assess higher order thinking with MCQs


From idea to action: a series of four workshops MCQ construction

Building a database Testing your MCQs

Face validity Testing on students

Talk-aloud protocols Scaling up

digitalising the project


Summary

Students’ learning is guided by examinations Backwash effect

Formative assessment facilitates learning Recurrent testing promotes learning

As course leader you can influence the design of your course and the ratio between teaching and examination.


Part two:MCQ


What do we want to assess? How do we use MCQs to achieve that? Different levels of learning/understanding

Identify (what is this muscle called) Describe/list (which muscle are involved in elbow flexion) Relate/apply (muscle x is injured how does it affect movement y) Theorise/generalise/Evolve

The different types of questions

Anatomy

”Problem solving”

Isolated structure

Single joint

Multiple Joints

1 2 3 4

Identify/list

Describe/apply

Describe/apply/evolve

The pros and cons of MCQ

What parts are there to a question?

The stem

Some things to think about when crafting questions

Cormac McGrathPedagogik för universitetslärare 35

Progression of verbSOLO taxonomy

IdentifiesSimple procedures

NumbersDescribesListsCombines

ComparesContrastsExplains connectionsAnalysesApplies

Misses the point

Creates theoriesGeneralisesHypothesises Reflects

(Biggs 2003)


References

Biggs, J. &C. Tang (2007). Teaching for Quality Learning at University. SRHE & Open University Press, Buckingham.

Carroll, J. &C.-M. Zetterling (2009). Guiding students away from plagiarism. KTH Learning Lab, Stockholm.

McConnell, DA., Steer, DN., & Owens, KD. (2003) Assessment and active learning strategies for introductory geology courses. Journal of Geoscience Education, v. 51 n:o 2, p. 205-216