Upload
alyson-allison
View
216
Download
0
Embed Size (px)
Citation preview
1Bacon – T. A. Webinar – 7 March 2012
Transforming Assessment with Adaptive Questions
Dick BaconDepartment of PhysicsUniversity of Surrey
Using assessments to encourage learning
2Bacon – T. A. Webinar – 7 March 2012
Transforming Assessment with Adaptive Questions
Effective feedback
Adaptive questions
Question design
‘Help’ design
Results and discussion
Conclusions
Student opinions
Context
3Bacon – T. A. Webinar – 7 March 2012
Context
Four level 1 electronics tests containing 27 questions, that are set as coursework
The questions have been used in SToMP assessment systems (this is the fourth) since 1995.
The latest system SToMP-II is QTI v2.1 compliant, and this paper discusses the deployment of 15 of these questions that have been modified to improve the effectiveness of their feedback.
Transforming Assessment with Adaptive Questions
www.stomp.ac.uk
4Bacon – T. A. Webinar – 7 March 2012
To be effective, feedback must be
This can all be achieved within one QTI v2.1 adaptive question
► targeted( use mal rules )
► relevant to student’s probable understanding( use history )
► attended to( give it immediately but randomise the data )
► applied( offer another try )
freely adapted from Gibbs and Simpson (2004)
Transforming Assessment with Adaptive Questions
5Bacon – T. A. Webinar – 7 March 2012
v2 includes support for adaptive questions.
A specification for the transfer of assessment questions between different testing systems.
The most widely implemented version is v1.2.1, which was published in March 2003.
v2.0 was published in May 2005. The structure is similar, but the detail is completely different.
Support for numerical questions is much improved in v2, including randomisation and the testing of both accuracy and precision.
QTI : Question Test Interoperability
6Bacon – T. A. Webinar – 7 March 2012
Adaptive questions allow a dialogue to be supported
► question text can be changed
► interactions can be changed
► feedback can be changed
When the submit button is pressed:
► further tries can be offered
!! There is no limit to the complexity !!
Transforming Assessment with Adaptive Questions
7Bacon – T. A. Webinar – 7 March 2012
Show question
Show feedback for right, approximate or precision error.Adjust mark.Set score = mark.
Show feedbackaddressing mal-rule mistake.Reduce mark.
right?mal-rule?
Show feedback for unrecognised error.Reduce mark.
mark > 3yesyes no
no
Show final ‘wrong’ feedback.Set score = 0
no
yes
Combine these for a multiple numeric response question . .
Single numeric response question
. . blanking the correct responses.
Set mark = 10
Transforming Assessment with Adaptive Questions
8Bacon – T. A. Webinar – 7 March 2012
Help was somehow to be provided for the student who did not know how to start the problem.
For clarity, this was offered as a separate entity.
Giving help
We also wanted to give the student some control over the amount of detail that was to be given, i.e. the level of help that was required.
Transforming Assessment with Adaptive Questions
9Bacon – T. A. Webinar – 7 March 2012
Transforming Assessment with Adaptive Questions
Q 3.2
10Bacon – T. A. Webinar – 7 March 2012
Describe how to tackle the problem [-2 marks] . .
Show how a solution can be obtained [-4 marks] . .
Split the problem into smaller parts [-6 marks] . .
discusses the principles involved in obtaining a solution
describes in more detail a process by which a solution can be obtained
breaks the problem down, getting the student to solve the components until the solution is obtained. Then it usually re-poses the original problem.
Transforming Assessment with Adaptive Questions
11Bacon – T. A. Webinar – 7 March 2012
no
yes
yes
Explain nth part and pose question.
Right ? 1st time ?Explain in simpler terms and usually give a numeric expression for the answer
Address the student’s inability to answer the question, but leave remediation to the student. Then continue to next part.
Dealing with feedback for the problem-parts help.
no
Transforming Assessment with Adaptive Questions
12Bacon – T. A. Webinar – 7 March 2012
In a five option MCQ, the probability of getting it right in three tries and thus gaining some marks without knowing the subject at all, is 0.7.
Thus, except for cases where there are many more options, a “don’t know” option can be the more attractive way of encouraging a student to engage.
Feedback can be made more detailed, or can offer a different viewpoint, each time ‘don’t know’ is selected. As the amount of help increases, the final mark is reduced.
Other question types
Transforming Assessment with Adaptive Questions
13Bacon – T. A. Webinar – 7 March 2012
Other question types
Two questions used the “don’t know” model(one single choice, one multiple selection)
Between them,59 right7 part right2 part wrong2 wrong2 “don’t know”s, then wrong
Transforming Assessment with Adaptive Questions
14Bacon – T. A. Webinar – 7 March 2012
Q1.2
15Bacon – T. A. Webinar – 7 March 2012
Other question types
Two questions used the “don’t know” model(one single choice, one multiple selection)
Between them,59 right7 part right2 part wrong2 wrong2 “don’t know”s, then wrong
One randomised single choice (from 6) offered multiple tries,
31 right5 right second try2 right third try3 wrong third try
Transforming Assessment with Adaptive Questions
16Bacon – T. A. Webinar – 7 March 2012
Q4.4
17Bacon – T. A. Webinar – 7 March 2012
32 right
4 mal right
4 wrong right
3 wrong wrong right
11 wrong wrong wrong wrong
4 wrong wrong
2 mal1 mal1/2 wrong wrong
2 wrong mal wrong wrong
1 mal
Q 4.4.a/b Voltage gain in LR filter
Transforming Assessment with Adaptive Questions
18Bacon – T. A. Webinar – 7 March 2012
Transforming Assessment with Adaptive Questions
Q 3.2
19Bacon – T. A. Webinar – 7 March 2012
16 right
2 mal-rule right
2 mal-rule mal-rule right
1 wrong right
2 wrong wrong right
1 mal-rule mal-rule wrong
1 wrong wrong wrong
1 wrong wrong help 1 right
1 wrong help 1 right
1 help 1 mal-rule wrong right
1 wrong wrong help 2 wrong
1 help 2 wrong wrong
1 help 3 right
1 help 3 mal-rule right
Charges on two capacitors in series
Multiple answers, with mal-rules and feedback
Transforming Assessment with Adaptive Questions
20Bacon – T. A. Webinar – 7 March 2012
questions with mal-rule feedback (+332 right)
24%
22%
10%
8%
15%
21% mal-to-right
wrong-to-right
multi-wrong-right
multi-mix-right
multi-mix-wrong
multi-wrong
ended right (196)
ended wrong (109)This chart
refers only to those who would have scored zero.
Transforming Assessment with Adaptive Questions
21Bacon – T. A. Webinar – 7 March 2012
questions with mal-rule and help fback (+71 right)
30%
6%
13%11%
13%
6%
1%
20%wrong-to-right
mal-to-right
multi-mix-right
help-to-right
help-mix-right
help-mix-wrong
multi-mix-wrong
multi-wrong
ended right (57)
ended wrong (22)This chart
refers only to those who would have scored zero.
Transforming Assessment with Adaptive Questions
23Bacon – T. A. Webinar – 7 March 2012
Transforming Assessment with Adaptive Questions
effects of mal-rule and wrong feedback, and help
0
10
20
30
40
50
60
70
tran
siti
on
co
un
t -
per
cen
tag
e
mal to right
mal to wrong
wrong to right
wrong to wrong
help to right
help to wrong
There were 23 instances of help being requested,156 mal-rule instances and 316 wrong instances.
156 316 23
24Bacon – T. A. Webinar – 7 March 2012
Student comments:
“Really liked the opportunity to have another go at questions with hints if you were unsuccessful in the first attempt.”
Best feature ?
“Giving hints if the answer entered is wrong.”
“The hints given as to how to do the question if you got it wrong in the first instance, and the chance to have another go, albeit for less marks.”
“the hints - they often nudge my attention to a key fact”
“the overall presentation was clear and easy to understand what was required of us”
“the ability to see your mark immediately afterwards “
Transforming Assessment with Adaptive Questions
25Bacon – T. A. Webinar – 7 March 2012
Student comments:
Worst feature ?
“Difficult to know how to enter data with powers.”
“would like a marks summary”
“The inability to review closed questions.”
“sometimes the hints when you got it wrong were totally irrelevant”
“Not very reliable - we experienced a few crashes and disruptments with the system over the year”
Transforming Assessment with Adaptive Questions
26Bacon – T. A. Webinar – 7 March 2012
“sometimes the hints when you got it wrong were totally irrelevant”
With multiple randomised variables, two different mal rules may well have very similar values, and the condition used for mal rule testing only checked within 5% (to cater for precision and rounding effects).
I have addressed this by checking for all mal rules when marking, and presenting all appropriate feedback for the mal rules that have matching values.
Transforming Assessment with Adaptive Questions
27Bacon – T. A. Webinar – 7 March 2012
“sometimes the hints when you got it wrong were totally irrelevant”
Otherwise, the feedback may indeed be badly written – so as to be unrecognisable by students with particular misconceptions, for example.
This will need a detailed analysis of what a student needs to know to be able to solve the problem (concept mapping).
It might be addressed by starting a dialogue (within the feedback) to establish what it is that each student needs to be told.
Transforming Assessment with Adaptive Questions
28Bacon – T. A. Webinar – 7 March 2012
Conclusions:
► looking for more mal-rules
► improving some of the mal-rule feedback text
► making ‘wrong’ feedback progressive, or interactive
2. That out of a total of 758 student responses, 205 were corrected by a combination of targeted and general feedback and online help, leaving only 111 as incorrect.
1. The questions need further development, specifically . .
► improving some of the help text
► providing more mark information for users ?
Transforming Assessment with Adaptive Questions
29Bacon – T. A. Webinar – 7 March 2012
The end.
If you are interested in getting involved in creating or using these sorts of questions, then do contact me: [email protected]
If you are interested in seeing what is available in QTI v2.1 systems,Then have a look at the JISC project that has been collecting resources and helping finalise the current specification,
http://cloud.niallbarr.me.uk/ .
To find out more about the SToMP system, look athttp://www.stomp.ac.uk .
To locate e-questions for a variety of assessment systems (including QTI v1 & 2) look at the Physical Sciences Question Bank,
http://www.heacademy.ac.uk/physsci/ (follow Q for Question bank) or (better) from a link in the SToMP site.
(The HEA site is a legacy site, but the SToMP site is maintained)
Transforming Assessment with Adaptive Questions