Writing short and long answer exam questions, Liz Norman 2017

Writing short and long answer

exam questions

Liz Norman

Massey University

“Effective item writers are trained, not born” (Downing, 2006)

Steven M Downing (2006) Twelve steps for effective test development, In: Downing SM and Haladyna TM (Eds), Handbook of Test Development, Lawrence Erlbaum Associates, Mahwah, NJ, USA, p.11

Breadth and level of the questions

• Are the questions focused on middle to higher level tasks such as creation, evaluation, analysis and application?

Recall-higher order classification

Fact recall Questions capable of being answered by

reference to one paragraph in a text or notes

(or several paragraphs for questions requiring

recall of several facts)

Applied

(higher order)

Questions that require the use of facts or

concepts, the solution of a diagnostic or

physiologic problem, the perception of a

relationship, or other process beyond recalling

discrete fact

Peitzman, S. J., Nieman, L. Z., & Gracely, E. J. (1990). Comparison of "fact-recall" with "higher-order" questions in multiple-choice examinations as predictors of clinical performance of medical students. Academic Medicine, 65(9), S59-60.

Knowledge-understanding classification

Knowledge knowing about; a body of coherent facts; can be

thought of as right or wrong.

Understanding knowing how and why; the meaning of facts; the

theory that links facts and provides meaning; how

sense is made of facts to enable them to be applied

to analysis, synthesis, evaluation; to be able to

explain why particular facts or skills are applicable

to a particular situation; to know which fact to

apply when; to be able to create new knowledge or

modify or adapt an idea to a new situation.

Wiggins, G., & McTighe, J. (2005). Understanding by design (2nd ed.). Alexandria, VA, USA: Association for Supervision and Curriculum Development.

SOLO taxonomy

http://pamhook.com/2012/01/20/creating-solo-taxonomy-symbols-in-many-colours/

Biggs, J. B., & Tang, C. S.-K. (2011). Teaching for quality learning at university (4th ed.). Maidenhead UK: McGraw-Hill.

Prestructural Question may be rephrased as the answer; almost completely

misses the point of the question.

Unistructural Able to identify, list, name, enumerate but does not describe,

explain, relate or elaborate multiple aspects of a response

Multistructural Able to list as well as describe distinct aspects of a response (such

as being able to describe aetiology, clinical features, management

of thrombotic stroke) but unable to explicitly explain causes for

observations; unable to present cause-effect relationships.

Relational Able to describe multiple aspects of a process and additionally

explain or elaborate observations into cause-effect relationships;

able to compare similarities and differences between apparently

distinct phenomena. This level is taken as suggesting that the

learner has understood.

Extended

abstract

Highly developed; able to explain mechanisms of phenomena and

apply this information to a novel context — able to develop novel

hypotheses, theories, and deduce principles; creative thinking.

Prakash, et al. (2010). Adv Physiol Educ, 34(3), 145-149.

Prestructural



Multistructural

Relational

Extended

abstract

Quantitative change

Qualitative change

Prestructural



Multistructural

Relational

Extended

abstract

surface

deep

Prestructural



Multistructural

Relational

Extended

abstract

recall

application

Prestructural



Multistructural

Relational

Extended

abstract

knowledge

understanding

Prestructural

Unistructural Paraphrase, define, identify, count, name, recite, follow simple

instructions, calculate, reproduce, arrange, recognise, find, note,

seek, sketch, pick

Multistructural Combine, classify, structure, describe, enumerate, list, do

algorithm, apply method, account for execute, formulate, solve,

conduct, prove, complete, illustrate, express, characterise

Relational Analyse, compare, contrast, integrate, relate, explain causes,

apply theory (to its own domain), argue, implement, plan,

summarize, construct, design, interpret (some senses), structure,

conclude, substantiate, exemplify, derive, adapt

Extended

abstract

Theorise, generalise, hypothesise, predict, judge, transfer theory

(to new domain), assess, evaluate, interpret (some senses),

critically reflect, predict, criticise, reason

Biggs, J. B., & Tang, C. S.-K. (2011). Teaching for quality learning at university (4th ed.). Maidenhead UK: McGraw-Hill.

Webb’s depth of knowledge

• Level 1 – recall– Recall information or perform a simple step, identify, measure,

describe, explain simple ideas

• Level 2 – skill/concept– Requires more than one step, comparing, interpreting,

estimating, making observations, explaining, organising and displaying data

• Level 3 – strategic thinking– Requires planning and using evidence, explaining reasoning,

make conjectures, draw conclusions, solving problems

• Level 4 – extended thinking– Complex reasoning, planning developing and thinking over an

extended period of time, making multiple connections, synthesis of ideas into new concepts

Webb, N. L. (2007). Issues related to judging the alignment of curriculum standards and assessments. Applied Measurement in Education, 20(1), 7-25



• Are the questions at the right level of difficulty?

Are the questions at the right level of difficulty?

• There needs to be a match between the depth of knowledge and degree of skill (both practical and cognitive) required to answer the question, and those specified for the topic in the learning outcomes.

• Learning outcomes specify what students are able to do when they have completed the course—so you need to think about the level of study of the student

• Day one competencies

Some factors that affect demand and difficulty

• Type of operation

• Degree of novelty

• Number of components or ideas involved

• Whether resources are provided or need to be generated by the candidate

• The question wording and any images, diagrams, or tables provided

• Degree of abstraction

• Response strategy – simple, stepwise, integrated

For EACH ECG, make a recommendation for management of the horse and explain your reasoning.

Question 1:

Explain the physiological actions of insulin.

Question 2:

Explain the physiological actions of ghrelin.

Question 1:

Compare and contrast the clinical signs of hypoadrenocorticism with those of hyperadrenocorticism in dogs.

Question 2:

Compare and contrast the clinical signs of diabetes mellitus with those of hyperadrenocorticism in dogs.

Question 3:

Compare and contrast the clinical signs of diabetes mellitus with those of hyperadrenocorticism in dogs and cats.

Question 4:

Compare and contrast the clinical signs of diabetes mellitus with those of hypoadrenocorticism in dogs and cats.

Describe the skull. (30 marks)

Question 1Discuss the use of insulin for the treatment of diabetes mellitus in cats (25 marks)

Question 2a) Describe the advantages and disadvantages of insulin

therapy for diabetes mellitus in cats (10 marks)b) Indicate the dose and frequency of administration of

insulin you would prescribe to a newly diagnosed cat with diabetes mellitus. (5 marks)

c) Describe the recommendations you would make for the frequency and timing of feeding in relation to insulin dosing in cats with diabetes mellitus (10 marks).

You have been contacted by a farmer producing Pacific oysters (Crassostrea gigas) intertidally, in a bay containing a number of oyster farms. The farmer is concerned with the amount of dead shell they are seeing during the current grading. Explain how you would approach this scenario. (20 marks)

Include in your answer how the information you could gather might influence your assessment, what differential diagnoses you consider and detail how you might further investigate potential causes and what advice you would provide.



• Are the questions at the right level of difficulty?

• Do the questions provide the right breadth of coverage and relative weighting of the learning outcomes?

Example weighting system

Scores for impact and frequency are multiplied to give a total weighting for each curriculum area.

McLaughlin, K., Lemaire, J., & Coderre, S. (2005). Creating a reliable and valid blueprint for the internal medicine clerkship evaluation. Med Teach, 27(6), 544-547.

Impact Weight Frequency Weight

Less important 1 Rarely seen 1

Essential 2 Relatively common 2

High impact 3 Very common 3

Examination structure

• Do the examination materials follow the exam structure format as described in the paper details?

• Are marks allocated to each question and separate subpart?

• Are the marks allocated to each question and each sub-part appropriate?

• Do the marks add up to the expected total?

• Is there sufficient time to answer all the questions?

Pacing examinations

• Reading time: 40 words per minute (Klatt & Klatt,

2011).

• Writing time: 16-20 words per minute (Summers

& Catarro, 2003)

• Marking schemes help you check the pacing

• Set exam to be 1 mark = 1 minute

Klatt, E. C., & Klatt, C. A. (2011). How much is too much reading for medical students? Assigned reading and reading rates at one medical school. Academic Medicine, 86(9), 1079-1083. Summers & Catarro (2003) Assessment of handwriting speed and factors influencing written output of university students in examinations. Australian Occupational Therapy Journal 50(3): 148-157

Summers & Catarro (2003) Assessment of handwriting speed and factors influencing written output of university students in examinations. Australian Occupational Therapy Journal 50(3): 148-157

Question structure

• Is it clear for each question what is required in terms of scope?

• Does each question contain an instructional verb that explains the task required to the student?

Don’t write questions; write tasks

What is your diagnosis?

State the most likely diagnosis

State the most likely diagnosis and explain your reasoning

Discuss the differential diagnoses you would consider in this case

or …..

Instructional verb examples

Compare: to find similarities between things, or to look for characteristics and features that resemble each other.

Contrast: to find differences or to distinguish between things.

Discuss: to present a detailed argument or account of the subject matter, including all the main points, essential details, and pros and cons of the problem, to show your complete understanding of the subject.

Define: to provide a concise explanation of the meaning of a word or phrase; or to describe the essential qualities of something.

Explain: to clarify, interpret, give reasons for differences of opinions or results, or analyse causes.

Illustrate: to use a picture, diagram or example to clarify a point.

Question structure


• Does each question contain an instructional verb that explains the task required to the student? Questions should not really be questions: they should be instructions instead.

Make short notes on THREE (3) of the following.

a) Pulmonary compliance.

b) Bicarbonate as a blood buffer.

c) The respiratory centre.

d) The respiratory functions of the nose.

Name two (2) diagnostic tests you would run next to investigate the cause of this dog’s current illness.

A dog is presented to your clinic after a road traffic accident.

a) What clinical signs would be consistent with urinary bladder rupture?

Outline and discuss a conceptual framework for differentiating between ryegrass varieties available in the New Zealand market place and for defining possible strengths and weaknesses of a particular cultivar. Your conceptual framework should reflect the various options currently employed by plant breeders in developing new cultivars. (10 marks)

How would you localise the site of the lesion?

Answer provided in the marking scheme:

Spinal lesion between T3 and L3

Question structure



• Does each question contain ONLY relevant content and no irrelevant content?

A client rings you up to arrange his quarterly visit as he is frustrated by ongoingproblems with his breeding herd. Some sows seem to take ages to cycle after weaning, a lot are returning (and at funny times), there is a steady trickle of abortions, and a lot of stillborns. When you get to the farm, the farmer mentions that he has also had a few sows go down at farrowing with a fever, they usually start panting and die, and there’s one right now in the old farrowing room he wants you to take a look at. As you walk through the farrowing rooms you notice that many of the sows, both expecting and lactating, have swollen vulvas. The sow in question farrowed yesterday. She clearly has a temperature and is panting. You also think she looks a bit anaemic and note that she doesn’t appear to have any milk. You suspect what the problem is but think it would be nice to confirm your diagnosis as, although it is suspected to occur in Australia, the disease has never been definitively confirmed. You take a blood sample and make a smear.

On your way home you drop the slide off at the lab and ask them if they will have a look at it for you. Later that afternoon the pathologist rings up and (very) excitedly tells you that after using Wright’s stain she spotted some cocci-like organisms attached to the red blood cells. The pathologist has rung Biosecurity Australia and they are not interested in pursuing this particular finding as they have always considered it to be present.

You ring the farmer to tell him that you have confirmed your suspicion and that he now needs to embark on the course of action you had discussed with him earlier. Write the farmer a description of the disease and outline the short and long-term course of action. Include in your report a brief discussion of what, if any, potential there is for eradication. (25 marks)

Which of the following statements is TRUE?

a. A ventral hernia is classified as a false hernia

b. A diaphragmatic hernia is classified as a true hernia

c. Umbilical hernias are classified as false hernias

d. False hernias have a hernia sac

e. Umbilical hernias are relatively uncommon

Question structure



• Does each question contain ONLY relevant content and no irrelevant content?

Question structure continued

• Are any ancillary materials used (images, radiographs, histopathology slides etc) fit for purpose, clear, legible and of good quality?

• Is a marking scheme provided which indicates the agreed standard by which students’ answers will be judged and marks awarded?

• Does the answer provided in the marking scheme answer the question that is being asked?

• Do you agree with the marking scheme?

Grammar and spelling

• Is the language used in each question clear and grammatically correct?

• Are there any spelling mistakes?

• Are units given as SI units, or the units that the student has been using in learning activities?

• Are any numerical values used realistic and correct?

http://www.slideshare.net/liznorman