CREATING EFFECTIVE QUESTIONS FOR ASSESSMENT AND AS AIDS IN LEARNING IN TODAY'S PHARMACOLOGY PROGRAMS George A. Dunaway, Ph.D. Emeritus Professor Department

CREATING EFFECTIVE QUESTIONS FOR CREATING EFFECTIVE QUESTIONS FOR ASSESSMENT AND AS AIDS IN LEARNING IN ASSESSMENT AND AS AIDS IN LEARNING IN

TODAY'S PHARMACOLOGY PROGRAMSTODAY'S PHARMACOLOGY PROGRAMS

George A. Dunaway, Ph.D.George A. Dunaway, Ph.D.

Emeritus Professor Emeritus Professor

Department of PharmacologyDepartment of Pharmacology

Southern Illinois University School of MedicineSouthern Illinois University School of Medicine

Springfield, ILSpringfield, IL

IS THERE MORE TO TESTING THAN WRITING IS THERE MORE TO TESTING THAN WRITING QUESTIONS?QUESTIONS?

Experimental Biology Meetings April 10, 2011 Washington, DCExperimental Biology Meetings April 10, 2011 Washington, DC

MAKING THE MOST OF AN ITEM ANALYSISMAKING THE MOST OF AN ITEM ANALYSIS

• Item analysisItem analysis information can be used to make information can be used to make important decisions for high-risk examinations.important decisions for high-risk examinations.

• For For present examinationpresent examination• Validity of each questionValidity of each question to decide to to decide to retainretain or or

omitomit exam questions exam questions

• Whole test validityWhole test validity to make to make passpass//failfail decisions decisions

• For For subsequent examinationssubsequent examinations, it can provide , it can provide insights into improvement of questions.insights into improvement of questions.


ITEM ANALYSIS INFORMATONITEM ANALYSIS INFORMATON• Test InformationTest Information

• Date, number of examinees and test itemsDate, number of examinees and test items• High and low scores, median and mean scores, SEM High and low scores, median and mean scores, SEM

and SD,and SD,• Test reliability, e.g., Cronbach’s alpha, which can be Test reliability, e.g., Cronbach’s alpha, which can be

interpreted as the mean of all possible split-half interpreted as the mean of all possible split-half coefficients. coefficients.

• Ranking of individual examination scoresRanking of individual examination scores

• Question informationQuestion information• Difficulty, i.e., % answering correct or Difficulty, i.e., % answering correct or “p” value“p” value• Discrimination power of each question, e.g., biserial or Discrimination power of each question, e.g., biserial or

point biserial (point biserial (rbprbp))• Frequency of selection of each optionFrequency of selection of each option• Test performance of group selecting each option, Test performance of group selecting each option,

correctly or incorrectly.correctly or incorrectly.


MAKING SENSE OF THEMAKING SENSE OF THEPOINT BISERIAL CORRELATION COEFICIENTPOINT BISERIAL CORRELATION COEFICIENT

rrpbpb = ( = (YYc c - - YYtt)/)/S S [ [NNc c /(/(NNt t - - NNcc) ) NNtt/(/(NNtt - 1)]- 1)]1/21/2

rrpbpb == discrimination power of a question, discrimination power of a question, i.e., how well does the ranking of students on each question correlate with their ranking using their test average.

YYcc == mean test score of students answering question mean test score of students answering question

correctlycorrectly

YYtt == mean test score of all studentsmean test score of all students

SS == standard deviation of test meanstandard deviation of test mean

NNcc == number answering question correctlynumber answering question correctly

NNtt == total answering questiontotal answering question• If If YYc c > > YYtt a positive correlation apparently exists between whole a positive correlation apparently exists between whole

test population and question population. test population and question population.

• The breadth of the S has an inverse effect on the rThe breadth of the S has an inverse effect on the rpbpb..

• Population variations are weighted using the last term.Population variations are weighted using the last term.

• The magnitude of rThe magnitude of rpbpb suggests the extent of correlation of suggests the extent of correlation of

student scores on question and test.student scores on question and test.

SETTING STANDARDS FOR QUESTIONSSETTING STANDARDS FOR QUESTIONS

• Average (p value) on question is similar to test Average (p value) on question is similar to test average.average.

• Majority of students (~70%) chose correct Majority of students (~70%) chose correct answer.answer.

• All responses have been selected, i.e., no All responses have been selected, i.e., no options were easily eliminated by guessing.options were easily eliminated by guessing.

• Performance of individuals on question Performance of individuals on question correlates with their performance on whole correlates with their performance on whole test. test.

• Realistically, we often have to settle for less.Realistically, we often have to settle for less.


• Consider MCQ’s used in a high risk testing environment.Consider MCQ’s used in a high risk testing environment.• Using item analysis information, to consider two aspects Using item analysis information, to consider two aspects

• Question suitability for current high risk examination.Question suitability for current high risk examination.• Deciding potential modifications for use on a later exam.Deciding potential modifications for use on a later exam.

• Question items that are particularly useful include Question items that are particularly useful include p valuep value, , rpbrpb, frequency of selection of each option, and , frequency of selection of each option, and student student test scores choosing each optiontest scores choosing each option..

• For each question, determine the following:For each question, determine the following:• What is the p value?What is the p value?• How well did the How well did the rpbrpb for for correctcorrect answer population correlate answer population correlate

with whole test population?with whole test population?• For each question how well does the population selecting For each question how well does the population selecting

the the incorrect incorrect options correlate with whole test?options correlate with whole test?• What is the frequency of selecting each option?What is the frequency of selecting each option?

PRACTICAL USE OF ITEM ANALYSIS TO EVALUATE PRACTICAL USE OF ITEM ANALYSIS TO EVALUATE TEST QUESTIONSTEST QUESTIONS


• If p value If p value , question is likely too easy and testing only , question is likely too easy and testing only memorization.memorization.

• If p value If p value , question is likely confusing, poorly written, or , question is likely confusing, poorly written, or testing obscure information not in learning issues. testing obscure information not in learning issues. • Action step:Action step: For high or for low p values, (1) revise and For high or for low p values, (1) revise and

retain tested concept, (2) discard question, and/or (3) retain tested concept, (2) discard question, and/or (3) improve learning resources.improve learning resources.

• If rpb for If rpb for correctcorrect answer population has a poor correlation answer population has a poor correlation with whole test population with whole test population • Action step:Action step: Consider possible keying error or poorly Consider possible keying error or poorly

written question needing to be edited.written question needing to be edited.• If one or more of the If one or more of the incorrect incorrect options are too poorly or options are too poorly or

highly selected.highly selected.• Action step:Action step: Consider revisions that exploit predictable Consider revisions that exploit predictable

misconceptions.misconceptions.

GENERALITIES USING ITEM ANALYSIS INFORMATION GENERALITIES USING ITEM ANALYSIS INFORMATION


ITEM ANALYSIS INFORMATION FROM RECENT ITEM ANALYSIS INFORMATION FROM RECENT EXAMINATION (2010)EXAMINATION (2010)

• Student test scores were segregated into deciles, Student test scores were segregated into deciles, which yielded a “pseudo-normal” distribution.which yielded a “pseudo-normal” distribution.

• Mean: 74.0% (Median score: 73.7%)Mean: 74.0% (Median score: 73.7%)• Std. Dev: 6%Std. Dev: 6%• High score: 92.2%High score: 92.2%• Low score: 55.7%Low score: 55.7%• Test Reliability: 0.88Test Reliability: 0.88• OutcomesOutcomes

• PassPass: Test average must at least = 1 SD below mean, : Test average must at least = 1 SD below mean, i.e., 68%i.e., 68%

• ConcernConcern**: Test average between 1 and 2 SD below : Test average between 1 and 2 SD below mean, i.e., 62% mean, i.e., 62% < < score < 68% score < 68%

• FailureFailure**: Score : Score ≤ ≤ 62%62%• **Unit-specific remediation(s) required at end of term Unit-specific remediation(s) required at end of term


EXAMPLE 1EXAMPLE 1• p value: 0.26p value: 0.26

• A:A: 18% selected 18% selected Test score:Test score: 67.1%67.1% rpb: -0.109rpb: -0.109

• B:B: 19% selected19% selected Test score:Test score: 68.6%68.6% rpb: -0.118rpb: -0.118

• C:C: 20% selected 20% selected Test score:Test score: 72.8%72.8% rpb: -0.008rpb: -0.008

• D:D: 17% selected 17% selected Test score:Test score: 65.8%65.8% rpb: -0.088rpb: -0.088

• E:E: 26% selected26% selected Test score:Test score: 81.4%81.4% rpb: +0.339rpb: +0.339

What are the primary concerns for this question?What are the primary concerns for this question?

1.1. Are there concerns with p value for question?Are there concerns with p value for question?

2.2. What is suggested by the rpb for What is suggested by the rpb for correct answercorrect answer population? population?

3.3. Was average test performance of those selecting Was average test performance of those selecting incorrect incorrect options consistent with their test performance?options consistent with their test performance?

4.4. What was the distribution of selection of the test question What was the distribution of selection of the test question options?options?

5.5. KeepKeep (test-worthy) or (test-worthy) or discarddiscard (not suitable for current test)? (not suitable for current test)?

6.6. For later use, what potential question-specific modifications For later use, what potential question-specific modifications would you consider?would you consider?



• A:A: 02% selected 02% selected Test score: 65.6%Test score: 65.6% rpb: -0.228rpb: -0.228

• B:B: 77% selected77% selected Test score: 77.4%Test score: 77.4% rpb: +0.182rpb: +0.182

• C:C: 05% selected 05% selected Test score: 73.5%Test score: 73.5% rpb: -0.045rpb: -0.045

• D:D: 08% selected 08% selected Test score: 67.8%Test score: 67.8% rpb: -0.213rpb: -0.213

• E:E: 08% selected08% selected Test score: 71.1%Test score: 71.1% rpb: -0.159rpb: -0.159




2.2. How well does rpb for How well does rpb for correct answercorrect answer population correlate with population correlate with whole test population?whole test population?

3.3. Was average test performance of those selecting Was average test performance of those selecting incorrect incorrect options consistent with test performance?options consistent with test performance?

4.4. Was there adequate selection of test question options?Was there adequate selection of test question options?




• A:A: 18% selected 18% selected Test score: 74.0%Test score: 74.0% rpb: -0.066rpb: -0.066

• B:B: 06% selected06% selected Test score: 62.6%Test score: 62.6% rpb: -0.452rpb: -0.452


• D:D: 12% selected 12% selected Test score: 72.9%Test score: 72.9% rpb: -0.109rpb: -0.109

• E:E: 58% selected58% selected Test score: 77.0%Test score: 77.0% rpb: +0.351rpb: +0.351










• A:A: 45% selected 45% selected Test score: 76.5%Test score: 76.5% rpb: +0.209rpb: +0.209

• B:B: 05% selected05% selected Test score: 66.5%Test score: 66.5% rpb: -0.267rpb: -0.267


• D:D: 30% selected 30% selected Test score: 75.4%Test score: 75.4% rpb: +0.101rpb: +0.101

• E:E: 06% selected06% selected Test score: 70.3%Test score: 70.3% rpb: -0.171rpb: -0.171









APPENDIXAPPENDIX


GOOD TEST RESULTS ARE GENERATED BY GOOD TEST RESULTS ARE GENERATED BY BOTH GOOD TEACHING AND QUESTIONSBOTH GOOD TEACHING AND QUESTIONS

• Give the students a reasonable expectation of test materialGive the students a reasonable expectation of test material• That is, a reasonable set of objectives or expected outcomes That is, a reasonable set of objectives or expected outcomes

and references for attaining the information.and references for attaining the information.

• Use a question format, which tests information to be Use a question format, which tests information to be learned by assessing skills, facts, knowledge in a context learned by assessing skills, facts, knowledge in a context that it will be usedthat it will be used..• That is, when the student applies it as a professional.That is, when the student applies it as a professional.

• Common mistakes reducing test effectivenessCommon mistakes reducing test effectiveness• Lack of reasonable or predictable association of expectations Lack of reasonable or predictable association of expectations

and tested materialand tested material• Questions that do not require adequate understanding of Questions that do not require adequate understanding of

tested materialtested material• Poor syntax and grammatical skills makes expectations Poor syntax and grammatical skills makes expectations

difficult to predict leading to poor responses.difficult to predict leading to poor responses.


COMMON COMMON (AVOIDABLE)(AVOIDABLE) MISTAKES MISTAKES LEADING TO UNRELIABLE MCQ LEADING TO UNRELIABLE MCQ

ASSESSMENTS ASSESSMENTS • Question construction gives clues to correct Question construction gives clues to correct

answer or allows elimination of incorrect answer or allows elimination of incorrect answersanswers

• Heterogeneous or nonparallel content Heterogeneous or nonparallel content choiceschoices

• Series of True/False options with no Series of True/False options with no particular relevance to stemparticular relevance to stem

• Use of “all of the above are correct” or “none Use of “all of the above are correct” or “none of the above are correct” as answer choices.of the above are correct” as answer choices.


• EQs require significant time and effort to composeEQs require significant time and effort to compose• EQs can be difficult to effectively and subjectively grade.EQs can be difficult to effectively and subjectively grade.• The level of knowledge that can be assessed by EQs is The level of knowledge that can be assessed by EQs is

somewhat different from other types of questions. somewhat different from other types of questions. • That is, a well-designed EQ can assess conceptual clarity, That is, a well-designed EQ can assess conceptual clarity,

organizational skills, and problem solving skillss. organizational skills, and problem solving skillss. • Further, insight can be gained by the teacher into their Further, insight can be gained by the teacher into their

teaching and curriculum design effectiveness. teaching and curriculum design effectiveness. • The benefit to the student is that this type of problem-solving The benefit to the student is that this type of problem-solving

environment simulates science career experiences. environment simulates science career experiences. • EQ assessment permits the student and teacher insight into EQ assessment permits the student and teacher insight into

basic knowledge and the ability to use it with their existing basic knowledge and the ability to use it with their existing knowledge base to solve problems.knowledge base to solve problems.

• Another advantage is that EQs can be stimulating and exciting Another advantage is that EQs can be stimulating and exciting for graduate students, which could reduce test anxiety. for graduate students, which could reduce test anxiety.

• With feedback, the student can use this experience to identify With feedback, the student can use this experience to identify the status and accessibility of their knowledge, and the status and accessibility of their knowledge, and recognition of the need for improvement.recognition of the need for improvement.

COMPOSING AND ASSESSING ESSAY QUESTIONS (EQS)COMPOSING AND ASSESSING ESSAY QUESTIONS (EQS)


• Examining learning objectives to decide information is to be tested.Examining learning objectives to decide information is to be tested.• Incorporate into an EQ as many concepts from learning objectives as Incorporate into an EQ as many concepts from learning objectives as

are practical to minimize the probes needed for their assessment.are practical to minimize the probes needed for their assessment.• After deciding on concepts to evaluate knowledge for each EQ,After deciding on concepts to evaluate knowledge for each EQ,

• Determine the extent of knowledge to be required.Determine the extent of knowledge to be required.• Consider what background knowledge the student should know and Consider what background knowledge the student should know and

what you will provide.what you will provide.• Create a “scenario” that poses a problem that requires recall and use Create a “scenario” that poses a problem that requires recall and use

of the information (new and existing) to be tested. of the information (new and existing) to be tested. • To provide an ability to evaluate the student’s response, the conundrum To provide an ability to evaluate the student’s response, the conundrum

can have multiple imbedded problems and distracters of varying levels of can have multiple imbedded problems and distracters of varying levels of conceptual difficulty. conceptual difficulty.

• The goal is to measure the student’s ability to use effectively their new The goal is to measure the student’s ability to use effectively their new knowledge in concert with an existing knowledge base. knowledge in concert with an existing knowledge base.

• Provocative EQs present situations that have not been previously Provocative EQs present situations that have not been previously discussed, are likely to unfamiliar, but can be analyzed using their discussed, are likely to unfamiliar, but can be analyzed using their knowledge.knowledge.

• After reading the goal of the EQ is to provoke:After reading the goal of the EQ is to provoke: • Recall all acquired knowledge relative to the problem, Recall all acquired knowledge relative to the problem, • Assembly of the informationAssembly of the information• Coherently integration all knowledge (previous and newly acquired)Coherently integration all knowledge (previous and newly acquired)• Composition of a cogent responseComposition of a cogent response

ELEMENTS OF EQ CONSTRUCTIONELEMENTS OF EQ CONSTRUCTION


• A critical aspect of EQ construction is minimizing unintended A critical aspect of EQ construction is minimizing unintended distractions and ensuring an understanding of the breadth and depth distractions and ensuring an understanding of the breadth and depth of expected response. of expected response.

• Students should understand clearly, what problem(s) must be Students should understand clearly, what problem(s) must be considered. considered. • Remember that unlike MCQ, test takers they do not have response options Remember that unlike MCQ, test takers they do not have response options

to cue them into what is being asked. to cue them into what is being asked. • Ambiguous questions often mislead informed students.Ambiguous questions often mislead informed students.

• Expectations should be consistent with how the graduate student will Expectations should be consistent with how the graduate student will need to use the information to solve career-associated problems. need to use the information to solve career-associated problems. • Instead of fact recall, questions should probe concepts primarily or Instead of fact recall, questions should probe concepts primarily or

secondarily associated with research or work related experiences.secondarily associated with research or work related experiences.• Ambiguities can also be reduced by avoiding the use words with Ambiguities can also be reduced by avoiding the use words with

multiple usages that could be confusing, using appropriate grammar, multiple usages that could be confusing, using appropriate grammar, spelling, and punctuation errors, spelling, and punctuation errors,

• Provide clear instruction concerning how the question SHOULD and Provide clear instruction concerning how the question SHOULD and SHOULD NOT be answered. SHOULD NOT be answered. • For example, indicate that responses to be graded should be complete For example, indicate that responses to be graded should be complete

sentences and that outlines will not substitute for answers. sentences and that outlines will not substitute for answers. • Adding time expectations to answer each question is instructional.Adding time expectations to answer each question is instructional.

• Ask others to read your question for clarity. Ask others to read your question for clarity. • Ask the reader if they can tell you what you are asking of the student. Ask the reader if they can tell you what you are asking of the student. • Reviewers will not need to know the correct answer to provide feedback.Reviewers will not need to know the correct answer to provide feedback.

ELEMENTS OF EQ CONSTRUCTIONELEMENTS OF EQ CONSTRUCTION


• Prior to grading, construct for each question an outline listing Prior to grading, construct for each question an outline listing all of the components that you expect for a perfect score.all of the components that you expect for a perfect score.

• Assign relative point values for each component to obtain a Assign relative point values for each component to obtain a score.score.

• Use outline to explain to student what your expected.Use outline to explain to student what your expected.

• Use anonymity until final grading decisions are madeUse anonymity until final grading decisions are made . .

• Prior to assessment, make sets of identical questionsPrior to assessment, make sets of identical questions• Referring to the question outline, evaluate all responses before Referring to the question outline, evaluate all responses before

proceeding to grading the next question.proceeding to grading the next question.

• Using grading outline for each question has many advantages.Using grading outline for each question has many advantages. • It is easier to be consistent and fairIt is easier to be consistent and fair

• It facilitates consistent discussion of grading standards. It facilitates consistent discussion of grading standards.

• It minimizes more subtle forms of unrecognized bias.It minimizes more subtle forms of unrecognized bias.

• Depending on EQ risk (e.g., course exam or Ph.D. exam) a Depending on EQ risk (e.g., course exam or Ph.D. exam) a percent grade or pass/fail recommendation can be chosen.percent grade or pass/fail recommendation can be chosen.

• For example, if a percentage grade is not needed, student For example, if a percentage grade is not needed, student performance could be categorized as (1) meeting, (2) exceeding, performance could be categorized as (1) meeting, (2) exceeding, or (3) beneath passing standards.or (3) beneath passing standards.

EQ ASSESSMENTEQ ASSESSMENT


Documents

CREATING EFFECTIVE QUESTIONS FOR ASSESSMENT AND AS AIDS IN LEARNING IN TODAY'S PHARMACOLOGY PROGRAMS George A. Dunaway, Ph.D. Emeritus Professor Department