1

Tutoring and Educational Applications

• 2pm: Question Generation Based on Numerical Entities in Basque (Itziar Aldabe, Montse Maritxalar, Ander Soraluze)

• 2:25pm: A Graph Theory Approach for Generating Multiple Choice Exams (Sarah Luger)

• 2:50pm: Generating Mathematical Word Problems (Sandra Williams)

• 3:05pm: Moderated discussion (Jack Mostow)

AAAI Symposium on Question Generation, Nov. 4-6, Arlington, VA

11/4/2011 2

Questions about Questions• Target: what does it take to answer the question?• Use: why ask the question?• Question type: cloze? wh-/how/so/…? find/compare/…?• Answer type: multiple choice? fill-in? open-ended?• Generation: how to construct question, answer, distracters?• Modality: menu? click? keyboard? speech? graphics? …• Assessment: how to score answer? how to generate feedback?• Evaluation: how well does question achieve use? how to tell?• Discussion:

– Why bother – why not just use cloze? – What kinds of difficulty are good, for what? – What’s novel? – What key idea(s) transfer?

11/4/2011 3

Question Generation Based on Numerical Entities in Basque

• Target: numerical fact (measure, date, time, number)• Use: tests• Question types: Which? When? How many?• Answer type (planned): multiple choice• Generation: find numerical entity; transform sentence• Modality: unspecified• Assessment: = correct answer?• Evaluation (human): grammatical? fluent?• Discussion: better than cloze? how?

11/4/2011 4

A Graph Theory Approach for Generating Multiple Choice Exams (Sarah Luger)

• Target: unspecified• Use: tests• Question type: unspecified• Answer type: multiple choice• Generation: none; given question, answer, and 4-5 distracters• Modality: unspecified• Assessment: correct choice?• Evaluation: difficulty = fool more good than bad students• Ideas: extract complete “virtual exams” from partial test data• Discussion: What features make distracters difficult? What kinds of

difficulty are good, for what uses? What can wrong answers reveal? Can Q-matrix or other learning identify types of students and questions? Relation to work on equating scores based on different item substs?

11/4/2011 5

Generating Mathematical Word Problems (Sandra Williams)

• Target: solve word problem• Use: practice numeracy in realistic contexts• Question type: multi-sentence problem involving two entities• Answer type: number• Generation: refactor and aggregate OWL, realize in English• Modality: unspecified• Assessment: = correct answer?• Evaluation: none yet• Ideas: Manipulate 5 difficulty factors: readability, irrelevant

numbers, extraneous information, order of numbers, conceptual difficulty of math

• Discussion: What kinds of difficulty are good, for what uses?