1

Matthieu Hermet, Stan Szpakowicz

Automated Analysis of Students’ Free-text

Answers for Computer-Assisted Assessment

University of Ottawa, Canada

2

CAA for CALL

• To address the specificity of CALL…→ where student material contains

syntactic and orthographic errors• …with minimal pre-encoded material :

– Content validation : simple– Form validation : difficult

→ A good case for automating based on Natural Language Processing

3

Text comprehension

• The uOttawa project: CALL solutions for helping French-as-a-Second-Language students to enhance autonomous reading comprehension

→ master the structure of text in order to understand the author’s discursive intention

→ guess the meaning of unknown words

→ develop reformulation and synthesis capabilities

4

DidaLect

…is a FSL tool aimed at teaching autonomous reading skill (designed for intermediate- and advanced-level students)

• Is an instance of eLearning Intelligent Tutoring System:– adaptation to individual student’s skills and

agenda– access to external resources (dictionaries)– built to reflect the cognitive concerns such as

matching feedback to the student’s behaviour

5

Intelligence in DidaLect

• DidaLect begins its operation with a placement test to determine a student’s initial level: – varying order of questions to pick up the best of a

student’s skill– the implementation includes fuzzy logic methods

• A separate element of DidaLect is the processing of free-text answers:– need of a robust CAA component– a trade-off between symbolic processing and Machine

Learning techniques

6

Free-text answer assessment

• The problem is to know in advance what material to expect in student answers.

• Usually implemented as a classification problem: a student answer must match reference answer(s).

→ Size and form of reference material affects the process

• Here, a reference answer is the text itself→ A case for trying symbolic processing using

techniques of Computational Linguistics

7

Expected limitations

• No possibility of modifying the size and form of the reference material, except by automatic processing to control reformulation.

• Therefore, this only works for limited forms of questions.– Strong need to ground selection of

questions in a firm didactic theory.– Questions on texts for Text

Comprehension (didactics offers a classification of question types).

8

Question types 1

• Text-Explicit: based on a single sentence

« d'habitude, l'hermaphrodisme frappe surtout les mâles, qu'elle dote de simulacres d'appareils génitaux féminins. »

Q : Qu’arrive-t-il aux ours mâles lorsqu’ils sont frappés d’hermaphrodisme ?

Ex R : « Ils ont les génitaux féminins »

• Text-Implicit : based on two co-referenced sentences

« le détecteur de décélération situé à l'avant du véhicule génère instantanément un courant électrique, qui déclenche une amorce, qui elle-même enflamme un mélange allumeur. Ce dernier met finalement le feu à l'agent propulseur responsable du gonflement du coussin. »

Q : Quelle est la réaction en chaîne qui se produit lorsque survient un impact ?

9

Question types 2

• Identification, cause-effect, goal, comparison, definition, instrumental…

• These categories express linguistically through lexical connectors

• Goal : for, so that, in order to…• Cause–Effect : because, therefore…

• So, the control of reformulation can be automated

10

Processing

1. Find lexical differences between the student’s answer S and reference R

2. Parse S and R, produce dependency relations

3. Process different words (using a dictionary) to detect synonyms

4. Control of syntax in S

5. Control of reformulation in S wrt R

11

Tools

• A robust parser that enables partial recovery from errors in student’s answer

• A dictionary of synonyms

• A derivational dictionary

• Locally derived resources:– State and action verbs– Ensemble of typical errors, set of

syntactic and reformulation structures

12

Semantics and synonyms

• Examine word set differences and commonalities in search for:

– Common words– Reformulated words– Different words

• Detect synonyms accross parts of speech:

1. Derive forms for a word lemma2. Search synonyms for each form and

look for a match in Word Sets

13

Syntax and reformulation

• Correct syntactical structures to verify syntax of student’s answer

• Lexicalized reformulation structures to verify discourse conformity

Ex : pollution has increased with the rise of transportationQ : Why has pollution increased ?Ans : With the rise of transportation is partially wrong→ Because of the rise of transportationOR it has increased due to the rise of transportationETC.

14

Parsing and tree-building

S : Le cardio-vasculaire d'un rat s'approche à une personne humain.

SUBJ(<approche^approcher:53>, <cardio-vasculaire^cardio-vasculaire:48>)

OBJ(<approche^approcher:53>, <personne^personne:56>)

VMOD_POSIT1(<approche^approcher:53>, <une^un:55>)

NMOD_POSIT1(<cardio-vasculaire^cardio-vasculaire:48>, <rat^rat:51>)

PREPOBJ(<une^un:55>, <à^à:54>)

PREPOBJ(<rat^rat:51>, <d'^de:49>)

DETERM(<rat^rat:51>, <un^un:50>)

The above are incrementally recomposed, based on lexical selection that maximizes promise of discovering material which diverges from R. That material is processed in parallel, in a similar fashion:

SUBJ( <OBJ(<approcher>, <NMOD(<être>, <humain>)>)>,

<NMOD(<NMOD(<système>, <cardio-vasculaire>)>, <rat>)>)

15

Main types of heuristics

• To address syntactic correctness and/or equivalence between S and R: the same sense but different structures

→ bank of typical errors and correct structures• To address discursive variations, detected as

supplementary material→ bank of state and action verbs: action verbs

must be present, possibly reformulated• To address, partially, errors in S→ word replacement to relaunch parsing when

stopped due to lexical mistakes

16

Reformulation rules

• Examples :• Abstraction : incidence sur le temps de gestation →

incidence sur la possibilité d’avoir une gestation écourtée (words like fact, chance, etc.)

• Cause-Effect : Le plasma augmente et dilue les paramètres chimiques → L’augmentation du plasma dilue les paramètres chimiques

• Is-A : Le rat est un animal qui + S → Le rat + S• Attribute : Le rat possède un système cardio-

vasculaire → Le système C-V du rat

17

Assessment

• Must give student feedback on:– Agreement and orthography– Syntax: signal errors and

provides correction via display of a correct structure

– Semantics: signals error and provides admissible words

– Completeness of content with respect to R

18

Example

R: Et puisque le rat est un animal qui possède un système cardio-vasculaire très semblable à celui de l’humain, il est donc permis de tirer les mêmes conclusions pour l’humain.

Q: Pourquoi peut-on tirer les mêmes conclusions pour l'humain et pour le rat ?

S: Le cardio-vasculaire d’un rat s’approche à une personne humain.

Start by creating wordlists :

1. Words of S absent in R→ s’approcher, personne2. Words of R absent in S→ animal, posséder, système,

semblable3. Common words→ rat, cardio-vasculaire, humain

19

Parse (partial output)S:

SUBJ(<approche^approcher:53>, <cardio-vasculaire^cardio-vasculaire:48>)

OBJ(<approche^approcher:53>, <personne^personne:56>)

VMOD_POSIT1(<approche^approcher:53>, <une^un:55>)

NMOD_POSIT1(<cardio-vasculaire^cardio-vasculaire:48>, <rat^rat:51>)

PREPOBJ(<une^un:55>, <à^à:54>)

PREPOBJ(<rat^rat:51>, <d'^de:49>)

DETERM(<rat^rat:51>, <un^un:50>)

R:

SUBJ(<est^être:2>,<rat^rat:1>)

OBJ_SPRED(<est^être:2>, <animal^animal:4>)

OBJ(<possède^posséder:6>, <système^système:8>)

COREF_REL(<animal^animal:4>, <qui^qui:5>)

NMOD_POSIT1(<système^système:8>, <cardio-vasculaire^cardio-vasculaire:9>)

NMOD_POSIT1(<système^système:8>, <semblable^semblable:11>)

NMOD_POSIT1(<celui^celui:13>, <humain^humain:16>)

ADJMOD(<semblable^semblable:11>, <celui^celui:13>)

PREPOBJ(<humain^humain:16>, <de^de:14>)

PREPOBJ(<celui^celui:13>, <à^à:12>)

DETERM(<système^système:8>, <un^un:7>)

DETERM(<animal^animal:4>, <un^un:3>)

DETERM_DEF(<rat^rat:1>, <le^le:0>)

CONNECT_REL(<possède^posséder:6>, <qui^qui:5>)

20

Comparison (partial)

SUBJ(< OBJ(<approche^approcherapproche^approcher:53>,<personne^personnepersonne^personne:56>) >,<NMOD_POSIT1(<cardio-vasculaire^cardio-cardio-vasculaire^cardio-vasculairevasculaire:48>,<rat^ratrat^rat:51>)>)

SUBJ(<OBJ_SPRED(<est^être:2>,< COREF_REL(<animal^animal:4>,< CONNECT_REL(< OBJ(<possède^posséder:6>,< NMOD_POSIT1(< NMOD_POSIT1(<système^système:8>,<cardio-vasculaire^cardio-cardio-vasculaire^cardio-vasculairevasculaire:9>) >,<ADJMOD(<semblable^semblablesemblable^semblable:11>,< NMOD_POSIT1(<celui^celui:13>,<humain^humainhumain^humain:16>) >)>)>) >,<qui^qui:5>) >) >)>,<rat^ratrat^rat:1>)

• Consider structures to assess for syntactic correctness• Heuristics to put some structures into equivalence :→ Here, «Le SCV du rat» and «Le rat est un animal qui possède un

SCV» are equivalent, but expressed in different syntactic structures

21

Synonyms

• - <RESULTS>• - <DEF>• - <E L="fr">•   <W>semblable</W> •   <SW

ENC="n">00730065006D0062006C00610062006C0065</SW>

•   <P>adj.</P> •   </E>• - <DF N="1">•   <W>qui ressemble à; comparable,

similaire.</W> •   </DF>•   </DEF>• - <DEF>…..• - <WD L="fr" W="2">•   <W L="fr">approchant</W> •   <SW

ENC="n">0061007000700072006F006300680061006E0074</SW>

•   </WD>

To retrieve a synonymy relation :

1. Produce derivations for all words in List 1

2. Find matches in a synonymy basis under entries for words of List 2

3. The search process can be repeated at most once, using <DEF> lexemes

→ semblable = approchantOR→ ressembler à = s’approcher de4. In this way we can catch both

synonyms and attached prepositions.

22

Conclusions and future work

• Automation is possible, with 2 main restrictions :– « bad faith » answers– Lexical errors based on homonymy

→ as long as S contains elements of answer, S can be evaluated

• Future Work : to assemble the parts through software engineering !