Investigating the Semantic Wave in Tutorial Dialogues: An AnnotationScheme and Corpus Study on Analogy Components

Jorge Del-Bosque-Trevino1, Julian Hough, Matthew Purver2

Computational Linguistics LabCognitive Science Research Group

School of Electronic Engineering and Computer ScienceQueen Mary University of London

1QMUL Education Lab 2Jožef Stefan Institutej.delbosque, j.hough, m.purver@qmul.ac.uk

AbstractWe introduce an annotation scheme and cor-pus study to investigate the use of base andtarget components of analogies in tutorial di-alogues. We present the development of thescheme and test its final form on a corpus ofone-to-one tutorial dialogues on computer sci-ence, for which we achieve over 0.77 multi-rater inter-annotator agreement. We then an-notate data from the same corpus to investigatethe use of semantic wave structures from Legit-imation Code Theory in tutoring, and we find aregular adherence to semantic wave structuresin explanations which use analogies. We fur-ther identified different semantic wave shapesand show their distributions. We concludethat semantic waves and the novel characterisa-tion of analogical explanations in tutorial dia-logues reported in this investigation can be use-ful tools for both the analysis of human tutorialdialogue and future implementation of tutorialdialogue systems.

1 Introduction

We present an empirical study of analogy stem-ming from the goal of building a spoken dialoguesystem for computer science tutoring capable of ex-plaining concepts using analogies. While there hasbeen work investigating the use of analogy in tutor-ing, it is currently insufficiently detailed to build asystem with the ability to generate analogies in ainteractively natural way; in fact, in general thereis an insufficient understanding of how people in-teract using analogies. In this paper we focus onthe sequential unfolding of analogies by tutors onan utterance-by-utterance incremental basis. Thepaper investigates how tutors go up and down thelevel of abstraction during their explanations– astructure known as semantic waves (Maton, 2013)– with the motivation that discovering how this isdone sequentially over the dialogue could eventu-ally be transferred to an artificial tutoring agent.

The rest of the paper is as follows: in Section 2we explain the theoretical and empirical founda-tions of explanations, analogies, tutorial dialoguesand semantic waves; Section 3 then outlines thefirst principal contribution of this paper, which isthe development of an annotation scheme of baseand target components in analogies within tutoringdialogues which achieves a high inter-annotatoragreement for three annotators; Section 4 thenpresents a corpus study on dialogues annotatedusing this verified scheme to establish the patternsof base and target annotations to check the extentto which semantic wave teaching is deployed bytutors in dialogue and the distribution of differenttypes of semantic wave, followed by concluding onthe findings in Section 5.

2 Background

2.1 Analogies in ExplanationsPeople continuously search, create and evaluateexplanations (Keil, 2006; Thagard, 1989) and ourexplanatory capacity is similar to our ability to rea-son analogically (Hummel et al., 2014). Analogyis habitually interpreted as a cognitive process in-volving a target domain and a base (or ‘source’)domain, the former being the one that is being ex-plicated and the latter functioning as a different butstructurally similar domain used to communicatethe concept (Gentner, 1983; Gick and Holyoak,1983). An example of the base and target compo-nents of an analogy in an utterance can be seen in(1) where the base is underlined and the target is inbold.

(1) um the stack is a lot like a Lego set, okay?

Analogies are used extensively in explanationsin instructional texts (Barbella and Forbus, 2011)and in one-to-one tutoring sessions to explain new

concepts to students (Holyoak et al., 2001). Rea-soning with analogies is conceptualised as mappinga single source to a single target (Hummel et al.,2014).

2.2 Tutorial DialoguesHuman one-to-one tutoring has been shown to be avery effective form of instruction (Chi et al., 2001)and is considered one of the most effective meth-ods of helping students to learn. However, thereare a number of variables which could either im-prove or impede learning gains during a tutoringsession, including the domain, tutor, tutee and ses-sion structure features (Hacker et al., 2009). Un-der the category of structure-related variables, anumber of pedagogical strategies have been testedempirically for their efficacy, including direct pro-cedural instructions, direct declarative instructions,positive feedback, negative feedback, worked-outexamples and analogies (Di Eugenio et al., 2013,2009; Alizadeh et al., 2015). Evidence on the ped-agogical efficacy of using analogical explanationsshows that their presence in combination with spe-cific dialogue acts correlate positively with learninggain (Alizadeh et al., 2015). However to our knowl-edge a statistical study on the use of base and targetcomponents of analogies within human one-to-onetutorial dialogues showing the structural nuancesof how tutors unfold analogies over time has notyet been researched.

2.3 Semantic WavesThe annotation scheme and corpus study wepresent here aim to uncover patterns of base and tar-get component utilisation in the tutoring dialoguesin terms of their adherence to the the structure ofsemantic waves. This concept is part of the Legit-imation Code Theory (LCT) (Maton, 2013) andprovides an explanatory framework of what consti-tutes an effective explanation (Waite et al., 2019).According to semantic waves, the complexity ofmeanings fluctuates in terms of semantic densityand semantic gravity. Semantic density is a con-tinuum that ranges from the use of common wordsutilised with their ordinary meaning at the low-est density to the use of specialized brief terms orsymbols at the highest density. Semantic gravitycontrasts between abstract concepts and real worldexamples (Maton, 2011). For a learning episode toadhere to the semantic waves construct, it shouldstart with high density and low gravity, descendto low density and high gravity and ascend back

Figure 1: The Semantic Wave technique

to the initial state (Curzon et al., 2018), as illus-trated in the diagram in Fig 11. In Fig. 1 initiallythe concept of algorithms is presented abstractly asprecise sequences or steps, then comparison to in-structions or recipes is made to unpack the meaningas the semantic density is reduced and a more con-crete or simpler base concept is used, followed bya repacking of meanings to go back to the abstractand complex meaning originally presented. In theanalogical explanations we study here, we con-sider the base component to be used at the troughof the wave, and the target component to be atthe two peaks. The annotation scheme and corpusstudy described below seek to answer the questionas to whether analogical explanations adhere tothe semantic wave structure in tutoring dialoguesempirically, and therefore whether the theoreticalconstruct is useful for modelling human-human tu-toring dialogue and for designing tutorial dialoguessystems.

3 Analogy Annotation Scheme

3.1 CorporaWe used 3 different corpora for developing an anal-ogy annotation scheme, which are the British Na-tional Corpus (BNC1994), the Basic Electricityand Electronics Corpus (BEEC) and Computer Sci-ence Tutorial Dialogues (CSTD). We selected sub-corpora from these three sources which have thefollowing characteristics:

1From the National Centre of Computing PedagogyQuick Read ‘Improving Explanations and learning ac-tivities in computing using semantic waves’, https://raspberrypi-education.s3-eu-west-1.amazonaws.com/Quick+Reads/Pedagogy+Quick+Read+6+-+Semantic+Waves.pdf

British National Corpus (BNC1994) We usedone career orientation dialogue of 700 utterancesfrom the British National Corpus (Burnard, 2000),obtained with the SCoRE tool (Purver, 2001).

Basic Electricity and Electronics Corpus(BEEC) We used a dialogue excerpt of 15 utter-ances and one entire dialogue of 292 utterancesof tutorial dialogue of the BEEC corpus (Litmanet al., 2009) also obtained with SCoRE.

Computer Science Tutorial Dialogues (CSTD)The largest and principal corpus we use is that of tu-torial dialogues on computer science data structurescollected in the late 2000’s (Di Eugenio et al., 2009)consisting of 54 one-to-one tutoring sessions onthe topics of linked lists, stacks and binary searchtrees. The corpus contains a total of 35,609 utter-ances annotated with tags signaling beginning andending of the pedagogical strategies of feedback,worked-out example and analogy. We created asubcorpus of the utterances within all the analogi-cal episodes plus a context of five utterances beforethe beginning and after the end of the episodes.Our subcorpus contains a total of 3,887 utterances–the size of our subcorpus relative to the size of thewhole corpus is as in Table 1.

Utterances withinanalogical episodes 2,528 7.10%

+ context 1,359 3.81%Utterances outsideanalogical episodes 31,722 89.09%

Total 35,609 100.00%

Table 1: Analogical episodes comprising the CSTDsub-corpus as percentage of whole corpus.

3.2 Base and Target Annotation SchemeDevelopment

The development of our annotation scheme in-cluded the participation of six researchers, two ofwhom are the first two authors of this paper. Fiveof them participated in the annotation exercises.The following paragraphs explain the settings andresults of each iteration.

3.2.1 Iteration 1For the first iteration, two non-native English speak-ers annotated the BEEC subcorpus described inSection 3.1. The annotators were instructed to markeach utterance as including only the base (B), only

the target (T), both (BT) or none of the analogycomponents. Before the annotation exercise, theannotators were provided with an annotated ex-ample of the career orientation dialogue from theBNC1994 subcorpus, which was 700 utteranceslong, of which 25 were annotated with B, T or BT.Each annotator then executed a practice run with anexcerpt of a BEEC dialogue of 15 utterances withreal-time feedback from the main author. Duringthe interactive practice, the disagreements were dis-cussed with a twofold purpose; identify any annota-tion rule which elicited disagreements and creatingnew rules if needed. Every modification and newrules were made explicit in an updated version ofthe manual. After the practice sessions, utilisingthe updated version of the annotation scheme man-ual and in individual sessions, they annotated theother BEEC dialogue comprised of 292 utterances.

The two-way Cohen’s Kappa (Siegel and Castel-lan, 1988) inter-annotator agreement results are asin Table 2 where G is the Gold Standard we as-sume, which are the annotations by the first author,compared against annotators A1 and A2. The right-hand column also gives the Fleiss’ Kappa multi-rater agreement of all three annotators reaching amoderate agreement level of 0.544.

G&A1 G&A2 G&A1&A2B, T, BT, N 0.644 0.304 0.544

Table 2: Kappa inter-annotator agreement on Iteration1.

Table 3 shows a tutoring dialogue in which thethree annotators agree about all the utterances con-taining the analogy component of type base. In thiscase, the tutor explains electrical potential energy(the target domain) by referring to a ball tossed inthe air (the base domain).

Table 4 shows a dialogue excerpt in which thetwo annotators agree about all the utterances con-taining the analogy component of type target. Inthis case, the tutor explains conservation of energy.

Finally, Table 5 shows a tutoring dialogue inwhich the two annotators agree about all the utter-ances containing both analogy components of typebase and target while the third annotator marks thelast two of these as only being base.

In addition to the potential ambiguity betweenboth (BT) and base (B), one of the most frequentsources of disagreement in this first iteration wasdue to not considering anaphoric references to base

P Utterance G A1 A2

TThink of a ball tossedinto the air.

B B B

At first the upward forcecaused byyour hand throwing itcauses it to move up.

B B B

But eventually it stops -gravity causes it to slowdown until it stops.

B B B

Then it falls down. B B B

Table 3: All three annotators in Iteration 1 agree onbase annotation. P = participant.

P Utterance G A1 A2

TAgain, energy wouldbe conserved.

T T

You just have to thinkwhat that energywas converted into.

T T B

Some of it would beconverted into heatbecause of the friction, etc.

T T B

Table 4: Two annotators agree on target.

P Utterance G A1 A2

T

You’re right that kineticenergy was zero, but atthe maximum hight,when the ball stops,the height makes it possiblefor it to start moving again.

BT BT

Now it’s going to startmoving in the oppositedirection.

B B

So that height, since it willmake it possible forthe ball to move,is a form of energy.

BT BT B

It’s the total energythat is conserved,not the kinetic energy,since the velocity ofthe ball is not constant.

BT BT B

Table 5: Two annotators agree on 4 utterances whilethe other annotator disagrees with them.

and target components and a lack of consistencyabout marking implicit references to components.As shown in the excerpt of the disagreement analy-sis of this first iteration on Table 6, the last utterancerefers to the base, in this case “a ball tossed in theair", which is only marked as such by A1.

P Utterance G A1 A2

TThink of a ball tossedinto the air.

B B B

At first the upward forcecaused byyour hand throwing itcauses it to move up.

B B B

But eventually it stops -gravity causes it to slowdown until it stops.

B B B

Then it falls down. B B BBut at every point,is not the energythe same?

BT BT

S except for when it stops. B

Table 6: Excerpt of disagreement in Iteration 1

3.2.2 Iteration 2For the second iteration, two monolingual nativeEnglish speakers were recruited as annotators, withthe purpose of increasing the inter-coder agreement.The set-up was adjusted such that annotators hadto decide whether each utterance contained a base(B) or not as a binary decision, and also whetherthe utterance contained a target (T) or not. Theannotators coded the same BEEC dialogue of 292utterances used in iteration 1 and received the samecoding rules, with the addition of the rule that con-siders anaphora. They were provided with the sameannotated example which was provided as per theprevious iteration and also executed the practiceannotation with the main author giving live feed-back on their decisions. This session allowed fordiscussion and clarification of the rules in the pro-vided manual. As in iteration 1, all changes wereregistered in an updated version of the annotationmanual. The results from iteration 2 on the twolabels are as in Table 7 (again with Cohen’s Kappafor the pair-wise agreement and Fleiss’ Kappa forthe three-way multi-rater agreement).

While very high agreement is reached on thebase component, there was large disagreement onidentifying target utterances, particularly the agree-

G&A1 G&A2 G&A1&A2B 0.878 0.880 0.807T 0.615 0.211 0.140

Table 7: Kappa inter-annotator agreement from Itera-tion 2.

ment between the gold standard annotation (firstauthor) and annotator A2.

3.3 Final Annotation SchemeFor the third and final iteration, we used the CSTDcorpus for both; the practice and the disagreementand language interpretation experiment, the mainreason being the fact that the CSTD was the corpuswe wanted to do the study of the semantic waveson. Another change in this iteration was the substi-tution of one of the two monolingual native Englishspeakers.

The disagreements with A1 and A2 from iter-ation 2 were discussed and the manual updatedaccordingly. The annotators coded a new dialogueusing the final version of the annotation manualbased on these insights. The definitions and exam-ples given to annotators for annotating base andtarget components is as in Fig 2. An expanded ver-sion of the instructions are as in the Appendix A.

The annotators were provided with an annotatedexample of 6 analogical episodes from the CSTDcorpus consisting of 193 utterances. The annota-tors executed a practice annotation exercise withanother selection of 3 analogical episodes and atotal of 116 utterances of the same subcorpus andreceived feedback from the main author with thepurpose of clarifying their questions when theyjudged the annotation rules did not fit in a particu-lar case. Once this practice was executed, to test theagreement the annotators annotated a new selectionof 5 analogical episodes for a total 188 utterances ofthe CSTD corpus. The final inter-annotator agree-ment Kappa results are as in Table 8, where it canbe seen a strong overall agreement with all threeannotators is reached for base and target at over77%.

4 Corpus Study on Analogical Episodesin Tutorial Dialogues

With the appropriate level of agreement reached,the main author annotated the entire subcorpus of

G&A1 G&A2 G&A1&A2B 0.886 0.731 0.779T 0.735 0.775 0.772

Table 8: Final Kappa inter-annotator agreement results

the CSTD analogical episodes of 3,887 utteranceswith the scheme described above (each utterancewith the two binary decisions for base and targetpresence) which contains all the sections annotatedas analogical episodes and, additionally, the singleanalogies (Di Eugenio et al., 2009) which con-sists of individual utterances, and in both casesthe 5-utterance context window either side of theanalogical utterances.

4.1 Descriptive statistics of all episodesThe histogram in Fig. 3 shows the distribution oflengths of the analogical episodes in number ofutterances.

Table 9 and Fig. 4 show the distribution of utter-ances labelled as containing only base, only target,both or no analogy components, derived from thetwo binary labels B and T. Note that 33% of ut-terances contained no analogical components andmainly communicative grounding types of dialogueacts. For the analysis that follows, and consis-tently with the concept of question under discus-sion (Ginzburg, 2012), we assume those to have aB, T or BT component still under discussion basedon the most recent label in the dialogue.

N B T BT Total1269 1302 859 457 388732.65% 33.50% 22.10% 11.76% 100%

Table 9: Distribution of analogy components labeled innumber of utterances and percentage of corpus.

4.2 Validation of Semantic Wave Structuresin Analogical Explanations

As discussed in the introduction we aimed to testwhether analogical episodes in tutoring dialoguesadhere to the structure of semantic waves fromLegitimation Code Theory (LCT) empirically. Weused the base and target component annotations tovalidate whether analogical explanations do in factadhere to this structure and if they do, what thedistributions of different types of wave might be.

Analogy definition and examples:An analogy is a linguistic device that uses a specific concept (a base*) to transfer information or meaningto another concept (a target*). Tutors use analogies to explain concepts. The following examples containa section of a tutoring dialogue which includes an analogy. The base concept is formatted with underlinedcharacters and the target concept is formatted with bold characters.

Utterance Examples Base Target B Tum the stack is a lot like a Lego set, okay? lego set stack 1 1uh a binary search tree is +// binary search tree 1one way of looking at it is like a family tree. family tree 1the comparison I like to make with linked lists is amovie line.

movie line linked lists 1 1

Figure 2: Annotation definitions and examples for Base and Target for Annotators

Figure 3: Distribution of analogical episodes length innumber of utterances

Figure 4: Distribution of analogy components

Vis-a-vis the complexity of meaning in semanticwaves, we map the concepts of linked lists, binarysearch trees and stacks, our target analogical com-ponents, to the notion of low gravity (i.e. abstractconcept), and the references to people queing atmovie theaters, family members, restaurant trays,sheets of paper and other tangible examples, whichare our base analogical components, to the notionof high gravity (i.e. concrete concept) in semanticwaves.

We would expect analogical episodes to beginwith the target component of an analogy, descendin terms of semantic gravity to the base compo-nent, and then ascend again to the target conceptat the end of the episode– see Section 2 for details.Here we define a semantic wave as any descent interms of semantic gravity between utterances andrise again, so, for instance beginning with a utter-ance with both base and target (BT), descendingto base (B) and then back to BT is still countedas a wave. Dialogues excerpts exemplifying thesemantic waves can be seen in Tables 10, 11 and12.

To test this, we automatically searched for se-mantic waves in the 138 analogical episodes of ourCSTD subcorpus and we found that 129 (93.47%)of them contained at least one, which supports theidea tutors use the semantic wave in analogicalepisodes. We also found that a mean of 2.1 wavesare used in every analogical episode in a tutorialdialogue explanation.

64 of the 138 episodes had at least two consec-utive semantic waves. The distribution over thenumber of waves per episode can be seen in Fig. 5.One episode contained 16 consecutive waves.

We additionally found that there were seven

Figure 5: Distribution of analogical episodes density innumber of semantic waves

Figure 6: Point Break Wave (T-B-T)

Figure 7: Point Break Wave (descending) (T-BT-B-T)

main types of waves which represent different pat-terns of base and target components. We take fromthe surfing domain the names of the types of waves,which vary from strong to weak. The strongest isthe point break wave as shown in Fig. 6, and in oursequence model it represents starting the analogi-cal episode with a target component, descending

Figure 8: Point Break Wave (ascending) (T-B-BT-T)

Figure 9: Reef Break Ascending Wave(BT-B-T)

Figure 10: Reef Break Descending Wave (T-B-BT)

Figure 11: Reef Break Standard Wave (T-BT-T)

to the base component at some point during the

Figure 12: Beach Break Wave (BT-B-BT)

Figure 13: Distribution of semantic wave types

episode, and finishing with the target again. Twosub-types of point break wave were also observed,namely the point break descending and point breakascending– see Figs. 7 and 8. The next type ofwave is moderate in intensity and is called a reefbreak, which can be ascending (Fig. 9), descending(Fig. 10) and standard (Fig. 11). Finally, the weak-est type of wave was observed, the beach breakwave (Fig. 12).

In total the 291 waves existing across the 138episodes were distributed by type is as in Figure 13.Example dialogue excerpts showing a point break,reef break (standard) and beach break wave can beseen in Tables 10, 11 and 12.

5 Discussion and Conclusion

We have presented, to our knowledge, the first anno-tation scheme and corpus study which investigateshow the base and target components of analogiesare deployed by human tutors during their expla-nations. We used the annotation scheme to verifywhether analogical explanations follow the struc-ture of semantic waves, whereby they begin fromthe target component, descend to the base com-

Tutor alright, stack is a very simpledata structure.

T

Tutor um, this is a shorter and shorterstack of paper.

B

Tutor and it has a top sheet. BTutor you can pick up the top sheet or

you can put another sheet on thetop.

B

Tutor so the stack +// lets make up onehere that has x@l in it, and d@land p@l and q@l how aboutthat?

T

Tutor so here is a stack of four ele-ments.

T

Tutor here are the operations you canapply to a stack.

T

Tutor you can pop it. TTutor and when you pop the stack,

that’s a destructive function thatreturns the top element.

T

Table 10: Point Break Wave Dialogue Example

Tutor it’s destructive it takes the q@loff and gives you the top elementwhat I call n@l xxx xxx.

T

Tutor uh the insert is called a push andthese come from the spring pop-ping this thing up and the springpushing down and you push onthe stack some value n@l.

BT

Tutor so this is a function that returnswhat popped off and that’s a voidfunction that takes the thing tomake it come off.

T

Table 11: Reef Break (standard) Wave Dialogue Exam-ple

ponent and return to the target. 93.47% of theepisodes contain the structure of a semantic waveand, 74% of the episodes used a series of semanticwaves consecutively. We showed there are a varietyof different wave types used and we define someshapes to understand these different structures. Weclaim this novel characterisation of analogical ex-planations in tutoring dialogues to be a formalisa-tion that could be used as a tool for both; the designof human tutorial dialogue pedagogical strategiesand intelligent tutoring systems.

Tutor *uh a binary tree is kind of likemother and father and xxx

BT

Student a family tree. BTutor no that’s not bad *uh that’s bad. NTutor it’s +// because families can have

more than two kids.B

Tutor so here what it means is that bi-nary is that each node can havetwo trees, two children.

BT

Table 12: Beach Break Wave Dialogue Example

In future, we intend to further investigate thesemantic wave structure in analogies in a more fine-grained level, to analyse the mapping between thebase and target domains which happens dynami-cally during the semantic wave explanation and toincorporate and test these explanatory models in aspoken dialogue system.

Acknowledgements

Del-Bosque-Trevino is partially supported by EP-SRC and AHRC Centre for Doctoral Training inMedia and Arts Technology (EP/L01632X/1) andCONACYT (National Council of Science and Tech-nology of Mexico). Purver is partially supportedby the EPSRC under grant EP/S033564/1, and bythe European Union’s Horizon 2020 programmeunder grant agreements 769661 (SAAM, Support-ing Active Ageing through Multimodal coaching)and 825153 (EMBEDDIA, Cross-Lingual Embed-dings for Less-Represented Languages in EuropeanNews Media). The results of this publication re-flect only the authors’ views and the Commissionis not responsible for any use that may be madeof the information it contains.Thanks to ThomasKaplan, Louise Bryce, Shamila Nasreen, MortezaRohanian and Jack Ratcliffe for their participationin the development of the annotation scheme andannotations.

ReferencesMehrdad Alizadeh, Barbara Di Eugenio, Rachel Hars-

ley, Nick Green, Davide Fossati, and Omar AlZoubi.2015. A study of analogy in computer science tu-torial dialogues. In Proceedings of the 7th Inter-national Conference on Computer Supported Edu-cation, volume 2, pages 232–237. SCITEPRESS-Science and Technology Publications, Lda.

David Michael Barbella and Kenneth D Forbus. 2011.Analogical dialogue acts: Supporting learning byreading analogies in instructional texts. In Twenty-Fifth AAAI Conference on Artificial Intelligence.

Lou Burnard. 2000. Reference Guide for the BritishNational Corpus (World Edition). Oxford Univer-sity Computing Services http://www.natcorp.ox.ac.uk/docs/userManual/.

Michelene TH Chi, Stephanie A Siler, Heisawn Jeong,Takashi Yamauchi, and Robert G Hausmann. 2001.Learning from human tutoring. Cognitive science,25(4):471–533.

Paul Curzon, Peter McOwan, J Donohue, SeymourWright, and William Marsh. 2018. Teaching com-puter science concepts. In Sue Sentance, EricBarendsen, and Carsten Shulte, editors, ComputerScience Education: Perspectives on Teaching andLearning in School, chapter 8, pages 91–108.Bloomsbury Publishing, London.

Barbara Di Eugenio, Lin Chen, Nick Green, DavideFossati, and Omar AlZoubi. 2013. Worked out ex-amples in computer science tutoring. In Interna-tional Conference on Artificial Intelligence in Edu-cation, pages 852–855. Springer.

Barbara Di Eugenio, Davide Fossati, Stellan Ohlsson,and David Cosejo. 2009. Towards explaining effec-tive tutorial dialogues. In Annual Meeting of theCognitive Science Society, pages 1430–1435.

Dedre Gentner. 1983. Structure-mapping: A theo-retical framework for analogy. Cognitive science,7(2):155–170.

Mary L Gick and Keith J Holyoak. 1983. Schema in-duction and analogical transfer. Cognitive Psychol-ogy.

Jonathan Ginzburg. 2012. The interactive stance. Ox-ford University Press.

Douglas J Hacker, John Dunlosky, and Arthur CGraesser. 2009. Handbook of metacognition in ed-ucation. Routledge.

Keith James Holyoak, Dedre Gentner, and Boicho N.Kokinov. 2001. Introduction: The place of anal-ogy in cognition. In Dedre Gentner, Keith JamesHolyoak, and Boicho N. Kokinov, editors, The Ana-logical Mind, pages 1–19. MIT Press, Cambridge,MA.

John E. Hummel, John Licato, and Selmer Bringsjord.2014. Analogy, explanation, and proof. Frontiers inHuman Neuroscience, 8:867.

Frank C Keil. 2006. Explanation and Understanding.Annual Review of Psychology, 57(1):227–254.

Diane J. Litman, Johanna D. Moore, MyroslavaDzikovska, and Elaine Farrow. 2009. Using natural

language processing to analyze tutorial dialogue cor-pora across domains modalities. In Proceedings ofthe 14th International Conference on Artificial Intel-ligence in Education, pages 149–156, Netherlands.IOS Press.

Karl Maton. 2011. Theories and things: the semanticsof discipinarity. In F Christie and Karl Maton, edi-tors, Disciplinarity: functional linguistic and socio-logical perspectives, pages 62–84. Continuum, Lon-don.

Karl Maton. 2013. Making semantic waves: A key tocumulative knowledge-building. Linguistics and ed-ucation, 24(1):8–22.

Matthew Purver. 2001. Score: A tool for searching thebnc.

S Siegel and Castellan. 1988. Nonparametric Statis-tics for the Behavioral Sciences. McGraw-Hill BookCo.

Paul Thagard. 1989. Explanatory coherence. Behav-ioral and brain sciences, 12(3):435–467.

Jane Waite, Karl Maton, Paul Curzon, and LucindaTuttiett. 2019. Unplugged computing and semanticwaves: Analysing crazy characters. In Proceedingsof the 1st UK & Ireland Computing Education Re-search Conference, pages 1–7.

A Annotation Instructions

Purpose of the annotation excercise The mainpurpose of this annotation protocol is to identifyinteractive patterns of explanations analogies inhuman-human tutoring conversations. This studywill be conducted on a dataset consisting of 54 one-to-one human basic computer science tutoring dia-logues collected in the late 2000’s. The dialoguestopics are limited to three basic computer science(CS) data structures, which are: stacks, linked listsand binary search trees

Analogy definition and examples An analogyis a linguistic device that uses a specific concept(a base*) to transfer information or meaning to an-other concept (a target*). Tutors use analogies toexplain concepts. The following examples containa section of a tutoring dialogue which includes ananalogy. The base concept is formatted with under-lined characters and the target concept is formattedwith bold characters.

Utterance Examples Base Targetum the stackis a lot like a Lego set,okay?

lego set stack

uh a binary search treeis +// one way of lookingat it is like a family tree.

family treebinarysearchtree

the comparison I like tomake with linked lists is amovie line.

movie linelinkedlists

Analogies are easily tractable when they are di-rectly observable, they can be denoted by the pres-ence of particular words or combination of wordsor keywords which depict either the base or thetarget of the analogy. Occasionally, analogies arestated in an utterance by the use of coreference.Analogies which are alluded by the use of corefer-ence should also be annotated.

CHILDES notation The research group whichannotated the 54 dialogues used the CHILDES no-tation. The following table contains the symbolsyou might encounter during your analysis. Use thistable as a reference when doing your annotations.

Anaphora, cataphora and refering nounphrases Some analogy bases and analogy targetsare alluded indirectly by the use of coreference.

Example Marker Meaninguh# you knowwhat a linked list is?

#pause betweenwords

it’s a +// it’s aconcept, not alanguage thing.

+// self interruption

so all you’regiven is thisheader,that why h@lis here.

h@l the letter h

<*uh, and theywant us>[//]oh O_K_.

<>

angle bracketsgroup wordsmarked bythe followingsymbol,in this case,retracingwith correction

<*uh, and theywant us>[//]oh O_K_.

[//]retracing withcorrection

then you’re losingall your xxx.

xxxunintelligiblespeech

yeah, but yeah,then you know +...

+...

The trailing offor incompletionmarker (plussign followed bythree periods)is the terminatorfor anincomplete,butnot interrupted,utterance.

<the second onewants>[///]so that wasan insertion,the secondone is a deletion.

[///]retracing withreformulation

(be)cause the xxx. (be)noncompletionof a word

you got to starthere at the root,just like in<a binary>[//] in a linked listyou have to startat the first node.

[//]retracingwith correction

++ right. ++ other completion

star+wars +compound orrote form marker

The three cases of coreference that we shouldbe able to spot and annotate are called anaphora,cataphora and coreferring noun phrases.

Continuation and preambles Some utterancescontain few words (4 or less) and are continuationsof the previous utterance of the same speaker, ora preamble of the following utterance. Assign thesame annotations that you gave to that speaker’sprevious or subsequent utterance.

Session management Session management ut-terances should not be considered or annotated.

Utterance Example (Metacognition)let’s start offlet me just grab a clean sheet of papper

Metacognition Utterances which are observa-tions or reflections of the mental processes thatoccur during the tutoring session should not beconsidered or annotated.

Utterance Example (Session Management)OK, now you are on to somethingI’m so happy that you understand nowIt’s good that you recognize that your answer doesn’t look right.That shows you ’re thinking!I like to see that

Additional considerations

• An utterance can be allocated as base and tar-get at the same time.

• Rely only on the text.

• Within the linked lists analogies, tutors andstudents sometimes refer to letters as if theywere people (e.g. b@l is looking to c@l). Ifthis is the case, annotate as BASE. Referenc-ing letters does not determine that speakersare talking about the TARGET

• Annotate as TARGET when there is an ex-plicit reference to concepts within the TAR-GET domain.