Bodong Chen (@bodong_c)Assistant Professor, University of Minnesota

September 9, 2015, KBSI2015, Trieste, Italy

Towards Multimodal Indicators of Idea Improvement: A Review of Possibilities

Idea Improvement in KB

▷ Why important?

▷ What is idea improvement?

Existing indicators

▷ Scientific sophistication (Zhang 2007)

▷ Levels of questioning (Law 2013)

▷ Lexical measures (Zhang & Sun 2011)

▷ Connectedness of concepts (Oshima 2013)

▷ ...

Multimodality

Multimodal interaction▷ Human-Computer Interaction▷ Multimodal Learning Analytics

Multiple modalities mediating meaning-making▷ Communication▷ Linguistics

Goal of this presentation: Explore emerging possibilities from other communities, with attention to both conceptions.

Two directions

Mining multimodal data in existing KB environments▷ Lexical▷ Syntactic▷ Graphic

Designing next-generation multimodal KB environments▷ Sociometrics▷ Mobile sensing

1. Mining multimodal data in existing environments

1.1. Lexical ▷ Text 1 – by a Grade 6 student: “My theory is solar eclipses are when the moon,

earth and sun are in a perfect line. Why solar eclipses don't happen every month is because the moon's axis around the earth is slightly tilted. The moon needs to be so close to the earth so that it covers the space of the sun. You have to be in a certain spot in the earth to see a full solar eclipse but there is an arc the sun and moon make across the earth but from other spots on the arc you would only see a partial eclipse.”

▷ Text 2 – from Wikipedia: “As seen from the Earth, a solar eclipse is a type of eclipse that occurs when the Moon passes between the Sun and Earth, and the Moon fully or partially blocks the Sun. This can happen only at new moon, when the Sun and the Moon are in conjunction as seen from Earth in an alignment referred to as syzygy. In a total eclipse, the disk of the Sun is fully obscured by the Moon. In partial and annular eclipses, only part of the Sun is obscured.”

Text 1 Text 2

Sentence count, number of sentences 4 4

Word count, number of words 100 87

Sentence length, number of words, mean 25 21.75

Word length, number of letters, mean 3.75 4

Text Easability PC Narrativity, z score 0.54 0.08

WordNet verb overlap 0.61 0.30

LSA overlap, adjacent sentences, mean 0.56 0.63

Causal verb incidence 10 34.48

1.2. Syntactic Text 1 Text 2

Words before main verb, mean 3.5 7

Sentence syntax similarity, adjacent sentences, mean 0.09 0.24

Noun phrase density, incidence 320 333.33

Verb phrase density, incidence 220 172.41

Adverbial phrase density, incidence 10 45.98

Preposition phrase density, incidence 100 172.41

Xerox Incremental Parser

Source: CIC, UTS

1.3. Graphic

▷ Layout of note network / view structure

low centralization; long path length

vs.

high centralization; short path length

x

y

1.3. Graphic

▷ View “landscape”

▷ Organizing behaviors by students

Other possibilities

▷ Affective states reflected in logs(Baker 2010; McDaniel 2007; Pardos 2013)

▷ Certainty and identity in text (Blikstein 2013)

▷ Sentiment in text (Chamlertwat 2012)

2. Designing multimodal knowledge building

environments

Motivations

▷ Pervasive knowledge building▷ “One discourse, multiple entry points”▷ Environments mediate knowledge

representation and meaning-making

2.1. Sociometrics

(Onnela et al., 2014)

Physical proximity(radio signal)

Speech time(microphone)

Body movement(accelerometer)

Social intimacy

Talkativeness Turn-takingPersonality

Energy level

2.2. Mobile & mobile sensing

▷ Multimedia○ Citizen science (Riesch, 2014)

▷ Multimodal sensors▷ Context awareness

○ Location○ Surroundings○ Proximity

▷ Augmented reality

Towards a holistic view

▷ “The Shadow Rabbit”▷ Bridge different modalities

(Shum, 2015)

Scatterplot of reading and writing activities in six classes

Towards a holistic view

▷ “The Shadow Rabbit”▷ Bridge different modalities

▷ Support multimodal representations of ideas

(Shum, 2015)

“Thank you!

Slides: https://goo.gl/pRxuV9Email: chenbd@umn.edu

Twitter: @bodong_c