View
448
Download
0
Category
Preview:
Citation preview
Bodong Chen (@bodong_c)Assistant Professor, University of Minnesota
September 9, 2015, KBSI2015, Trieste, Italy
Towards Multimodal Indicators of Idea Improvement: A Review of Possibilities
Idea Improvement in KB
▷ Why important?
▷ What is idea improvement?
Existing indicators
▷ Scientific sophistication (Zhang 2007)
▷ Levels of questioning (Law 2013)
▷ Lexical measures (Zhang & Sun 2011)
▷ Connectedness of concepts (Oshima 2013)
▷ ...
Multimodality
Multimodal interaction▷ Human-Computer Interaction▷ Multimodal Learning Analytics
Multiple modalities mediating meaning-making▷ Communication▷ Linguistics
Goal of this presentation: Explore emerging possibilities from other communities, with attention to both conceptions.
Two directions
Mining multimodal data in existing KB environments▷ Lexical▷ Syntactic▷ Graphic
Designing next-generation multimodal KB environments▷ Sociometrics▷ Mobile sensing
1. Mining multimodal data in existing environments
1.1. Lexical ▷ Text 1 – by a Grade 6 student: “My theory is solar eclipses are when the moon,
earth and sun are in a perfect line. Why solar eclipses don't happen every month is because the moon's axis around the earth is slightly tilted. The moon needs to be so close to the earth so that it covers the space of the sun. You have to be in a certain spot in the earth to see a full solar eclipse but there is an arc the sun and moon make across the earth but from other spots on the arc you would only see a partial eclipse.”
▷ Text 2 – from Wikipedia: “As seen from the Earth, a solar eclipse is a type of eclipse that occurs when the Moon passes between the Sun and Earth, and the Moon fully or partially blocks the Sun. This can happen only at new moon, when the Sun and the Moon are in conjunction as seen from Earth in an alignment referred to as syzygy. In a total eclipse, the disk of the Sun is fully obscured by the Moon. In partial and annular eclipses, only part of the Sun is obscured.”
Text 1 Text 2
Sentence count, number of sentences 4 4
Word count, number of words 100 87
Sentence length, number of words, mean 25 21.75
Word length, number of letters, mean 3.75 4
Text Easability PC Narrativity, z score 0.54 0.08
WordNet verb overlap 0.61 0.30
LSA overlap, adjacent sentences, mean 0.56 0.63
Causal verb incidence 10 34.48
1.2. Syntactic Text 1 Text 2
Words before main verb, mean 3.5 7
Sentence syntax similarity, adjacent sentences, mean 0.09 0.24
Noun phrase density, incidence 320 333.33
Verb phrase density, incidence 220 172.41
Adverbial phrase density, incidence 10 45.98
Preposition phrase density, incidence 100 172.41
Xerox Incremental Parser
Source: CIC, UTS
1.3. Graphic
▷ Layout of note network / view structure
low centralization; long path length
vs.
high centralization; short path length
x
y
1.3. Graphic
▷ View “landscape”
▷ Organizing behaviors by students
Other possibilities
▷ Affective states reflected in logs(Baker 2010; McDaniel 2007; Pardos 2013)
▷ Certainty and identity in text (Blikstein 2013)
▷ Sentiment in text (Chamlertwat 2012)
2. Designing multimodal knowledge building
environments
Motivations
▷ Pervasive knowledge building▷ “One discourse, multiple entry points”▷ Environments mediate knowledge
representation and meaning-making
2.1. Sociometrics
(Onnela et al., 2014)
Physical proximity(radio signal)
Speech time(microphone)
Body movement(accelerometer)
Social intimacy
Talkativeness Turn-takingPersonality
Energy level
2.2. Mobile & mobile sensing
▷ Multimedia○ Citizen science (Riesch, 2014)
▷ Multimodal sensors▷ Context awareness
○ Location○ Surroundings○ Proximity
▷ Augmented reality
Towards a holistic view
▷ “The Shadow Rabbit”▷ Bridge different modalities
(Shum, 2015)
Scatterplot of reading and writing activities in six classes
Towards a holistic view
▷ “The Shadow Rabbit”▷ Bridge different modalities
▷ Support multimodal representations of ideas
(Shum, 2015)
“Thank you!
Slides: https://goo.gl/pRxuV9Email: chenbd@umn.edu
Twitter: @bodong_c
Recommended