Systemizing Emotions: Using Interactive Multimedia as a Teaching Tool

by Ofer Golan and Simon Baron-Cohen

Chapter 10

Nonverbal Communication

Fluent social and communication skills are dependent on the ability to infer emotional and other mental states based on: Facial expressionsVocal intonationContext

Facial Emotion Recognition

Individuals with autism have more difficulty than neurotypicals or those with cognitive delays in recognizing the six basic emotions expressed in the face:

•Happiness•Sadness•Fear

•Anger•Surprise•Disgust

Facial Emotion Recognition

Individuals with Asperger Syndrome or HFA may recognize the basic emotions, but have trouble with more complex emotions such as:EmbarrassmentInsincerityIntimacy

Processing Faces Differently

Individuals with ASDs tend to process faces by looking at individual features (mouth, nose, or eyes)

Neurotypicals tend to process faces as a whole

Processing Facial Emotions Differently When individuals with ASDs process

facial emotions, there is less activation (than with neurotypicals) in regions of brain that processFaces (fusiform gyrus)Emotions (amygdala)

Vocal Recognition of Emotion Less research in this area, but several

studies have reported a deficit in high-functioning adults with ASDs, compared to neurotypicals

One neuroimaging study showed less intensive/extensive activation in the medial frontal area of the brain (used for judging others’ mental states) during an vocal emotion recognition task

Contextual Recognition of Emotion Individuals w/ ASDs have more difficulty

inferring emotional/mental state based on contextual information (i.e., “reading between the lines”)

Neuroimaging study showed reduced activation of the left medial prefrontal cortex during the Strange Stories Test*

*assesses ability to interpret nonliteral statements made by story characters, based on context

Multimodal Recognition of Emotion

In real life, we have to interpret and integrate facial, vocal, body language, and contextual information simultaneously

Individuals with ASDs show deficit in this area, as evidenced in studies that use life-like social situations (e.g., movies)

Multimodal Recognition of Emotion

Individuals with ASDs may have difficulty because they don’t pick up the correct cues, AND/OR

They don’t integrate the cues into a whole (i.e., weak central coherence), and focus too much on the detailsThis could be due to under-connectivity

among frontal brain regions

Systemizing in Autism

Individuals with ASDs show good, and sometimes superior, skills in systemizingThe drive to analyze or build systems

in order to understand and predict the behavior of rule-based events

Systemizing in Autism

Individuals with ASDs show superior abilities in tasks that requireSearching for detailAnalyzing and manipulating systems

Could high-functioning individuals w/ ASDs use their systemizing skills to compensate for their difficulties recognizing emotions?

Early Attempts to Teach Emotion Recognition

Either focused on just the basic emotions, or

Were included as part of social skills group curriculum, but didn’t focus on systemizing training

Advantages of Using Computers to Teach Emotion Recognition Computerized environment is predictable,

consistent, and free from social demands Many individuals w/ ASDs enjoy using

computers Lessons can be repeated over and over

until mastery is achieved Interest and motivation can be maintained

through different and individually selected computerized rewards

Mind Reading Software

Interactive guide to emotions and mental states Designed by Baron-Cohen, Golan,

Wheelwright & Hill (2004) Teaching tool for emotion recognition

for learners with ASDs

Mind Reading’s Systemizing Approach 412 emotions and mental states, each

defined and demonstrated by: 6 silent films of faces 6 voice recordings 6 written examples 18 x 412 = 7,416 units of emotion info

• Grouped by: 24 emotion groups 6 developmental levels (ages 4-adult)

Emotions Database Applications Emotions Library

Allows users to freely browse, compare, and add their own notes

Learning Center Structured approach to lessons, quizzes,

rewards Game Zone

5 educational games to play while studying emotions

Controlled Study of Adults Using Mind Reading Software 3 groups matched on age, verbal and

performance IQ, handedness, and gender Intervention Group: 19 adults with AS or

HFA each used software at home for 10-20 hours over 10 weeks

AS/HFA Control Group: 22 adults participated only in pre- and post- assessments, with no intervention

Typical Control Group: 28 participants

Intervention Group

Allowed to use Emotions Library and Learning Center as they wished, but told to limit games to no more than 1/3 of usage time

Encouraged to analyze facial and vocal stimuli systematicallye.g., Distinguish “insincere” from

“interested” by looking not just at smiling mouth but at eyes as well

Findings

Intervention group significantly improved their ability to recognize complex emotions/mental states from both faces and voicesImprovement was limited to taught

stimuli, (i.e., faces and voices taken from Mind Reading)

Generalization Issues

Participants found it difficult to generalize to other tasks of emotion recognition from voices and eyes

No improvement on integration of facial, vocal, and contextual cues

Software usage time was correlated with higher film task scores, so perhaps using software longer would increase generalization

Qualitative Data

Participants reported paying greater attention to faces and emotions, as well as improved eye contact

A year later, participants who continued to use Mind Reading reported having improved friendship skills

Weak Generalizing or Strong Systemizing?

Authors’ view is that reduced generalization is not so much a deficit as a reflection of the strong drive to systemize in people with ASDs

But socio-emotional world is flexible, context-dependent, and unpredictable, making it difficult--if not impossible--to systemize

Potential Problem

Attempting to systematically teach about emotions to individuals with ASDs may lead to rigid application of the system, and eventually to limited generalization

Mind Reading’s Role in an Intervention Plan

Authors recommend that Mind Reading be viewed as a first step in a training programExamples demonstrate what emotions

look like and sound like Next steps should deal with context

and integration of socio-emotional cues into one (flexible) picture

Question for Future Research

When participants improve their emotion recognition by using Mind Reading software, are regions of their brains activating in patterns more like those of neurotypicals?

Or are they using compensatory strategies in other parts of the brain?

Conclusions

Complex emotion recognition can be improved over a relatively brief length of time using Mind Reading’s systemizing approach, but …

Additional methods are required to improve generalization.