Affective Storytelling

  • View
    218

  • Download
    3

Embed Size (px)

DESCRIPTION

storytelling

Text of Affective Storytelling

  • Affective StorytellingAutomatic Measurement of Story Effectiveness fromEmotional Responses Collected over the Internet

    Daniel McDuffPhD Proposal in Media Arts & Sciences

    Affective Computing Group, MIT Media Labdjmcduff@media.mit.edu

    June 6, 2012

    Executive SummaryEmotion is key to the effectiveness of narratives and storytelling whether it be in influencing

    memory, likability or persuasion. Stories, even if fictional, have the ability to induce a genuineemotional response. However, the understanding of the role of emotions in storytelling and ad-vertising effectiveness has been limited due to the difficulty in measuring emotions in real-lifecontexts. Video advertising is a ubiquitous form of a short story usually 30-60 seconds designedto influence, persuade and engage, in which media with emotional content is frequently used andthis will be one of the focuses of this thesis. The lack of understanding of the effects of emotion inadvertising results in large amounts of wasted time, money and other resources.

    Facial expressions, head gestures, heart rate, respiration rate and heart rate variability can in-form us about the emotional valence, arousal and attention of a person. In this thesis I propose todemonstrate how automatically detected naturalistic and spontaneous facial responses and physio-logical responses can be used to predict the effectiveness of stories.

    I propose a framework for automatically measuring facial and physiological responses in addi-tion to self-report and behavioral measures to content (e.g. video advertisements) over the Internetin order to understand the role of emotions in story effectiveness. Specifically, I will present anal-ysis of the first large scale data of facial, physiological, behavioral and self-report responses tovideo content collected in-the-wild using the cloud. I will develop models for evaluating the ef-fectiveness of stories (e.g. likability, persuasion and memory) based on the automatically extractedfeatures. This work will be evaluated on the success in predicting measures of story effectivenessthat are useful in creation of content whether that be in copy-testing or content development.

    i

  • Affective StorytellingAutomatic Measurement of Story Effectiveness fromEmotional Responses Collected over the Internet

    Daniel McDuffPhD Proposal in Media Arts & Sciences

    Affective Computing Group, MIT Media Lab

    Thesis Committee

    Rosalind PicardProfessor of Media Arts and Sciences, MIT Media LabThesis Supervisor

    Jeffrey CohnProfessor of PsychologyUniversity of Pittsburgh

    Ashish KapoorSenior Research ScientistMicrosoft Research, Redmond

    Thales TeixeiraAssistant Professor of Business AdministrationHarvard Business School

    ii

  • Abstract

    Emotion is key to the effectiveness of narratives and storytelling whether it be in influ-encing memory, likability or persuasion. Stories, even if fictional, have the ability to induce agenuine emotional response. However, the understanding of the role of emotions in storytellingand advertising effectiveness has been limited due to the difficulty in measuring emotions inreal-life contexts. Video advertising is a ubiquitous form of a short story usually 30-60 sec-onds designed to influence, persuade and engage, in which media with emotional content isfrequently used and this will be one of the focuses of this thesis.

    Facial expressions, head gestures, heart rate, respiration rate and heart rate variability caninform us about emotional valence and arousal and attention. In this thesis I propose to demon-strate how automatically detected naturalistic and spontaneous facial responses and physiolog-ical responses can be used to predict the effectiveness of stories. The results will be used toinform the creation and evaluation of new content.

    I propose a framework for automatically measuring facial and physiological responses inaddition to self-report and behavioral measures to content (e.g. video advertisements) over theInternet in order to understand the role of emotions in story effectiveness. Specifically, I willpresent analysis of the first large scale data of facial, physiological, behavioral and self-reportresponses to video content collected in-the-wild using the cloud. I will develop models forevaluating the effectiveness of stories (e.g. likability, persuasion and memory) based on theautomatically extracted features.

    1 IntroductionThere remains truth in Ray and Batras [28] statement: an inadequate understanding of the roleof affect in advertising has probably been the cause of more wasted advertising money than anyother single reason. This statement applies beyond advertising to many other forms of media andis due in part to the lack of understanding about how to measure emotion. This thesis proposaldeals with evaluating the effectiveness of emotional content in storytelling and advertising beyondthe laboratory environment using remotely measured facial and physiological responses. I will an-alyze challenging ecologically valid data collected over the Internet in the same contexts in whichthe media would normally be consumed and build a framework and set of models for automaticevaluation of effectiveness based on affective responses.

    The face is one of the richest sources of communicating affective and cognitive informa-tion [11]. In addition, physiological reactions, such as changes in heart rate and other vital signs,are partially controlled by the autonomic nervous system and as such are manifestations of emo-tional processes [36]. Recent work has demonstrated that both facial behavior and physiologicalinformation can be measured directly from videos of the human face and as such emotion valenceand arousal can be measured remotely.

    Previous work has shown that many people are willing to engage and share visual images fromtheir webcam over the Internet and these images and videos can be used for training automaticalgorithms for learning [32, 34, 22]. Moreover, webcams are now ubiquitous and have become astandard component on many media devices, laptops and tablets. In 2010, the number of cameraphones in use totaled 1.8 billion, which accounted for a third of all mobile phones1. In addition,

    1http://www.economist.com/node/15865270

    1

  • about half of the videos shared on Facebook every day are personal videos recorded from a desktopor phone camera2.

    Traditionally consumer testing of video advertising, whether by self-report, facial response orphysiology, has been conducted in laboratory settings. Lab-based studies, while controlled, aresubject to bias from the presence of an experimenter and other factors (e.g. comfort with the con-text) unrelated to advertising interest that may impact the participants emotional experience [35].Conducting experiments outside a lab-based context can help avoid such problems.

    Self-report is the current standard measure of affect, where people are typically interviewed,asked to rate their feeling on a Likert scale or turn a dial to quantify their state (affect dial ap-proaches). While convenient and inexpensive, self-report is problematic because it is also subjectto biasing from the context, increased cognitive load and other factors of little relevance to thestimulus being tested [30]. Self-report has a number of drawbacks including the difficulty for peo-ple to access information about their emotional experiences and their willingness to report feelingseven if they didnt have them [8]. For many the act of introspection is challenging to performin conjunction with another task and may in itself alter that state [21]. Although affect dial ap-proaches provide a higher resolution report of a subjects response compared to a post-hoc survey,subjects are often required to view the stimuli twice in order to help the participant introspect ontheir emotional state.

    Unlike self-report, facial expressions and physiological responses are implicit, non-intrusiveand do not interrupt a persons experience. In addition, as with affect dial ratings, facial andphysiological responses allow for continuous and dynamic representation of how affect changesover time. This represents a much richer data than can be obtained via a post-hoc survey. Asmall number of marketing studies consider the measurement of emotions via physiological [6],facial [18] or brain responses [3]. However, these are invariably performed in laboratory settingsand are restricted to a limited demographic.

    Advertising and online media is global: movie trailers, advertisements and other content cannow be viewed the world over via the Internet and not just on selected television networks. It isimportant that marketers understand the nuances in responses across a diverse demographic and abroad set of geographic locations. For instance, advertising that works in certain cultural contextsmay not be effective in others. A majority of the studies of emotion in advertising have onlyconsidered a homogeneous subject pool, such as university undergraduates or a group from onelocation. There is evidence to suggest that emotions can be universally expressed on the face [10]and our framework allows for the evaluation of advertising effectiveness across a large and diversedemographic much more efficiently than is possible via lab-based experiments.

    The aim of the proposed research is to utilize a framework for measuring facial, physiological,self-report and behavioral responses to commercials over the Internet in order to understand therole of emotions in advertising effectiveness (e.g. likability, persuasion and sales) and to designan automated system for predicting success based on these signals. This incorporates first-in-the-world studies of measurement of these parameters via the cloud and allows the robust explorationof phenomena across a diverse d