Mobile Phones based Continuous Sensing Systems

  • Published on

  • View

  • Download


Mobile Phones based Continuous Sensing Systems. Kiran Rachuri Computer Laboratory University of Cambridge. Sensors in a Smart P hone. Microphone Magnetometer GPS Bluetooth Accelerometer Camera Ambient light Proximity. What Can be Done. - PowerPoint PPT Presentation


<p>EmotionSense: A Mobile Phones based Adaptive Platform for Experimental Social Psychology Research</p> <p>Mobile Phones based Continuous Sensing SystemsKiran LaboratoryUniversity of Cambridge</p> <p>11</p> <p> Microphone</p> <p> Magnetometer</p> <p> GPS</p> <p> Bluetooth</p> <p> Accelerometer</p> <p> Camera</p> <p> Ambient light</p> <p> ProximitySensors in a Smart Phone</p> <p>22Microphone Speaker Recognition</p> <p>3What Can be Done</p> <p>Camera Face Recognition</p> <p>Accelerometer Activity Recognition</p> <p>Health monitoring</p> <p>4Application ScenariosSocial sensing</p> <p>Location based services</p> <p>Sense continuously</p> <p>Energy-Accuracy trade-offs</p> <p>Continuous Sensing Mobile Systems</p> <p>5Mobile Phone LimitationsEnergy, processing, and memory constraints</p> <p>Google Nexus One, and HDC HD2 are equipped with1GHz processor and 512MB RAM</p> <p>Energy is still a scarce resource</p> <p>67EmotionSenseEmotions of users and how they are influencedSocial Psychology - Emotions</p> <p>8Speech patterns of usersSocial Psychology - Speech Patterns</p> <p>9Oh yes, I was very happy todaySelf reports But they are biased towards positive emotions</p> <p>One-time behavioural study in laboratoryUsers hide their natural behaviour</p> <p>Social Psychology - Existing Methods</p> <p>10</p> <p> Use Mobile phones Powerful sensors Ubiquitous UnobtrusiveOur Solution11 Challenges Sensors not built for this purpose Battery powered Processing Main memory limitations Privacy concerns</p> <p>11EmotionSense - Architecture12EmotionSense ManagerInference EngineSpeakerMonitorHTTP ModuleDeclarative DatabaseRemote ServerColocationMoniorMovementMonitorLocationMonitorHTKHTK: Hidden markov ToolKitImplemented in PyS60 and Symbian C++1213EmotionSense Flow of DataClassifiersFacts BaseInference EngineActions BaseEmotionSense ManagerSensor MonitorsRaw dataE.g. X Y Z of AccelerometerE.g. User is MovingE.g. fact(Moving, True)E.g. fact(action, LocationSamplingInterval, 2)Emotion &amp; Speech RecognitionCollect voice dataUser/Emotion specific modelsTrain background GMM[2] M. Liberman, K. Davis, M. Grossman, N. Martey, and J. Bell. Emotional prosody speech and transcripts, 2002.Training ProcedureAt RuntimeRecord voice on phoneExtract PLPs using HCopyCompare using HERest[1] Gaussian Mixture ModelPLP: Perceptual Liner PredictionHcopy, HERest: Tools of Hidden markov ToolKit (HTK)Speaker Recognition: Participants voice dataEmotion Recognition: from libraryLoad speaker and emotion models on phoneClustering of emotions</p> <p>Emotion CategoriesBroad EmotionNarrow EmotionHappyElation, Interest, HappySadSadnessFearPanicAngerDisgust, Dominant, Hot angerNeutralNeutral normal, Neutral conversation, Neutral distant, Neutral tete, Boredom, Passive</p> <p>15(a) Used by psychologists (b) Improves accuracyWhy?</p> <p>Optimizations</p> <p>Adaptive framework</p> <p>16What About Energy?Silence detectiontrain an additional GMM using silence audio</p> <p>Comparisons driven by co-location informationA recorded audio sequence is compared only with the models co-located usersThis improves accuracy and saves energy</p> <p>Speaker Recognition - Optimizations</p> <p>17Implemented using Pyke, a knowledge based inference engine [] Activate GPS only when user is movingAdaptation Frameworkset_location_sampling_intervalforeach facts.fact($factName, $value) check $factName == 'Activity' facts.fact($actionName, $currentInterval) check $actionName == 'LocationInterval' $interval = update($value, $currentInterval)assert facts.fact('action', 'LocationInterval', $interval) </p> <p>18Facts BaseInference EngineActions BaseE.g. fact(Moving, True)E.g. fact(action, GPS, ON)18Sensor Sampling19Time SleepSenseSleepSenseSleepEventsTime SleepSenseEventsSenseSleepSleepSensor Sampling IssuesContinuous sampling degrades battery life</p> <p>Long sleep durations result in loss of sensor data</p> <p>Not all sensors are similar</p> <p>Accuracy varies with sensors and classifiers</p> <p>20Sampling Interval21SleepSense OnceSleep 0Minimum Sampling IntervalMaximum Sampling IntervalConstantDesign Methodology22Missable EventSleepNot all events are importantE.g.: Microphone recording when there is no audible soundClassify events as Unmissable and MissableUse functions to control the sleep intervalBack-off FunctionE.g.: f(x) = 2x, where x is sleep intervalUnMissable EventAdvance FunctionE.g.: f(x) = x/2, where x is sleep intervalSenseBack-off and Advance FunctionsTypeBack-off functionAdvance functionLinearQuadratic</p> <p>Exponential</p> <p>Minimum</p> <p>N/AMinimum intervalMaximum</p> <p>Maximum intervalN/A</p> <p>23x: sleep intervalDynamic Adaptation24Dynamically switch functions from least to most aggressiveMissable EventSleepSequence CountSenseLinear back-off functionQuadratic back-off functionExponential back-off functionUpdate Sleep Interval&lt; Linear Threshold&lt; Quadratic Threshold&gt; Quadratic ThresholdMicro-benchmarks</p> <p>Social psychology experiment</p> <p>Meeting experimentEvaluation25</p> <p>Speaker Recognition - Benchmarks Accuracy Effect of noise on accuracy 26Speaker Recognition Benchmarks Energy consumption Latency27Why ?a) Computationally intensive processingb) High latencyThen why compute locally ?a) Privacy concernsb) Users can use their own sim cardsEmotion Recognition - Benchmarks Accuracy Energy consumption28Clustering helpsNokia 6210 Navigator mobile phones</p> <p>18 participants, 10 days</p> <p>Users filled in daily diary questionnaire</p> <p>Voice data is discarded immediately</p> <p>All computation performed locally on phone</p> <p>Social Psychology Experiment</p> <p>29 Emotion distribution similarity EmotionSenseSelf reports</p> <p>Social Psychology Experiment - Results30Users indicated ``happy'' emotion to represent their mental state, and not necessarily verbal expressionSocial Psychology Experiment - ResultsCorrelation with time of day and co-location31EmotionSense can also be used to analyze the speech patterns</p> <p>Considerable amount of consistency in verbal behaviour</p> <p>Social Psychology Experiment - Results</p> <p>3211 participants , 30 minutes discussion</p> <p>We identified conversation leaders in each time slot of length 5 minutes</p> <p>The result shows the top five most active speakersMeeting Experiment</p> <p>33</p> <p>33Continuous sensing mobile systems</p> <p>EmotionSenseSpeaker and Emotion RecognitionSensor MonitorsAdaptive and Programmable Framework</p> <p>EvaluationMicro-benchmarksSocial Psychology ExperimentMeeting Experiment</p> <p>Summary</p> <p>34Thank You</p> <p>Kiran RachuriComputer LaboratoryUniversity of</p> <p>EmotionSense</p> <p>3535</p> <p> 40</p> <p> 50</p> <p> 60</p> <p> 70</p> <p> 80</p> <p> 90</p> <p> 100</p> <p> 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16</p> <p>Accu</p> <p>racy</p> <p> [%]</p> <p>Sample length [seconds]</p> <p> 0</p> <p> 20</p> <p> 40</p> <p> 60</p> <p> 80</p> <p> 100</p> <p> 0 0.02 0.04 0.06 0.08 0.1 0.12</p> <p>Accu</p> <p>racy</p> <p> [%]</p> <p>Brownian noise amplitude</p> <p> 0</p> <p> 20</p> <p> 40</p> <p> 60</p> <p> 80</p> <p> 100</p> <p> 120</p> <p> 140</p> <p> 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16</p> <p>Late</p> <p>ncy [</p> <p>seco</p> <p>nds]</p> <p>Sample length [seconds]</p> <p>Local computationRemote computation</p> <p> 0 5</p> <p> 10 15 20 25 30 35 40 45 50</p> <p> 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16</p> <p>Ener</p> <p>gy co</p> <p>nsum</p> <p>ption</p> <p> [joule</p> <p>s]</p> <p>Sample length [seconds]</p> <p>Local computationRemote computation</p> <p> 10 20 30 40 50 60 70 80 90</p> <p> 100</p> <p> 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16</p> <p>Accu</p> <p>racy</p> <p> [%]</p> <p>Sample length [seconds]</p> <p>Emotion narrow classEmotion broad class</p> <p> 15</p> <p> 20</p> <p> 25</p> <p> 30</p> <p> 35</p> <p> 40</p> <p> 45</p> <p> 50</p> <p> 55</p> <p> 60</p> <p> 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16</p> <p>Ener</p> <p>gy co</p> <p>nsum</p> <p>ption</p> <p> [joule</p> <p>s]</p> <p>Sample length [seconds]</p> <p> 0</p> <p> 10</p> <p> 20</p> <p> 30</p> <p> 40</p> <p> 50</p> <p> 60</p> <p> 70</p> <p>HAPPY SAD FEAR ANGER NEUTRAL</p> <p>Perc</p> <p>enta</p> <p>ge</p> <p> 0 5</p> <p> 10 15 20 25 30</p> <p>HAPPY SAD FEAR ANGER</p> <p> 0 10 20 30 40 50 60 70 80 90</p> <p> 100</p> <p>HAPPY SAD FEAR ANGER NEUTRAL</p> <p>Perc</p> <p>enta</p> <p>ge</p> <p> 0 0.5</p> <p> 1 1.5</p> <p> 2 2.5</p> <p> 3 3.5</p> <p> 4 4.5</p> <p>HAPPY SAD FEAR ANGER</p> <p> 0</p> <p> 5</p> <p> 10</p> <p> 15</p> <p> 20</p> <p> 25</p> <p>HAPPY SAD FEAR ANGER NEUTRAL</p> <p>Perc</p> <p>enta</p> <p>ge</p> <p> 0 0.2 0.4 0.6 0.8</p> <p> 1 1.2 1.4</p> <p>HAPPY SAD FEAR ANGER</p> <p>9-1111-1313-1515-1717-19</p> <p> 0 10 20 30 40 50 60 70 80 90</p> <p> 100</p> <p>HAPPY SAD FEAR ANGER NEUTRAL</p> <p>Perc</p> <p>enta</p> <p>ge</p> <p> 0 1 2 3 4 5 6 7</p> <p>HAPPY SAD FEAR ANGER</p> <p>1-23-45-67-8</p> <p> 0</p> <p> 2</p> <p> 4</p> <p> 6</p> <p> 8</p> <p> 10</p> <p> 12</p> <p> 0 1 2 3 4 5 6 7</p> <p>Num</p> <p>ber o</p> <p>f det</p> <p>ectio</p> <p>ns p</p> <p>er h</p> <p>our</p> <p>Day</p> <p>user1user2user3user4</p> <p> 0</p> <p> 10</p> <p> 20</p> <p> 30</p> <p> 40</p> <p> 50</p> <p> 60</p> <p>0-5 5-10 10-15 15-20 20-25 25-30</p> <p>Perc</p> <p>enta</p> <p>ge</p> <p>Time slot [5 minutes]</p> <p>user1user2user3user4user5</p>


View more >