GlimpseData: Towards Continuous Vision-Based Personal Analytics

GlimpseData: Towards Continuous Vision-Based Personal AnalyticsSeungyeop Han withRajalakshmi Nandakumar, Matthai Philipose, Arvind Krishnamurthy, and David Wetherall

University of WashingtonMicrosoft1

Personal Analytics: Measure, analyze, and share data about life2

2What?What you have eatenCalorie counterRemind medication

Where?Vision-based LocalizationRemind visitsIndoor-location based ads

SeungyeopWho?Whom you have met todayRemind peoples nameFoster social interactionCheck peoples mood Visual Data adds New Dimensions

Hamburger354 cal33Challenges of Vision-based AnalysisResourcesCycles, bandwidth, power are limitedVision Algorithms Privacy and securityUser interaction with applications

44Challenges of Vision-based AnalysisResourcesCycles, bandwidth, power are limitedVision Algorithms Privacy and securityUser interaction with applications

55Resource is the Problem670 mW avg

Phone700 mW avg150 gCloud.01 serverWWAN700mW10GB/moWiFi500mW5MbpsResource consumption vs budget for mobile imager/cloud-based classifier:100-300mW imager(10mW)700mW WWAN(70mW)675GB/mo WWAN data(5GB/mo)1 server/wearercompute(0.01 server/wearer)Not All Frames are InterestingInput Visual StreamIs the frame interesting?Low-Power Sensors

YESCan we use lower power sensors to filter out uninteresting frames?77GlimpseDataData collection and analysis framework to study continuous sensor-augmented visual data

Case study on predicting whether a frame contains faces88Requirements for Data Collection System

Inclusive wrt. sensorsWidely available platformUnobtrusive

99Smartphone as a Data-Collecting Front-end

1010Smartphone as a Data-Collecting Front-end

SensorsAudioCamera

Location

Thermal CameraAndroid applicationRunning as a service~5 video frame per secondSync with timestampsCollect all possible sensorsCustom-built 16x4 temp array40x15 FOV

1111Collected DatasetTotal 116 minutes over 7 days~1M Sensor readings~100k Thermal camera frames>30k RGB frames (~5% are face frames)

1212 [demo] VisualizationJS visualizer running on browsers.

1313GlimpseDataData collection and analysis framework to study continuous sensor-augmented visual data

Case study on predicting whether a frame contains faces1414Case Study: Filtering Non-Face Frames with Low Data-rate SensorsBuild classifier determining non-face frames.Test with OpenCV face detector, and manually inspect frames.

Goal: filter out as many frames as possible while not missing frames with faces1515Not All Frames Have FacesInput Visual StreamDoes the frame have faces?Low-power sensor data

YESAccelerometer,Gyroscope, Light,Sound, Location,Thermal SensorFilterCan we determine if a frame is unlikely to have a face before running face detector?less than 5% frames contain faces in the data1616Thresholding on Single Sensor17Applying a threshold on each sensor value is promising

17Thresholding on Single Sensor18Applying a threshold on each sensor value is promising

Note: Face detector is resilient to noise18

Joint Classifier19With a logistic regressor using all sensors, it can filter out 60% of frames while missing only 10% frames with faces.19DiscussionHow to contribute data for the research communityNeed data with better quality (e.g., synchronization is an issue)Privacy concerns with sharing data

Beyond filteringDesign APIs for multiple applicationsNeed better classification methods20SummaryVisual data is extremely rich and could be useful and continuous analysis may be feasible.But, collecting large, rich dataset is challenging.

We presented GlimpseData, a data collection and analysis framework. A case study showed promising results for filtering uninteresting frames and feasibility of such study.

2121Q&Ahttps://github.com/UWNetworksLab/GlimpseData

2222