Donnie H. Kim, Nicolai M. Petersen, Mohammad Rahimi, Jeff Burke, Deborh Estrin

Center for Embedded Networked Sensing, UCLALenore Arab

David Geffen School of Medicine, UCLA

Presented by Reg Arvidson

Phones are everywhere! Increasingly carry imaging and

location capabilities Creation of assisted recall systems

• Record aspects of the environment for later playback

Rewind supports data collection on specific behaviors and situations• As opposed to life blogging systems

Scalable system of everyday mobile phones and supporting web services

Explore how client/server-side image processing can…• lower bandwidth needs• streamline user navigation

Original pilot was to assist in recall of dietary intake

Other short/exploratory trials developed

Take advantage of the application’s constraints in…• capturing images• presenting images• processing images• and uploading images

Manages data flow among system components• Interacts using

SSL/TLS encryption and authenticated transmission

Provides a user interface to view captured images• Password protected

web interface

Web Interface – communication over HTTPS through URLs, stored in secure file system and relation database

Image Handling Services – checks if processing is required for each image• Resizing – generates thumbnail for web

interface• Reaping – deletes images marked by user or

filters• Image processing – pushes images to IPS

and retrieves results

Can potentially capture significant personal data• Family members, computer screen contents…• Even occasional image inside a restroom

Privacy addressed at earliest phases of prototyping

Secure HTTP over SSL with a X.509 public key certificate for the web server

Images viewable only by the individual owner, no identifiable information stored

Users given authenticated access to Image Viewer with a User ID/Password

Shown a subset of images (thumbnails) based on time clustering and quality rank

Tasked by DMS to…• Process images• Classify images• Annotate images

Time-consuming and application-dependent image processing tasks separated from core data flow services• Easily scalable through

addition of IPSs

Matlab used as computational engine Extended with an internal TCP/IP

server to provide an interface for external applications

Can handle image processing as a scheduled task or in a FIFO manner

IPS classifies image into four categories…• Clear• Blurred• Exposed• BlurExposed

Class determined from four well-known features• Mean of Intensity• Standard Deviation of Intensity• Number of Edges• Sum of High Frequency Coefs. Of Discreate

Cosine Transform (DCT)

Nokia N80 S60 3rd Edition User Interface Symbian v9.1 Operating System 3 megapixel camera Both 802.11 b/g and GSM connectivity Runs Campaignr

• Acquires data from hardware/software sensors• Immediately stores to internal memory, queues

for upload to DMS• Application-specific XML file specifies which

sensors to collect data from

Clear Clear BlurredMotion of Carrier

BlurredMotion of Subject

ExposedPoor Lighting

BlurExposed

FullDayDietary pilot provided a large and varying data set to work with

Many images were blurred due to motion of individual of subjects of image

Additional images were either over- or under-exposed

Individuals marked the classification of images to produce groundtruth data

83% of images correctly classified93% of clear images correctly classified2% of low quality images were incorrectly classified

Latency dominated by image processing computation

Very small deviation in processing latency per image• Roughly predetermine time to process

images

DietaryRecall• 10 users – 11,090 images uploaded• Device turned on only during meals• Experience kept simple, 35 images max per

episode FullDayDietary

• 14 users – 14,958 images released (6 users)• Ran while outside home, 6 images/minute

PosterSessionCapture• 15 users – visitors to a research conference

poster session• Tested system with many simultaneous users

Pilots showed large number of filtered images

Wireless upload channel became congested with presence of co-located users

Image processing became non-negligible with many users

Added extensions to the system to support local image processing

Early pilots resulted in many low quality images and congestion on the upload link• 33% of DietaryRecall images were marked low qual

Filter out extremely low quality images that would be filtered by back-end server anyways

Also resulted in interesting requirements for prioritized and real-time upload

•Stores image to database upon capture with other sensed data•Images annotation - fetches image, extracts selected feature, stores results•Classification – read computed features, classifies image using decision tree•Clustering – generates cluster ID using the capture time•Upload Ranker – ranks images based on configured upload policy

DCT ignored due to high cost on IPS Edge count not implemented File size used due to a high correlation with

number of edges AND normalized sum of DCT coefs.

DCT on IPS vs size on phone Edge count on IPS vs size on phone

82% of images correctly classified Only 14% of low quality images incorrectly

classified

Phones collecting multiple images per second can easily congest narrow upload channels

Extended periods of disconnect can result in a sizable backlog

Prioritizing upload order can improve usability of the system• Reverse Chronological Order – uploads the most

recently captured image first• Prioritized Upload – ranked by quality and cluster

Not an issue if users do not attempt to view images for a duration of time

In event of rapid review of images, selective uploading can enhance interactivity and responsiveness

Can show best images throughout the day (best of clusters) or reverse chronological (reverse order) as opposed to chronological order

Waiting time before viewing the most recently captured image following different disconnection times.

Completion delay comparison following different disconnection times.

Within a cluster, prioritized method uploads high quality image first

Performance gain insignificant when every image is of low quality

Rewind: Leveraging Everyday Mobile Phones for Targeted Assisted Recall (UCLA Technical Report 2008)

Urban Sensing – CENS/UCLA• http://urban.cens.ucla.edu/projects/

dietsense/