18
System architecture for data collection campaigns Joseph Kim 2007-10-26 Faculty: Jeff Burke, Deborah Estrin, Mark Hansen, Mani Srivastava UCLA Center for Embedded Networked Sensing UCLA Center for Research in Engineering, Media and Performance Graduate Students: Gong Chen, August Joki, Eitan Mendelowitz, Rafael Laufer, Nicolai Munk Petersen, Nithya Ramanathan, Sasank Reddy, Jason Ryder, Vids Samantha, Thomas Schmid, Nathan Yau UCLA Departments of Computer Science, Electrical Engineering, Statistics CENS Urban Sensing is collaborative work of many faculty, staff, and students: Mark Allman, Jeff Burke, Gong Chen, Dana Cuff, Ryan Dorn, Deborah Estrin, Mark Hansen, August Joki, William Kaiser, Jerry Kang, Joseph Kim, Eitan Mendelowitz, Andrew Parker, Vern Paxson, Nicolai Munk Petersen, Nithya Ramanathan, Sasank Reddy, Jason Ryder, Vids Samanta, Thomas Schmid, Mani Srivastava, Fabian Wagmister, Nathan Yau, and others. In partnership with NSF NeTS-FIND, Cisco, Nokia, Schematic, Sun, UCLA REMAP, UCLA ITS, Walt Disney Imagineering R&D

System architecture for data collection campaigns Joseph Kim 2007-10-26 Faculty: Jeff Burke, Deborah Estrin, Mark Hansen, Mani Srivastava UCLA Center for

  • View
    218

  • Download
    0

Embed Size (px)

Citation preview

Page 1: System architecture for data collection campaigns Joseph Kim 2007-10-26 Faculty: Jeff Burke, Deborah Estrin, Mark Hansen, Mani Srivastava UCLA Center for

System architecture for data collection campaigns

Joseph Kim 2007-10-26

Faculty: Jeff Burke, Deborah Estrin, Mark Hansen, Mani SrivastavaUCLA Center for Embedded Networked SensingUCLA Center for Research in Engineering, Media and Performance

Graduate Students: Gong Chen, August Joki, Eitan Mendelowitz, Rafael Laufer, Nicolai Munk Petersen, Nithya Ramanathan, Sasank Reddy, Jason Ryder, Vids Samantha, Thomas Schmid, Nathan YauUCLA Departments of Computer Science, Electrical Engineering, Statistics

CENS Urban Sensing is collaborative work of many faculty, staff, and students: Mark Allman, Jeff Burke, Gong Chen, Dana Cuff, Ryan Dorn, Deborah Estrin, Mark Hansen, August Joki, William Kaiser, Jerry Kang, Joseph Kim, Eitan Mendelowitz, Andrew Parker, Vern Paxson, Nicolai Munk Petersen, Nithya Ramanathan, Sasank Reddy, Jason Ryder, Vids Samanta, Thomas Schmid, Mani Srivastava, Fabian Wagmister, Nathan Yau, and others.

In partnership with NSF NeTS-FIND, Cisco, Nokia, Schematic, Sun, UCLA REMAP, UCLA ITS, Walt Disney Imagineering R&D

Page 2: System architecture for data collection campaigns Joseph Kim 2007-10-26 Faculty: Jeff Burke, Deborah Estrin, Mark Hansen, Mani Srivastava UCLA Center for

2

Data Collection Campaigns

• Exploring systems that involve personal sensing to shed insight in people’s everyday life

• Campaign Framework (CFM) - New base platform for future Campaigns

Page 3: System architecture for data collection campaigns Joseph Kim 2007-10-26 Faculty: Jeff Burke, Deborah Estrin, Mark Hansen, Mani Srivastava UCLA Center for

3

Energetics - Continuous Image Capture for Dietary Recall

• Objective - Improve the accuracy of 24-hour diet survey

• Pilot experiment in collaboration with Lenore Arab

• Observation day

– Wear a Nokia n80 around neck during meal times

– Take bio marker

• Recall day

– Provided blood and urine sample

– Document diet via 24-hour diet survey using diet day and energetics application to aid recall

• Analysis - Compare inferred dietary intake from survey with analysis of blood and urine sample

Staff and participant training user interface

Page 4: System architecture for data collection campaigns Joseph Kim 2007-10-26 Faculty: Jeff Burke, Deborah Estrin, Mark Hansen, Mani Srivastava UCLA Center for

4

Energetics - Human Subjects Research

Page 5: System architecture for data collection campaigns Joseph Kim 2007-10-26 Faculty: Jeff Burke, Deborah Estrin, Mark Hansen, Mani Srivastava UCLA Center for

5

Energetics - System Architecture

Campaignr

Energetics

BatchProcess

ImageAnnotation

MySql

Apache

Python Matlab

Symbian C++

• Data captured by phones transferred to modified sensorbase application

• Batch process – Resize images– Enforce data retention

policy– Annotate images for

blurriness, intensity, etc. via image web service

• User interface renders summary of images captured taking into account annotated information

Page 6: System architecture for data collection campaigns Joseph Kim 2007-10-26 Faculty: Jeff Burke, Deborah Estrin, Mark Hansen, Mani Srivastava UCLA Center for

6

Energetics - Image Annotation

Page 7: System architecture for data collection campaigns Joseph Kim 2007-10-26 Faculty: Jeff Burke, Deborah Estrin, Mark Hansen, Mani Srivastava UCLA Center for

7

Energetics - Lessons Learned

• Web Services• Admin and system management• Future Work

– Batch– Flexible data model– Phone processing– Rapid UI development

Page 8: System architecture for data collection campaigns Joseph Kim 2007-10-26 Faculty: Jeff Burke, Deborah Estrin, Mark Hansen, Mani Srivastava UCLA Center for

8

Trends in Web System Design

• Custom form-fit software

• Open, simple data protocols for the web

Clay Shirky, NYU on “Situated Software”:

“Part of the future I believe I'm seeing is a change in the software ecosystem which, for the moment, I'm calling situated software. This is software designed in and for a particular social situation or context. This way of making software is in contrast with what I'll call the Web School (the paradigm I learned to program in), where scalability, generality, and completeness were the key virtues.”

Adam Bosworth, Vice President of Engineering, Google on data protocols for the web:

“… do for information what HTTP did for user interface… “

“… provide an open, simple data model to easily serve up information similar to the way a web server delivers content to the browser. Delivering an information server that is capable of federating information across the web, intelligently caching and scaling linearly is the next big database challenge.”

Page 9: System architecture for data collection campaigns Joseph Kim 2007-10-26 Faculty: Jeff Burke, Deborah Estrin, Mark Hansen, Mani Srivastava UCLA Center for

9

Trends in Web System Design - Examples

• Custom form-fit software

• Open, simple data protocols for the web

Page 10: System architecture for data collection campaigns Joseph Kim 2007-10-26 Faculty: Jeff Burke, Deborah Estrin, Mark Hansen, Mani Srivastava UCLA Center for

10

General system architecture for campaigns

• Data captured by sensor stored

• Higher order data generated by classifiers and inference models

• Data republished and rendered via multiple customized user interfaces

ParticipatorySensor

Data Store andApplication Engine

Data classification and inference

(Spatial)DB

User Interface

CollaborativeInterface

AnnotationInterface

AdminInterface

SecondaryData Data access

API

Page 11: System architecture for data collection campaigns Joseph Kim 2007-10-26 Faculty: Jeff Burke, Deborah Estrin, Mark Hansen, Mani Srivastava UCLA Center for

11

REST (Representational State Transfer) Style Architecture

Roy Fielding’s UCI Ph.d. Dissertation:

http://www.ics.uci.edu/~fielding/pubs/dissertation/top.htm

Wikipedia description of the REST-style architecture

• Application state and functionality are divided into resources

• Every resource is uniquely addressable using a universal syntax for use in hypermedia links

• All resources share a uniform interface for the transfer of state between client and resource, consisting of

• A constrained set of well-defined operations

• A protocol that is:

– Client/Server

– Stateless

– Cacheable

– Layered

Page 12: System architecture for data collection campaigns Joseph Kim 2007-10-26 Faculty: Jeff Burke, Deborah Estrin, Mark Hansen, Mani Srivastava UCLA Center for

12

Primary data format - JSON

• Think nest-able hash tables, arrays and other primitives from most programming languages

• Exampleperson = {

"name": "Simon Willison",

"age": 25,

"height": 1.68,

"urls": [

"http://simonwillison.net/",

"http://www.flickr.com/photos/simon/",

"http://simon.incutio.com/"

]

}

Partial visual grammar of JSON from http://json.org/

Page 13: System architecture for data collection campaigns Joseph Kim 2007-10-26 Faculty: Jeff Burke, Deborah Estrin, Mark Hansen, Mani Srivastava UCLA Center for

13

Flexible Data Model

• Hierarchy of documents

• Documents composed of fields, name-value pairs that are hierarchically related

• Can express most types of web data

• Example Atom:

– Document collection: parent document - /blur

{ “title”:”…”, “description”:”….”, “author”:”…”

– Entries: Child document

{ “title”:”…”, “body”:”…” }

• Example Yelp Review:

– Restaurant: parent document - /taco_bell/

{ “name”:”Taco Bell”, “Address”:”….”, “images”:”…” }

– Specific reviews: child document - /taco_bell/r1

{ “body”:”….”, “stars”:5 }

Page 14: System architecture for data collection campaigns Joseph Kim 2007-10-26 Faculty: Jeff Burke, Deborah Estrin, Mark Hansen, Mani Srivastava UCLA Center for

14

Activity Areas

• Triggering - event driven notification of data change

• Encryption• System management

– Search engine of exposed services– Generalized components that can fully

managed by CFM– Custom components running in CFM sandbox– Fully independent modules that communicate

via CFM Module APIs

Page 15: System architecture for data collection campaigns Joseph Kim 2007-10-26 Faculty: Jeff Burke, Deborah Estrin, Mark Hansen, Mani Srivastava UCLA Center for

15

PEIR (Personal Environmental Impace Report) Prototype

Speed (MPH) Automobile CO2 Emission (grams/mile or grams/idle-hour

0 0

5 991.467

10 747.097

15 585.227

20 476.29

25 402.605

30 353.387

35 322.034

40 304.609

45 298.995

50 304.464

55 321.534

60 352.069

65 399.665

• Plug GPS Trace information into CO2 Emmission Model

Page 16: System architecture for data collection campaigns Joseph Kim 2007-10-26 Faculty: Jeff Burke, Deborah Estrin, Mark Hansen, Mani Srivastava UCLA Center for

16

PEIR - GPS Trace

Page 17: System architecture for data collection campaigns Joseph Kim 2007-10-26 Faculty: Jeff Burke, Deborah Estrin, Mark Hansen, Mani Srivastava UCLA Center for

17

PEIR - CO2 Emission Heatmap

Page 18: System architecture for data collection campaigns Joseph Kim 2007-10-26 Faculty: Jeff Burke, Deborah Estrin, Mark Hansen, Mani Srivastava UCLA Center for

18

PEIR Prototype

• Location Services– LA Zoning classification of given longitude and

latitude point– Nearest LA streets within 100 meters of given

longitude and latitude point– Haversine distance between two longitude

and latitude points