IS4: A Storage System for Organizing Physical Data

IS4:A Storage System for

Organizing Physical Data

Jorge OrtizUniversity of California, Berkeley

Talk overview

• Integrated sensor stream storage system (IS4)– Recap from last retreat

• Physical data– Representation of:

• Physical devices• Measurements• Environments

– Management challenges

• Current features and future work

3

Inst

rum

enta

tion

Inst

rum

enta

tion

Mod

els

Mod

els

ControlsControls

Building OSBuilding OS

Plug LoadsPlug Loads

LightingLightingFacilitiesFacilities

BuildingIn

stru

men

tatio

nIn

stru

men

tatio

nM

odel

sM

odel

s

Routing/ControlRouting/Control

Grid OSGrid OS

Demand ResponseDemand Response

Load FollowingLoad FollowingSupply FollowingSupply Following

Grid

Facility-to-Building

Facility-to-Building

Facility-to-Building

Facility-to-Building

Facility-to-Building

Facility-to-Building

Gen-to-BuildingGen-to-Building

Inst

rum

enta

tion

Inst

rum

enta

tion

Mod

els

Mod

els

ControlControl

CompressorScheduling

CompressorScheduling

Temperature MaintenanceTemperature Maintenance

Fridge as a model Thermal storage system

Storage-to-Building

Storage-to-Building

Inst

rum

enta

tion

Inst

rum

enta

tion

Mod

els

Mod

els

Power-AwareCluster Manager

Power-AwareCluster Manager

Load Balancer/Scheduler

Load Balancer/Scheduler

Web ServerWeb Server

Web App LogicWeb App LogicDB/StorageDB/Storage

Machine RoomMR-to-BuildingMR-to-Building

Multi-scale Energy Network

Gen-to-Grid

Gen-to-Grid

uGrid-to-GriduGrid-to-Grid

Building-to-Grid

Building-to-Grid Building-

to-GridBuilding-to-Grid

Wind Modeling

IS4

SCADA SystemsSupervisory Control and Data Acquisition

+SODA4R787__ART

+Data

Problems with SCADA

• No systematic organization of data and metadata– Metadata embedded in UI– Difficult to find context information

• Incomplete data and metadata– Mostly 0’s– Missing type descriptions, location information

Capturing building layout

Chiller

Pump

Chiller

Pump

AHUSF EF

Vent Vent

Zone

Entity relationships Building Systems Spaces/Zones Structural relationship Sensors/Actuators

Inside systems Inside zones/spaces

Types

Chiller

Pump

Chiller

Pump

AHUSF EF

Vent Vent

Zone

System

Space

Building entity schema

Relevant queries

• List temperature sensors on 4th floor of Soda hall– SELECT id FROM zone_sensors WHERE

building=‘Soda’ and floor=4 AND id LIKE ‘%ART’

• Get all temperature sensor data in the last day on the 4th floor of Soda– SELECT * FROM sensor_data WHERE id=[list of

sensors] AND timestamp >= ’ 2009-12-16 00:00:00’ and timestamp<=now()

System objectives

• Physical data storage repository– Append only, forever

• Management of data and metadata– Changes occur over time

• Physical-data queries– Historical and real time

• Security– Secure access to data streams, distillates, metadata

What is physical data?

• Data collected from the environment• Why is it different from other kinds of data?

– Data is produced by sensing devices– Devices are placed in the physical environment– Placement matters

• Deployment changes over time– Devices, configuration, environment

Data, metadata relationship-managementis challenging

Sensing device metadata

• Make– Dent SmartLogger

• Model– CTLogger TOU-CT

• url– http://www.dentinstruments.com/media/

PDF/SMARTware2009_Manual.pdf

• Make– ACme

• Model– Revision 1

• url– http://smote.cs.berkeley.edu:8000/

tracenv/wiki/ACME

Object schema

{ "description":"object_stream_schema", "type":"object", "properties": {

“type":{"type":"string","optional":false, "options":[{"value":"object_stream"}]}, "device_name":{"type":"string", "optional":"false"}, "model":{"type":"string","optional":false}, "desc":{"type":"string","optional":true},“url":{"type":"string",”format”:”url”,"optional":true

}}

{ "description":"object_stream_schema", "type":"object", "properties": {

“type":{"type":"string","optional":false, "options":[{"value":"object_stream"}]}, "device_name":{"type":"string", "optional":"false"}, "model":{"type":"string","optional":false}, "desc":{"type":"string","optional":true},“url":{"type":"string",”format”:”url”,"optional":true

}}

{ “type“:"object_stream", “$schema”:”object_stream_schema.json”,

“device_name":“dentMeter123", "model":“CTLogger TOU-CT", "desc":“dent meter”,“url”:” http://www.dentinstruments.com/media/PDF/SMARTware2009_Manual.pdf”

}

{ “type“:"object_stream", “$schema”:”object_stream_schema.json”,

“device_name":“dentMeter123", "model":“CTLogger TOU-CT", "desc":“dent meter”,“url”:” http://www.dentinstruments.com/media/PDF/SMARTware2009_Manual.pdf”

}

inst

ance

inst

ance

sche

ma

sche

ma

Communication information

• Network address or URL

• Function calls, parameter description

• Functional schema or url documentation

2001:252:0:1::2008:6 and 2001:252:0:1::2008:8

http://coryacme.cs.berkeley.edu/data/33

Logic schema

{ "description":“logic_stream_schema", "type":"object", "properties": {

“type":{"type":"string","optional":false, "options":[{"value":“logic"}]}, “$schema”:”logic_stream_schema.json”,“address”:{“type”:”string”, “format”:[“string”, “ipv4”, “ipv6”, “url”],

“optional”:”false”},“functions”:{“type”:”array”, “options”:[{

“function_name”:”{“type”:”string”},“operation”:{“type”:”string”, “optional”:”true”},“parameters”:{“type”:”array”, “options”:[”object”, “string”, “$ref”]}}

}}

{ "description":“logic_stream_schema", "type":"object", "properties": {

“type":{"type":"string","optional":false, "options":[{"value":“logic"}]}, “$schema”:”logic_stream_schema.json”,“address”:{“type”:”string”, “format”:[“string”, “ipv4”, “ipv6”, “url”],

“optional”:”false”},“functions”:{“type”:”array”, “options”:[{

“function_name”:”{“type”:”string”},“operation”:{“type”:”string”, “optional”:”true”},“parameters”:{“type”:”array”, “options”:[”object”, “string”, “$ref”]}}

}}

{ “type“:“logic", “$schema”:”logic_stream_schema.json”,“address”:”http://is4/resource”,“function”:[{

“function_name”:”a”, “opertation”:”GET”,“parameters”:[{“field”, “name”}]}

}

{ “type“:“logic", “$schema”:”logic_stream_schema.json”,“address”:”http://is4/resource”,“function”:[{

“function_name”:”a”, “opertation”:”GET”,“parameters”:[{“field”, “name”}]}

}

inst

ance

inst

ance

sche

ma

sche

ma

Sensor context information

Temperature

Humidity

Vibration

Pressure

Electric load tree+

Electrical Physical

Humidity

PAR/TSR

Temperature

Panel 1 Panel 2

A

B

A

B

Panel 1 Panel 2

A

B

A

B

1

5

9

13

17

21

25

29

33

37

41

3

7

11

15

19

23

27

31

35

39

A1

5

9

13

17

21

25

29

33

37

41

3

7

11

15

19

23

27

31

35

39

1

5

9

13

17

21

25

29

33

37

41

3

7

11

15

19

23

27

31

35

39

A2

6

10

14

18

22

26

30

34

38

42

4

8

12

16

20

24

28

32

36

40

B2

6

10

14

18

22

26

30

34

38

42

4

8

12

16

20

24

28

32

36

40

2

6

10

14

18

22

26

30

34

38

42

4

8

12

16

20

24

28

32

36

40

B

1

5

9

13

17

21

25

29

33

37

41

3

7

11

15

19

23

27

31

35

39

A1

5

9

13

17

21

25

29

33

37

41

3

7

11

15

19

23

27

31

35

39

1

5

9

13

17

21

25

29

33

37

41

3

7

11

15

19

23

27

31

35

39

A2

6

10

14

18

22

26

30

34

38

42

4

8

12

16

20

24

28

32

36

40

B2

6

10

14

18

22

26

30

34

38

42

4

8

12

16

20

24

28

32

36

40

2

6

10

14

18

22

26

30

34

38

42

4

8

12

16

20

24

28

32

36

40

B

Circuit/breaker panel level power monitoring

Vibration

Load Tree

CT mains power monitoring

ACme: plug load energy monitor and controller

+ Slide courtesy of Xiaofan Jiang

Context description

• General description• Google JSON GeoCoding object

– http://code.google.com/apis/maps/documentation/javascript/v2/services.html#Geocoding_Object

• IS4 context graph– Schema that expresses contextual objects and

relationships as a graph

IS4 Context graph schemasc

hem

asc

hem

a

Context Node:{ "label":{"type":"string", "optional":false},

"name":{"type":"string", "optional":false},"type":{"type":"string","options":[{"value":"context_node"}], "optional":false},"description":{"type":"string", "optional":false}, "properties": {

"AssociatedDevices":{ "type":"array","options":[{"value":"string","format":"UUID"}], "optional":true },

"parents":{ "type":"array", "options":[{"value":"string","properties":{"$ref":"#.properties.cnid"}}], "optional":true }, "children":{ "type":"array", "options":[{"value":"string",

"properties":{"$ref":"#.properties.cnid"}}], "optional":true }, "cnid": { "type":[“string"] "description":"local node identifier", "optional":false }}}

Context Edge{…

"label":{"type":"string", "optional":false, "description":"String with spaces"}, "name":{"type":"string", "optional":false, "description":"String without space"}, …"sourceNode":{"type":[“string"], "description":"local node identifier",

"optional":false}, "destinationNode":{"type":[“string"], "description":"local node identifier",

"optional":false}, "AssociatedDevices":{ "type":"array",

"options":[{"value":"string","format":"UUID"}], "optional":true }}

Context Node:{ "label":{"type":"string", "optional":false},

"name":{"type":"string", "optional":false},"type":{"type":"string","options":[{"value":"context_node"}], "optional":false},"description":{"type":"string", "optional":false}, "properties": {

"AssociatedDevices":{ "type":"array","options":[{"value":"string","format":"UUID"}], "optional":true },

"parents":{ "type":"array", "options":[{"value":"string","properties":{"$ref":"#.properties.cnid"}}], "optional":true }, "children":{ "type":"array", "options":[{"value":"string",

"properties":{"$ref":"#.properties.cnid"}}], "optional":true }, "cnid": { "type":[“string"] "description":"local node identifier", "optional":false }}}

Context Edge{…

"label":{"type":"string", "optional":false, "description":"String with spaces"}, "name":{"type":"string", "optional":false, "description":"String without space"}, …"sourceNode":{"type":[“string"], "description":"local node identifier",

"optional":false}, "destinationNode":{"type":[“string"], "description":"local node identifier",

"optional":false}, "AssociatedDevices":{ "type":"array",

"options":[{"value":"string","format":"UUID"}], "optional":true }}

Cory hall load tree

Load Tree Image Graphical representation

Context state snapshots

Time t1 Time t2

Queries

• What floor consumes the most power per month?

• Send alert if the air conditioner in room 465 consumes more than Y kW in a 24 hour period

• Send alert if independent zones become dependent

• What is the most common state of the air/water flow graph?

Expressing physical-data queries

• Semantic graph + temporal data– Physical and logical relationships– Data streams associated with graphical elements

• Easily express queries across both data types• Efficient storage• Efficient access

IS4 Current Status

• Data publishers identified by unique id (and IS4 resource URI).

• Publishing process includes object and logic JSON objects as input

• Context data managed separately• Publisher may belong to multiple contexts• Data and metadata stream explicitly bound

Current Architecture

StorageStorage

HTTP/RESTHTTP/REST

Publish ManagerPublish Manager Subscribe ManagerSubscribe ManagerProxy

ManagerProxy

Manager

HumidityPAR/TSRTemperature VibrationCT mains power monitoring

ACme: plug load energy monitor and controller

PublishPublish

SubscribeSubscribe

Context ManagerContext Manager

IS4 Interface Overview

/ #root – status information [GET] /is4 # a particular channel [GET] /info # statistical system information [GET] /publish # publishing resource [GET/PUT]

/all # list of all publishers [GET] /id # sub-children [GET]

/<id> # id of publisher [GET/POST] /name # name alias for this id [GET]

/mysubs # list/add to pub subscriptions [GET/PUT/DELETE]

/metadata# returns publisher-associate metadata [GET]

POST requests supply JSON objects as arguments:POST:

http://is4.cs.berkeley.edu/is4/publish/id/123456/mysubs?username=jortiz&pw=8832749823{“SubscribeTo":[13241324,234213,45456745], “UnsubscribeFrom":[45432,567365,21465765,2435786]}

Publisher Id Username and pw hash

Ongoing work

• Security– Token-based security– Tokens encapsulate access rights to resources

• Resources referenced by IS4 URI• Permissions: GET, PUT, POST, DELETE

– GET = read– PUT = create– POST = update– DELETE = delete

• Query interface– Semantic graph + timeseries

Future Work

• Incorporate security model into current implementation

• Move the data store into Amazon cloud– 14 Dents, 1 min period = ~15 MB/hour 130

GB/year (uncompressed)

• Data processing functionality– Interpolation, extrapolation, other time series

processing

• JSON streaming query engine

Summary

• Current building management systems poorly handle physical data

• Physical data is tightly bound metadata and context

• IS4 provides schemas to express metadata• IS4 implements mechanisms to manage

data/metadata binding

Thank you

• More information– http://smote.cs.berkeley.edu:8000/tracenv/wiki/is4

• Jorge Ortiz– jortiz@cs.berkeley.edu

• Questions?

Extra Slides

Snapshots of physical information

• Time t1 • Time t2

Publishers and Subscribers

Open design issues

• Doesn’t PowerMeter already do (some) of this?– They also deal with the data stream; context is not

used

• How different from pubsubhubbub?– Explicit context management

Interpreting data across contextual changes

• All the data is timeseries• Real time and historical querying

– User can be notified if context changes have occurred over query-time interval

General notes

• Talk about what I presented at the last retreat• Include previous work• Include a summary slide at the end• Update the opening slide (missing sponsors)• Look at Stephen’s slides and figure out how to

incorporate them– You have to make connections between the two

• Ask Gil for his slides to see how to segway these into that

General notes (2)