WESCML: A Data Standard for Exchanging Water and Energy Supply and Consumption Data… to support Big Data!

WESCML: A Data Standard for Exchanging Water and Energy Supply and Consumption Data… to support Big Data!

CSIRO LAND AND WATER

Bruce A. Simons, Jonathan Yu*, Benjamin Leighton Environmental Informatics – water informatics, information platforms and data ecosystems

22 August 2016

Jonathan Yu | A Water and Energy Supply and Consumption Data Hub for Australia

AURIN & WESC Data Hub Project$24 AUD million e-infrastructure project

• Secure online portal • Urban data & tools • Access: researchers and

government staff in research and policy analysis

https://aurin.org.au

Clockwise - Felix Lipkin, Magnus Moglia, Ben Leighton, Fareed Mirza, Jonathan

Yu, Matthew Inman, Bruce Simons, Ben Caradoc-Davies, Ramneek Singh,

Richard Goh

Liveable-Resilient Cities + Environmental Informatics

AURIN Team: Chris Pettit, Emma Joughin, Martin Tomko, Luca

Morandini, Serryn Eagleson, Rachel Lerm, Phil Delaney, Jack Barton, Emma Williams, Stewart Wallace, others…

2 |

https://aurin.org.au/

Jonathan Yu | WESCML. A Data Standard for exchanging water and energy supply consumption data... to support Big Data!

Outline1. Background and data challenges

2. WESCML SF-0/SF-1 data standard

3. Application of WESCML for Geospatial Big Data and Cloud Computing

3 |

Towards linked data conventions for delivery of environmental data using netCDF | Jonathan Yu

We’re not data poor“90% of the world’s data has been produced over the last two years”1

Where is it? What is it? Is it relevant?How do I use it?

4 |

1 http://www-01.ibm.com/software/data/bigdata/what-is-big-data.html

Jonathan Yu | WESCML. A Data Standard for exchanging water and energy supply consumption data... to support Big Data!5 |

(Big) data challenges

Dealing with data silos

Unconnected, unknown, unmanaged Improving data access

and connectivity

Making sense of the data

Understanding and using data in context.

Integrating it with other datasets.

http://www.wired.com/2013/03/big-data-2/

https://www.hc1.com/laboratory-informatics-and-data-silos/


Urban policy, sustainability goals, city indicators rely on better, integrated data… and at scale

6 |



7 |



waterenergy

8 |


Specific domain challenge1. Enable consistency between water/energy datasets

2. Enable improved data integration across WESC & other domains

3. “Catch ‘em all!” i.e. public/private org, NFP… nationally (~50-300 orgs)9 |


Specific domain challenge1. Enable consistency between water/energy datasets

2. Enable improved data integration across WESC & other domains

3. “Catch ‘em all!” i.e. public/private org, NFP… nationally (~50-300 orgs)10 |

Easy! Piece of cake! No worries!


Researchers


Researchers

Data sources

Data infrastructure


Researchers

Data sources

Data infrastructureLots of work in the tooling side


Reality

Need to fix the supply chain


Current data collection, analysis, reporting

Govt. Agencies

Data Custodians & Providers

Utilities

poor linkages

ad-hoc, inefficient

not well discovered

value of data not maximised

X

15 |

Policy makers

End users

Generalpublic

Researchers


Know data is available

Most cases data needs to join up

How do we enable access to do big data

analysis?


vs

Traditional database/warehouse Data ecosystem like the internetPrefer this – plug in data nodes and grow out


Know data is available

Most cases data needs to join up

How do we enable access to do big data

analysis?

Tools to build data connections

Design ‘data pipe network’ for its content

Build it!

Standardise ‘data pipes’


WESCML SF1, SF0 Data Standard

http://wescml.org

19 |

http://wescml.org/


Aggregated water/energy consumption

Utility Supply Data

Water Supply

Electricity Supply

WESCML Conceptual models Vocabulary def’s Enabling access to data

Standardised language and terminology

(easy for data to make sense)

Rapid deployment of datainfrastructure on the cloud… and at scale

(enable easy connectivity)

Connecting people with data

Data providers

Researcher/Policy Analyst

Community building

(enable access to datasets)

20 |


Minutely Hourly Daily Weekly Month Season Annual


Aggregated Consumption model (SF-0 model) - CombinedMeterReading

Postcode/ Locality/ABSGeographies

22 |


Supply Model – Water and electricity

23 |


WESC Controlled vocabularies

http://wescml.org/vocab http://demo.sissvoc.info/uwdc/vocab-viz/

24 |

http://wescml.org/vocab

http://demo.sissvoc.info/uwdc/vocab-viz/


WESC SF-1 ModelWESC info model:

• Application Schema reusing generic ISO 19000 series information models (Observations and measurements)

• Accommodates specialisation of recent OGC TimeseriesML standard

25 |


WESC SF-1 Model – modularisation of agents, commodities, monitoring locations (meters)

26 |


WESC SF-1 Model – modularisation of agents, commodities, monitoring locations (meters)

27 |


WESCML SF-0 and SF-1

28 |

Standardised language and terminology for encoding and delivery of water and energy consumption and supply data.

Easier to make sense of the data.

Gives standard data types and attributes to slice and dice data.

https://github.com/CSIRO-enviro-informatics/wescmlhttp://doi.org/10.4225/08/574D1DEEA50DD


Application of WESCML for Geospatial Big Data and Cloud Computing

29 |


wescml.org

Individual WESC Data Service infrastructure

WESCData serviceWESC XML

Schema

WESC Conformance

Tests

WESC Controlled

Vocabularies

WESC Info Model

AURIN

AURIN Portal

PostGISDB

WESC Data Service


Postcode/ Locality/ABSGeographies

Water/Elec.Data

30 |


Rapid deployment of data infrastructure on the cloud… and at scale

31 |

Rapidly spin up more

containers and connect

up data


WESC Data Hub

• Discover• View• Analyse• Visualise• Download

AURIN


Via the WESC data

protocol

32 |


Melb – Res water consumption by postcode

Source:

Yarra Valley WaterCity West WaterSouth East Water

2011Q1

33 |


Melb – Res electricity consumption vs Res water consumption by postcode

Source:

Yarra Valley WaterCity West WaterSouth East Water

2011Q1

34 |


Melb - Electricity consumption vs # separate houses

Source:

Electricity consumption (ABS household electricity 2011)

Number of Separate houses (ABS Dwelling structure from 2011 survey)

35 |


National scale datasets

WESC Data contributors/collaborators

36 |


CSIRO-AURIN WESC Data Hub – building bridges

Policy makers

End users

Govt. Agencies

Data Providers

Utilities

37 |


Takeaway #1

39 |

http://www.wired.com/2013/03/big-data-2/

Challenged by making sense of big data – volume and variety?

Particularly water/energy consumption and supply?

Aggregated water/energy consumption

Utility Supply Data

Water Supply

Electricity Supply

Conceptual models Vocabulary def’s

Standardised approach enables dealing with making sense of big data simply

WESCML


Takeaway #2

DataAnalysis

Data DataSensor

ReportProduct/ServiceProduct/Service

Do you have data silos?

Data infrastructure tools to enable

easy connectivity

40 |

Data infrastructure ecosystem


SummaryIf we want to be able to harness big data, we must fix the ‘pipes’!

Standardise and systematize each part of the data supply chain so we can deploy and connect easily and rapidly.

Socio-technical challenge.

WESCML and its application is an example of doing this for water/energy supply and consumption

41 |

LAND AND WATER

Land and WaterJonathan YuResearch data architectt +61 3 9545 2457e jonathan.yu [at] csiro.au

http://wescml.org http://aurin.org.au

Thanks and acknowledgements

Technology

WESCML: A Data Standard for Exchanging Water and Energy Supply and Consumption Data… to support Big Data!