SC7 Workshop 1: Big Data in Secure Societies

BIG DATA EUROPE

Integrating Big Data, Software & Communities for Addressing Europe’s Societal Challenges

negative conotation

Big Data has often a

6-oct.-15www.big-data-europe.eu

Big Data in Marketing


Big Data in Intelligence


BigDataEurope aims to help maximizing

the societal value of Big Data

Health, demographic change and wellbeing;

Food security, sustainable agriculture and forestry, marine and maritime and inland water research, and the Bioeconomy;

Secure, clean and efficient energy;

Smart, green and integrated transport;

Climate action, environment, resource efficiency and raw materials;

Europe in a changing world - inclusive, innovative and reflective societies;

Secure societies - protecting freedom and security of Europe and its citizens.


The three Big Data „V“ – Variety

is often neglected

Quelle: Gesellschaft für Informatik

© Fraunhofer-Allianz Big Data 7

Proactive Maintenance at Rolls Royce

New Business Model integrating Sensor Data & Big Data Analytics

Dr. Dirk Hecker

Condition Monitoring, Proactive maintenance, „Power-by-the-hour“,

as-a-service Business Model – payment modell by flight hours

Quelle: www.springboeck.ch/SR_Technics.htm

© Mark Hillary | Flickr


The rolling Smartphone

New Business Models for the Automotive Industrywith Data Value Chains

Dr. Dirk Hecker

Windshield wiper as rain sensors for micro wether prognosis

Automotive industry can become data provider for other industries

Qu

elle

: G

TÜQ

uelle

: ww

w.fa

rmin

g-sim

ula

tor.co

m


Predictive Analytics

Dr. Dirk Hecker

From Business Intelligence to Big Data Analytics

Business Intelligence Monitoring Predictive Analytics

What happenedbefore?

What happensnow?

What will happen soon?

What shouldhappen?

Prescriptive Analytics

„the last Mile“

“prescriptive analytics suggests decision options on how to take advantage of a future opportunity”

Quelle: BMW Quelle: www.7-forum.com Quelle: BMW Quelle: Volvo

BigDataEurope Rationale

Show societal value of Big Data

Lower barrrier for using big data technologies

o Required effort and resources

o Limited data science skills

o Lack of Generic Architectures, components

Help establishing cross-

lingual/organizational/domain Data Value Chains

o Multiple Data Sources

o Required: Integration, Harmonisation


BigDataEurope: Objectives


COORDINATION

Stakeholder Engagement

(Requirements Elicitation)

SUPPORT

Design, Realise, Evaluate

Big Data Aggregator

Platform

Create and Manage

Societal Big Data

Interest Groups

Cloud-deployment

ready

Big Data Aggregator

Platform

CSA

Measures

Results

Orthogonal Dimensions of Big Data Ecosystems

Generic Big Data Enabling Technologies

Data Value Chain

Data Generation & Acquisition

Data Analysis & Processing

Data Storage & Curation

Data Visualization &

Usage

Data-driven Services

So

cie

tal

Ch

all

en

ge

s

Do

ma

in S

pe

cifi

c D

ata

Ass

ets

& T

ech

no

log

y

Healthcare

Food Security

Energy

Intelligent Transport

Climate & Environment

Inclusive & Reflective Societies

Secure Societies

Stakeholder Engagement Cycle

© Fraunhofer-Institut für Intelligente Analyse- und Informationssysteme IAIS

Open-Source Technologies for Big Data Apps (small selection :-)

14


Big Data - Technologies

Volume

VelocityVariety

Storm

15


Groups of Technologies

Big Data Technolog

ies

Data Storage Technologies

Data Processin

g

Workflow Coordinati

on

Querying/

Processing

Search

Data Export/ Import

Data Analysis

Statistics

Text Mining

16


Big Data Requirements

Analysis of historical dta

Millions of entries

Varying analysis quesitions

Years of input data

=> Big Data Batch Processing

Interactive analysis by online queries

Thousands of users online

Extremely fast response time

Super high availability

=> Big Data Databases

Analysis of actual data with low latency in

"real-time"

React to newest trends

Low-Latency change detection

Real-time online monitoring

=> Big Data Stream Processing

But how to put it together ?17


The „traditional“ Hadoop Ecosystem + NoSQL

components

a Big Data Management System

ZooKeeper

askaban

Kafka

cassandra

voldemort

MongoDB

CouchDB

elastic search

solr

lucene


Batch

Function

Speed

Function

Data Storage

pages with

postings

Batch View

Realtime

View

me

ssa

ge

pa

ssin

g

message passing

Application

Horizontal Scalability in the Lambda Architecture

19

> volume

> users

> users, volume

> velocity> volume, velocity

Blueprint of the Data Aggregator Platform

Follows typical Lambda Architecture

Integrated on top of existing Big Data distribution

+ Semantic Layer (Retaining Semantics using LD approach )

Batch Layer

Speed Layer

Data Storage

Real-time data &

Transactions …

Batch View

Real-time

View

mess

ag

e p

ass

ing

message passing

Applications & Showcases

Real-time dashboards

Domain-specific BDE apps

Big Data Analytics

In-stream Mining

BD

E P

latfo

rm&

Intellig

ence

Input data

Stream

Spatial

Social

Statistical

Temporal

Transactional

Imagery

BDE Platform based on BigTop

Packaging Smoke testing Virtualization

Package RPMs and DEBs, so

that you can manage and

maintain your own cluster.

Integrated smoke testing

framework

Vagrant recipes, raw images,

and docker recipes for

deploying BigData

infrastructures from zero.

6-oct.-15www.big-data-europe.eu+ Semantic Layer - Retaining Semantics using Linked Data

Data Aggregator Platform Challenges

Ingest semantic (RDF) and non-semantic (CSV,

JSON, XML, …) data

o Integrate various mapping techniques (R2RML, CSV on

the Web, JSON-LD)

preserve semantics, provenance and metadata in

Big Data processing chains

o Preserve URI/IRIs

o Preserve triples

Exploit semantics for aggregations


Current Activities – Year#1

2015 BDE Societal Workshops (7) Planned

o Schedule on Website

7 W3C Interest Groups set up: Please Join!o SC1: HEALTH https://www.w3.org/community/bde-health/join

o SC2: FOOD & AGRICULTURE https://www.w3.org/community/bde-food/

o SC3: ENERGY https://www.w3.org/community/bde-energy/

o SC4: TRANSPORT https://www.w3.org/community/bde-transport/

o SC5: CLIMATE & ENVIRONMENT https://www.w3.org/community/bde-climate/

o SC6: SOCIETIES https://www.w3.org/community/bde-societies/

o SC7: SECURITY https://www.w3.org/community/bde-secure-societies/

www.big-data-europe.eu

http://www.big-data-europe.eu/first-round-of-bigdataeurope-workshops-announced/

https://www.w3.org/community/bde-health/join

https://www.w3.org/community/bde-food/

https://www.w3.org/community/bde-energy/

https://www.w3.org/community/bde-transport/

https://www.w3.org/community/bde-climate/

https://www.w3.org/community/bde-societies/

https://www.w3.org/community/bde-secure-societies/

BDE Partners

Sören Auer

Big Data Europe Coordinator

Fraunhofer IAIS & University of Bonn

[email protected]

Thanks


mailto:[email protected]

Energy/Climate Example: Greenshifting


Technology

SC7 Workshop 1: Big Data in Secure Societies