NIST BIG DATA WGReference Architecture Subgroup

Intermediate Report

Co-chairs:Orit Levin (Microsoft)James Ketner (AT&T)Don Krapohl (Augmented Intelligence)

August 21, 2013

NIST Big Data WG / Ref Arch Subgroup 2

Starting Point From the Previous Joint Call

• Transformation includes• Processing functions• Analytic functions• Visualization functions

• Data Infrastructure includes• Data stores• In-memory DBs• Analytic DBs

Sources

Transformation

Usage

Dat

a In

fras

truc

ture

Secu

rity

Man

agem

ent

Clou

d Co

mpu

ting

Net

wor

k

8/15/2013

NIST Big Data WG / Ref Arch Subgroup 3

The Latest RA Diagram

8/15/2013

NIST Big Data WG / Ref Arch Subgroup 4

RA Current View

• The current RA view is technological, meaning that• It doesn’t show topology or deployment models• It doesn’t show business stakeholders or business relationships

• By using different sets of taxonomies, the same baseline diagram can be modified to show topologies or business relationships• These different views would need to be shown separately (e.g., technologies

vs. deployment models vs. business stakeholders)

• A proposal: Discuss these aspects in the context of the RoadMap scope and direction and sync on the direction and responsibilities

8/15/2013

NIST Big Data WG / Ref Arch Subgroup 5

Deliverables

• Delivery #1: White Paper describing different RA approaches• The proposed Outline is in M0150• Call for review• Call for volunteers to produce content for different sections

• Delivery #2: Common RA Draft• The proposed Outline is in M123• Call for additional volunteers to produce content for different sections

• A number of participants volunteered already, more are welcome

8/15/2013

NIST Big Data WG / Ref Arch Subgroup 6

Deliverable #2:Dependencies on Other Subgroups2 Big Data Reference Architecture: Use Cases Requirements• This section will be prepared by the NIST BDWG Requirements SG and will

contain high level requirements relevant to the design of the Reference Architecture.

6 Big Data Reference Architecture: Security and Privacy• This section will be prepared by the NIST BDWG Security and Privacy SG and

will contain high level security and privacy considerations relevant to the design of the Reference Architecture.

7 Big Data Taxonomy• This section will be prepared by the NIST BDWG Def&Tax SG and will contain

high level taxonomy relevant to the design of the Reference Architecture.

8/15/2013

NIST Big Data WG / Ref Arch Subgroup 7

Backup Slides

8/15/2013

NIST Big Data WG / Ref Arch Subgroup 8

Reference Architecture Objectives(No Change)• Addresses a broad range of stakeholders (e.g., data owners, industries,

academia, policy makers)• Wide scope:

• Encompasses the whole data life cycle or in the ecosystem• Can be applied to different use cases (including various verticals)• Represents different system architectures (e.g., an enterprise data warehouse,

distributed cloud-based system using multiple service providers)

• Focus• Potentially with initial focus on the Big Data analytics and tools• Assists in identifying security and privacy issues

• Agnostic to any specific technologies8/15/2013

NIST Big Data WG / Ref Arch Subgroup 9

Input for Discussion on the BD RANext Level of Details• The next slides are based on various input documents to NIST BD

subgroups• They are included here to provide insight on the evolution of the NIST

BD RA work

8/15/2013

NIST Big Data WG / Ref Arch Subgroup 10

Proposal I: Highlighting the Different DB Approaches

Secu

rity

Man

agem

ent

Clou

d Co

mpu

ting

Net

wor

k

Sources

Transformation

Usage

Data Infrastructure

Collection

Export

Curation

Pre-analytics

Visualization

Dat

a M

anag

er

Stor

age

(dis

k, m

emor

y, e

tc.)

File

Sys

tem

Spec

ializ

ed A

bstr

actio

ns

Data Mining

Analytics

Buffe

r Man

ager

Map

Red

uce

Rela

tiona

l DB

NoS

QL

DB

Aggregation

Integration

Transfer

Search

Statistics

RT Analytics

VOLUMEVARIETY

VELOCITY

Streaming

Interactive An.

Batch Analytics

8/15/2013

NIST Big Data WG / Ref Arch Subgroup 11

Proposal II: Based on Submitted Requirements

8/15/2013