Upload
others
View
10
Download
0
Embed Size (px)
Citation preview
GMU Big Data Symposium, NIST Big Data Stds Activities, Wo Chang, Jan. 7, 2016 1
NIST Big Data PWG & Standardization Activities
Wo Chang, NISTDig i ta l Data Adv i so r
N IST B ig Data Pub l i c Work ing Group , Co ‐Cha i rI SO/ IEC J TC 1/WG 9 Work ing Group on B ig Data , Convenor
wchang@ni st . gov
J anua r y 7 , 2016
GMU Big Data Symposium, NIST Big Data Stds Activities, Wo Chang, Jan. 7, 2016GMU Big Data Symposium, NIST Big Data Stds Activities, Wo Chang, Jan. 7, 2016
NIST Big Data Public Working Group - Goal
2
Develop a secured reference architecture that is vendor-neutral, technology- and infrastructure-agnostic to enable any stakeholders (data scientists, researchers, etc.) to perform analytics processing for their given data sources without worrying about the underlying computing environment.
GMU Big Data Symposium, NIST Big Data Stds Activities, Wo Chang, Jan. 7, 2016GMU Big Data Symposium, NIST Big Data Stds Activities, Wo Chang, Jan. 7, 2016 3
Internet of Things
Analytics Engine
Social Media
Electronic Health Record Life Science
Others…
GMU Big Data Symposium, NIST Big Data Stds Activities, Wo Chang, Jan. 7, 2016GMU Big Data Symposium, NIST Big Data Stds Activities, Wo Chang, Jan. 7, 2016 4
Data Sources‐ Sensors‐ Simulations‐ Modeling‐ Etc.
NIST / ISO Big Data Standards – A Big Roadmap
Data Consumers‐ End users‐ Repositories ‐ Systems‐ Etc.
Data Scientist
BDRA InterfaceResource Management/Monitoring, Analytics Libraries, etc.
BDRA Ecosystem Components
Computing Resources
AnalyticsResources
Distributed File System ServicesInfrastructure Services
Database ServicesData Sources ServicesSupport Infrastructure
Value-added Content ServicesSecurity and Privacy Services
Visualization & BI ServicesAnalytics Services
Analytics Application
GMU Big Data Symposium, NIST Big Data Stds Activities, Wo Chang, Jan. 7, 2016GMU Big Data Symposium, NIST Big Data Stds Activities, Wo Chang, Jan. 7, 2016
NIST Big Data Standardization Activities
5
Approaches to Establish Interoperable Ecosystem
NIST Big Data Public Working Group (NBD-PWG)
ISO/IEC JTC 1/WG 9 Working Group on Big Data, with collaborations:
ISO/IEC JTC 1/WG 11 – MPEG
ISO/TC 69 – Applications of Statistical Methods
ISO/TC 204 – Intelligent Transportation
Work Across Academic, Industry, and Standards to achieve interoperability
GMU Big Data Symposium, NIST Big Data Stds Activities, Wo Chang, Jan. 7, 2016
NIST Big Data Standardization Activities
5 Subgroups (July 2013 – now):
1. Definitions & Taxonomies
2. UC & Requirements
3. Security & Privacy
4. Reference Architecture
5. Standards Roadmap
V1 (high-level RA components and descriptions) Big Data Interoperability Framework:Released on September 16, 2015:http://bigdatawg.nist.gov/V1_output_docs.php
6
NIST Big Data Public Working Group (NBD-PWG)
NIST SP1500-1: Definitions
NIST SP1500-1: Definitions
NIST SP1500-2: Taxonomies
NIST SP1500-2: Taxonomies
NIST SP1500-3: Use Cases &
Requirements
NIST SP1500-3: Use Cases &
Requirements
NIST SP1500-4: Security &
Privacy
NIST SP1500-4: Security &
Privacy
NIST SP1500-5: Architecture
Survey – White Paper
NIST SP1500-5: Architecture
Survey – White Paper
NIST SP1500-6: Reference
Architecture
NIST SP1500-6: Reference
Architecture
NIST SP1500-7: Standards Roadmap
NIST SP1500-7: Standards Roadmap
GMU Big Data Symposium, NIST Big Data Stds Activities, Wo Chang, Jan. 7, 2016
NIST Big Data Standardization Activities
7
NIST Big Data Public Working Group (NBD-PWG)
7CODATA Big Data Workshop, Wo Chang, NIST/ITL, June 9, 2014
Vendors Big Data architectures
GMU Big Data Symposium, NIST Big Data Stds Activities, Wo Chang, Jan. 7, 2016GMU Big Data Symposium, NIST Big Data Stds Activities, Wo Chang, Jan. 7, 2016
NIST Big Data Standardization Activities
8
V2 focuses on interface between NBD-RA components through use cases by:
Analyze activities diagrams
Analyze functional diagrams
Apply DevOps on small scale implementations
Goals:
Aggregate low-level interactions into high-level general interfaces
Produce set of white papers to demo how NBD-RA can be used
NIST Big Data Public Working Group (NBD-PWG)
GMU Big Data Symposium, NIST Big Data Stds Activities, Wo Chang, Jan. 7, 2016GMU Big Data Symposium, NIST Big Data Stds Activities, Wo Chang, Jan. 7, 2016
NIST Big Data Standardization Activities
9
Selection of use cases: (a) available of datasets and (b) available of analytics codes
Fingerprints Matching Human and Face Detection from Video
Twitter Feeds Spatial Big Data/GIS Healthcare Payment Fraud
• Data warehousing• Global Cities
• Earth Science• Life Science
• IoT• Others…
NIST Big Data Public Working Group (NBD-PWG)
GMU Big Data Symposium, NIST Big Data Stds Activities, Wo Chang, Jan. 7, 2016GMU Big Data Symposium, NIST Big Data Stds Activities, Wo Chang, Jan. 7, 2016
NIST Big Data Standardization Activities
10
NISTGlobal City
TeamsChallenges NIST
Cyber-PhysicalSystems
PWG
IEEEInternet of
Things (IoT)
NISTCloud PWG
NIST Big Data
PWG
ISOSTANDARDSSCS/WGS
JTC 1/WG10Working Group
OnInternet of
Things (IoT)
JTC 1/WG9Working Group
On Big Data
JTC 1/SC32Data
Management andInterchange
JTC 1/SC38Cloud Computing
and Distributed Platforms
JTC 1/SC27Security and
Privacy
NISTPUBLIC
WORKINGGROUPS
Explore collaboration by working with industry, academic and governments to harmonize analytic ecosystems
GMU Big Data Symposium, NIST Big Data Stds Activities, Wo Chang, Jan. 7, 2016GMU Big Data Symposium, NIST Big Data Stds Activities, Wo Chang, Jan. 7, 2016
ISO/IEC Big Data Standardization Activities
11
ISO/IEC JTC 1/WG 9 Working Group on Big Data
130+ from 21 NBs: Australia, Austria, Brazil, Canada, China, Finland, France, Germany, Ireland, Italy, Japan, Korea, Luxembourg, Netherlands, Norway, Russian Federation, Spain, Singapore, Sweden, UK, US
Current Projects
• ISO/IEC 20546 Information technology – Big data – Overview and vocabulary
• ISO/IEC 20547 Information Technology – Big data Reference architecture (5 Parts)
Part 1: (TR) Framework and Application Process Part 2: (TR) Use Cases and Derived Requirements Part 3: (IS) Reference Architecture Part 4: (IS) Security and Privacy Fabric Part 5: (TR) Standards Roadmap
ISO/IEC Liaisons: SC 6/WG 7, SC 27, SC 29, SC 32, SC 36, SC 38, SC 39, ISO/TC 69, ISO/TC 204, ITU-T SG13
GMU Big Data Symposium, NIST Big Data Stds Activities, Wo Chang, Jan. 7, 2016GMU Big Data Symposium, NIST Big Data Stds Activities, Wo Chang, Jan. 7, 2016
ISO/IEC Big Data Standardization Activities
12
Identify and characterize existing multimedia Big Data deployment
Identify Big Media use cases
Identify MPEG tools relevant for Big Media
ISO/IEC JTC 1/SC 29/WG 11 (MPEG) on Big Media
Create AHG between SC 29/WG11 and WG9 to
GMU Big Data Symposium, NIST Big Data Stds Activities, Wo Chang, Jan. 7, 2016GMU Big Data Symposium, NIST Big Data Stds Activities, Wo Chang, Jan. 7, 2016
ISO/IEC Big Data Standardization Activities
13
ISO/TC69 – Applications of Statistical Methods
Apply standard statistical methodologies (CRISP, SEMMA, etc.)
Create AHG between TC69, WG9, and NIST Big Data PWG to:
Explore new Big Data statistical methods
Identify use cases (healthcare fraud, live twitter feeds, etc.)
Implement use cases using best practice Big Data computing ecosystem
Document findings
Standardize new Big Data statistical methodologies
GMU Big Data Symposium, NIST Big Data Stds Activities, Wo Chang, Jan. 7, 2016GMU Big Data Symposium, NIST Big Data Stds Activities, Wo Chang, Jan. 7, 2016
ISO/IEC Big Data Standardization Activities
14
ISO/TC204 – Intelligent Transportation
Apply standard statistical methodologies (CRISP, SEMMA, etc.)
Create AHG between TC204 and WG9 to
Review SDOs in the Big Data area particularly architecture models, semantic definitions, metadata issues and APIs
Identify “Big Data topics” needed for transport data exchange and external data sources; gather and / or generate use cases related to big data topics for ITS
Examine TC204 work that support the Big Data areas and identify the gaps to fit into the foundation / architecture currently under development by SDOs (e.g., ISO/IEC, IEEE, SAE)
Examine security, privacy, ownership, and usage issues related to Big Data ITS applications
Recommend future work items (if any) to be developed by TC204
Recommend liaisons with SDOs for which collaboration is needed
GMU Big Data Symposium, NIST Big Data Stds Activities, Wo Chang, Jan. 7, 2016GMU Big Data Symposium, NIST Big Data Stds Activities, Wo Chang, Jan. 7, 2016
Work Across Academic, Industry, and Standards to achieve interoperability
15
Research(‐ Google log file
‐ academic)
Development
Deployment
Community/Industry/Product
BigTable(2002)
• Hadoop• Hbase, Hive
• Others…
• Cloudera Impala• IBM BigInsights
• Hortonworks HDP• Others…
Custom
er‐based
Standards D
evelop
ment
Tradition
al
Standards D
evelop
ment Focuses/Activities
• Functional Research• Apply Research• Experimental• Testbed• Best Practices• Consortium• Industry Practices/ Standards
Focuses/Activities• Functional Research• Apply Research• Experimental• Testbed• Best Practices• Consortium• Industry Practices/ Standards
Focuses/Activities• Functional Research• Apply Research• Experimental• Testbed• Best Practices• Consortium• Industry Practices/ Standards Ac
adem
ic and
R&D Labs