Apache Atlas: Tracking dataset lineage across Hadoop components

Apache Atlas: Tracking dataset lineage across Hadoop components

2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Disclaimer

This document may contain product features and technology directions that are under development, may be under development in the future or may ultimately not be developed.

Project capabilities are based on information that is publicly available within the Apache Software Foundation project websites ("Apache"). Progress of the project capabilities can be tracked from inception to release through Apache, however, technical feasibility, market demand, user feedback and the overarching Apache Software Foundation community development process can all effect timing and final delivery.

This document’s description of these features and technology directions does not represent a contractual commitment, promise or obligation from Hortonworks to deliver these features in any generally available product.

Product features and technology directions are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.

Since this document contains an outline of general product development plans, customers should not rely upon it when making purchasing decisions.


Speakers

Andrew AhnGovernance Director Product Management


Agenda

• Atlas Overview• Near term roadmap• Cross Component Lineage• Questions


Apache Atlas Overview


STRUCTURED

UN

STRUCTU

RED

Vision - Enterprise Data Governance Across Platfroms

TRADITIONALRDBMS

METADATA

MPP APPLIANCES

Project 1

Project 5

Project 4

Project 3

Metadata

Project 6

DATALAKE

STREAMING

GOAL: Provide a common approach to data governance across all systems and data within the enterprise

TransparentGovernance standards and protocols must be clearly defined and available to allReproducibleRecreate the relevant data landscape at a point in timeAuditableAll relevant events and assets but be traceable with appropriate historical lineageConsistentCompliance practices must be consistent


Ready for Trusted Governance

OPERATIONS SECURITY

GOVERNANCE

STORAG

ESTO

RAG

E

MachineLearningBatch

StreamingInteractive

Search

GOVERNANCE

YA R ND A T A O P E R A T I N G S Y S T E M

Data Managementalong the entire data lifecycle with integrated provenance and lineage capability

Modeling with Metadataenables comprehensive data lineage through a hybrid approach with enhanced tagging and attribute capabilities

Interoperable Solutionsacross the Hadoop ecosystem, through a common metadata store


DGI* Community becomes Apache Atlas

May2015

Proto-typeBuilt

Apache AtlasIncubation

DGI groupKickoff

Feb2015

Dec 2014

July2015HDP 2.3 FoundationGA Release

First kickoff to GA in 7 months

Global FinancialCompany

* DGI: Data Governance Initiative

Faster & SaferCo-Development driven by customer use cases


Apache Atlas: Metadata Services

• Cross- component dataset lineage. Centralized location for all metadata inside HDP

• Single Interface point for Metadata Exchange with platforms outside of HDP

• Business Taxonomy based classification. Conceptual, Logical And Technical

Apache Atlas

Hiv

e

Ran

ger

Falc

on

Sqoo

p

Stor

m

Kaf

ka

Spar

k

NiF

i


Big Data Management Through Metadata

Management ScalabilityMany traditional tools and patterns do not scale when applied to multi-tenant data lakes. Many enterprise have silo’d data and metadata stores that collide in the data lake. This is compounded by the ability to have very large windows (years). Can traditional EDW tools manage 100 million entities effectively with room to grow ?

Metadata Tools

Scalable, decoupled, de-centralized manage driven through metadata is the only via solution. This allows quick integration with automation and other metamodels

Tags for Management, Discovery and Security

Proper metadata is the foundation for business taxonomy, stewardship, attribute based security and self-service.


Apache Atlas High Level Architecture

Type System

Repository

Search DSL

Brid

ge

Hive Storm

Falcon Others

REST API

Graph DB

Sear

ch

Kafka

SqoopCo

nnec

tors

Mes

sagi

ng F

ram

ewor

k


Near Term Roadmap: Summer 2016


Dynamic Access Policy Driven by metadata


Business Catalog

Breadcrumbs for taxonomy context path

Contents at taxonomy context


Hadoop Cross Component Data Lineage


Sqoop

TeradataConnector

ApacheKafka

Atlas: Tracks Metadata + Lineage in one place

Custom Activity Reporter

MetadataRepository

RDBMS


Technical and Logical Metadata Exchange

Knowledge Store

AtlasREST API

StructuredUnstructured

Files:XML / JSON

3rd Party Vendors

CustomReporter

Non-Hadoop


Hive Integration: Model for integration

Apache Atlas

Hive Bridge(Client)

Hive Hook(Post-execution)

REST API


HDF: Dataflow Governance Solution

20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved HORTONWORKS CONFIDENTIAL & PROPRIETARY INFORMATION

Dataflow Security Use case Requirements

Accelerated Data Collection: An integrated, data source agnostic collection platform

Increased Security and Unprecedented Chain of Custody: Secure from source to storage with high fidelity data provenance

The Internet of Any Thing (IoAT): A Proven Platform for the Internet of Things

http://hortonworks.com/hdf/


Enterprise Grade Governance Dataflow Solution

Filtered Metadata

• HDP Taxonomy• Centrallized

Metadata Repository

• Downstream HDP Impacts

• Cross component lineage

• 3rd Party integration

• Guaranteed Delivery

• Data Buffering• Prioritized

Queueing• Flow specific QoS• Visual Command

& ControlMonthsLineage

YearsLineage

ReferenceTaxonomy

(Tags)

Event level versus Dataset

level

HDF - NiFI

Operation Control

MaximumFidelity

Event Level

HDP – Atlas

GovernanceManagement

Medium / LowFidelity

Dataset Level


Demo

• Tutorial• Atlas Tour• Sqoop Lineage• Kafka / Storm Linage


Availability: - Tech Preview VMs: May 2016 - GA Release: Summer 2016


Questions ?


Reference


Online Resources

VM: https://s3.amazonaws.com/demo-drops.hortonworks.com/HDP-Atlas-Ranger-TP.ova —> Download Public Preview VM

Tutorial: https://github.com/hortonworks/tutorials/tree/atlas-ranger-tp/tutorials/hortonworks/atlas-ranger-preview

Blog: http://hwxjojo.wpengine.com/blog/the-next-generation-of-hadoop-based-security-data-governance/ (this is giving an error, right now)

Learn More: http://hortonworks.com/solutions/atlas-ranger-integration/

https://s3.amazonaws.com/demo-drops.hortonworks.com/HDP-Atlas-Ranger-TP.ova



https://github.com/hortonworks/tutorials/tree/atlas-ranger-tp/tutorials/hortonworks/atlas-ranger-preview

https://github.com/hortonworks/tutorials/tree/atlas-ranger-tp/tutorials/hortonworks/atlas-ranger-preview

http://hwxjojo.wpengine.com/blog/the-next-generation-of-hadoop-based-security-data-governance/



http://hortonworks.com/solutions/atlas-ranger-integration/


Thank You

Technology

Apache Atlas: Tracking dataset lineage across Hadoop components