Embed Size (px)
Citation preview
@xebiconfr #xebiconfr
Data Lake done
right!Matthieu
Blanc
@xebiconfr #xebiconfr 2
Data Lake?
@xebiconfr #xebiconfr
WHY?
@xebiconfr #xebiconfr
Centralisation
4
@xebiconfr #xebiconfr
Self Service
5
@xebiconfr #xebiconfr 6
Data lakes will only succeed if they become shared resources.
@xebiconfr #xebiconfr
Challenge Ahead
7
@xebiconfr #xebiconfr 8
Complex Ecosystem
@xebiconfr #xebiconfr 9
Skill Gap
@xebiconfr #xebiconfr 10
A Data Lake need to be managed
@xebiconfr #xebiconfr
HISTORY
@xebiconfr #xebiconfr
Siloed Data
12
@xebiconfr #xebiconfr
Distributed File System
13
@xebiconfr #xebiconfr
Technologies
14
Hadoop
AWS S3
Google Cloud Storage
@xebiconfr #xebiconfr 15
@xebiconfr #xebiconfr
Data Warehouse
16
@xebiconfr #xebiconfr
Data Lake
17
@xebiconfr #xebiconfr
Data Mart
18
@xebiconfr #xebiconfr
Data Lake
19
@xebiconfr #xebiconfr
Data Swamp?
20
@xebiconfr #xebiconfr
HOW?
@xebiconfr #xebiconfr
Data Landfill
22
Data Scientists in front of raw data in a “Data Lake”
@xebiconfr #xebiconfr
Data Catalog
23
@xebiconfr #xebiconfr
Metadata Repository
24
Datasets SearchV1 V2
API
Web UI
...
Metadata
Catalog
@xebiconfr #xebiconfr 25
SQL
Clean, trusted, prepared Data
Raw data
@xebiconfr #xebiconfr 26
@xebiconfr #xebiconfr
Automate Data Provisioning
27
Raw data Master Data
CSV, JSON, XML, Logs ... Parquet, Avro, ORC
@xebiconfr #xebiconfr
Data Organization
28
@xebiconfr #xebiconfr
Data Zones
29
LandingZone
Master DataZone
Work Area
ConsumptionZoneData
Sources
Data Sources
Data Sinks
Change Data Capture Zone
@xebiconfr #xebiconfr
Quality Control
30
@xebiconfr #xebiconfr
Data Governance
31
Raw data Master Data
CSV, JSON, XML, Logs ... Parquet, Avro, ORC
Validation criteria
Web UI
Operational/statistical metadata
@xebiconfr #xebiconfr
Security?
32
@xebiconfr #xebiconfr
Data Zones
33
LandingZone
Clean DataZone
Work Area
ConsumptionZoneData
Sources
Data Sources
Data Sinks
@xebiconfr #xebiconfr
LandingZone
Enforce security rule during data transformation
34
Clean DataZone
Sensitive Data
Work Area
ConsumptionZoneData
Sources
Data Sources
Data Sinks
@xebiconfr #xebiconfr
Enforce security rule during data transformation
35
Raw data Master Data
CSV, JSON, XML, Logs ... Parquet, Avro, ORC
Data privacy metadata
Web UI
sensitive data encrypted
some data marked as sensitive
@xebiconfr #xebiconfr
Avoid Data Swamp
Catalog your data
Automate Data Provisioning
Create Governance Zones
Provide Data Discovery Tools
@xebiconfr #xebiconfr 37
Data Democratization
@xebiconfr #xebiconfr 38
Most Data Lakes initiatives will fail
@xebiconfr #xebiconfr
XData
39
@xebiconfr #xebiconfr
Thankyou!Matthieu Blanc
@xebiconfr #xebiconfr 41
Mainframe RDBMS NoSQL DBs Logs DWH Queues
HR Financial CRM Web BI Social Media
