View
374
Download
1
Category
Preview:
Citation preview
Analyzing Data With DockerAndreas Dewes (@japh44)
EuroPython 2016 - Bilbao
Outline
Data Analysis: Small & Large-Scale, Easy & Difficult
Introduction To Docker
Containerizing our Data Analysis
Possible Approaches
Relevant Technologies & Outlook
Data Analysis: Use Cases
small-scale large-scale
automated
interactive
Interactive, UI-based analysis(e.g. iPython notebook)
analysis scripts usingLocal data sources(e.g. databases)
non-interactive analysis pipelines(e.g. Apache Hadoop)
Interactive “Big Data” tools, e.gApache Spark or Google BigQuery
So what's so difficult about data analysis?
Sharing Data & Tools
Reproducibility
Scaling
Enter Docker....
What is Docker?
A tool that allows us to deploy applications inside "software containers".
Containers work at the process level and isolate the view of the operating system (i.e. the processes, resources and files an application sees)
Provides a high-level API to manage, version-control, deploy and network containers.
Docker Swarm
Docker Core-Concepts
Docker EngineDocker Engine
Docker API
Registry
CLI
Image
Image
ImageContainer
Container
Container
Container
Container
Images Are Space-Efficient(or at least more efficient than VMs)
Containers Have Little Overhead
https://domino.research.ibm.com/library/cyberdig.nsf/papers/0929052195DD819C85257D2300681E7B/$File/rc25482.pdf
Containers Are Self-Sufficient
Containers Are "Lego" For Data Analytics!
Container
output
inputsconfiguration
datanetworked containers
We Can Build Reproducible Data-Analysis Workflows With Them
Map Apache
logs
Map Nginx logs
BI
Aggregate results Filtering Monitoring
Archiving
Example: Analyzing Github Data
analysis script
log filesfrom Github
output
analysis process(es)
Repository with code: https://github.com/adewes/docker-map-reduce-example
Live Demo (fingers crossed)
Containerizing Our Analysis
analysis script
log filesfrom Github
output
analysis container
image
analysis container
analysis container
supervisor
Live demo (what could go wrong?)
Advantages DisadvantagesEasy to share
Each analysis step is self-sufficient
Analysis components are "plug & play"
Easy to parallelize (for the right problems)
Versioning included
Requires to prepare containers
Requires Docker on each machine
Slightly decreases interactivity & flexibility
Which Parts Are Missing?
Orchestration
Dependency Management
Resource ManagementResource Management
Rouster:A Python Tool for Containerized Data Analysis
Built on top of the Docker API"Make for Docker"
Resource ManagementContainer OrchestrationDependency Management
Rouster Uses Recipes to Describe Data Analysis Workflows
Resources(including dependencies)
Services
Actions
versioning, dependency calculation,backup / copying, distribution, ...
startup (including dependencies),resource provisioning, networking, ...
scheduling, monitoring, exceptionhandling, logging, ...
Live Demo: CSV -> Postgres
Open Questions
How to handle communication between containers(through files, network, ...)?
How to provide resources/data to containers in adistributed environment?
Pachyderm is a data lake that offers complete version control for data and leverages the container ecosystem to provide reproducible data processing. Built on top of Kubernetes.
http://www.pachyderm.io
Pachyderm
LuigiLuigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
https://github.com/spotify/luigi
Other relevant technologies
Summary & Outlook
Containers are here to stay!
They are useful in various data analysis contexts.
They don't solve all our problems though.
We need additional tools to use them effectively.
Thanks!Want to contribute?
https://github.com/7scientists/rouster
Andreas Dewes (@japh44)
Image Licenses:
https://commons.wikimedia.org/wiki/File:Matryoshka_dolls_(3671820040)_(2).jpghttps://pixabay.com/de/nordlichter-lager-zelt-abenteuer-1203289/https://en.wikipedia.org/wiki/Orchestrahttps://de.wikipedia.org/wiki/Graph_(Graphentheorie)http://www.library.illinois.edu/prescons/disaster_response/high_density_storage_disaster_plan/https://brookeborel.com/2011/06/02/363/https://en.wikipedia.org/wiki/Data_sharing
Recommended