Python in an Evolving Enterprise System (PyData SV 2013)

DMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMD MMM8OOOOOOOOOOO8MMMM8OOOOOOOOOOOOOOODMMMMOOOOOOOOOOOOOOMMMN DMMIIIIIIIIIIIII$MMMM$IIIIIIIIIIIIIIIOMMMM7III?IIIIIIIIII7MM MMOIIIIIIIIIIIII7MMMMOIIIIIIIIIIIIIIIIMMMM8IIIIIIIIIIIIIIIMM MMOIIIIIIIIIIIIIIMMMMDIIIIIIIIIIIIIIIIDMMMM7IIIIIIIIIIIIIIMM MMOIIIIIIIIIIIIIIMMMMM7IIIIIIIIIIIIIII?MMMMMIIIIIIIIIIIIIIMM MMOIIIIIIIIIIIIIIMMMMM8IIIIIIIIIIIIIIIIMMMMMOIIIIIIIIIIIIIMM MMOIIIIIIIIIIIIIIOMMMMMIIIIIIIIIIIIIIIIIMMMMMMIIIIIIIIIIIIMM MMOIIIIIIIIIIIIIIIMMMMMIIIIIIIIIIIIIIIIIZMMMMMMIIIIIIIIIIIMM MMOIIIIIIIIIIIIIIIZMMMM8IIIIIIIIIIIIIIIII7MMMMMMOIIIIIIIIIMM MM8$$IIIIIIIIIIIIIIMMMMMIIIIIIIIIIIIIII?III8MMMMMMMZIIIIIIMM MMMMMMMMMMN87IIIIII8MMMMDIIIIIIIIIIIIIIIIIIIZMMMMMMMMMNZIIMM MMMMMMMMMMMMMMMMMOII$MMMMMIIIIIIIIIIIIIIIIIIIIIMMMMMMMMMMMMM MMMMMMMMMMMMMMMMMMMM8MMMMMZIIIIIIIIIIIIIIIIIIIIII8MMMMMMMMMM MMOIIIIIIII7NMMMMMMMMMMMMMMMIIIIIIIIIIIIIIIIIIIIII?IIII$ODMM MMOIIIIIIIIII?I8MMMMMMMMMMMMDIIIIIIIIIIIIIIIIIIIIIIIIIIIIIMM MMOIIIIIIIIIIIIIIIIZMMMMMMMMMM7II?IIIIIIIIIIIIIIIIIIIIIIIIMM MMOIIIIIIIIIIIIIIIIII7NMMMMMMMMIIIIIIIIIIIIIIIIIIIIIIIIIIIMM MMOIIIIIIIIIIIIIIIIIIIIIDMMMMMMMMIIIIIIIIIIIIIIIIIIIIIIIIIMM MMOIIIIIIIIIIIIIIIIIIIIIIIMMMMMMMMMIIIIIIIIIIIIIIIIIIIIIIIMM MMOIIIIIIIIIIIIIIIIIIIIIIII7MMMMMMMM8IIIIIIIIIIIIIIIIIIIIIMM MMOIIIIIIIIIIIIIIIIIIIIIIIIII7MMMMMMMMM$IIIIIIIIIIIIIIIIIIMM MMOIIIIIIIIIIIIIIIIIIIIIIIIIIIOMMMMMMMMMM8I?IIIIIIIIIIIIIIMM MMOIIIIIIIIIIIIIIIIIIIIIIIIIIII7MMMMMMMMMMMMMN7IIIIIIIIIIIMM MMMMMMD$IIIIIIIIIIIIIIIIIIIIIIIIIMMMMMMMMMMMMMMMMDZIIIIIIIMM MMMMMMMMMMMNIIIIIIIIIIIIIIIIIIIIIINMMMMDZNMMMMMMMMMMMMMMMMMM MMMMMMMMMMMMMM?IIIIIIIIIIIIIIIIIIIIMMMMMIII7NMMMMMMMMMMMMMMM MMOIII7DMMMMMMMM$IIIIIIIIIIIIIIIIIIZMMMMNIIIIIIII7$DNMMMMMMM MMOIIIIII7MMMMMMM7IIIIIIIIIIIIIIIIIIOMMMM8IIIIIIIIIIIIIIIIMM MMOIIIIIIIIIMMMMMMNIIIIIIIIIIIIIIIIIIMMMMMIIIIIIIIIIIIIIIIMM MMOIIIIIIIIIIIMMMMMNIIIIIIIIIIIIIIIIIDMMMM7IIIIIIIIIIIIIIIMM MMOIIIIIIIIIIIIMMMMMMIIIIIIIIIIIIIIII7MMMMDIIIIIIIIIIIIIIIMM MMOIIIIIIIIIIIIIMMMMMZIIIIIIIIIIIIIIIIOMMMM$IIIIIIIIIIIIIIMM MMOIIIIIIIIIIIIIIMMMMMIIIIIIIIIIIIIIIIIMMMM8IIIIIIIIIIIIIIMM MMOIIIIIIIIIIIIIIMMMMMNIIIIIIIIIIIIIIIIMMMMMIIIIIIIIIIIIIIMM MMOIIIIIIIIIIIIIIOMMMMMIIIIIIIIIIIIIIIIMMMMMI??IIIIIIIIIIIMM $MMIIIIIIIIIIIIIIIMMMMMIIIIIIIIIIIIIIIIMMMMMIIIIIIIIIIIII8MM MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM 777777777777777777777777777777777777777777777777I77 MD$N? MMM8MN OMM8MZ OMMMDM MMDM+ MD~NO M= MM ZZMI +ZI M7 M O7 MO OM M OMMMMM 8M7 M= MM MN7 MM?+I M7 M O7 MO OM M O+ 8M7 M~ MM ZMI MDMMN MMNMM7 OMMMM OM M MMM8M MD:N8 MMMMMM MO8M M7 O7 M7 O7

PYTHON IN AN EVOLVING ENTERPRISE SYSTEM EVALUATING INTEGRATION SOLUTIONS WITH HADOOP

DAVE HIMROD

STEVE KANNAN

ANGELICA PANDO

Building today’s most powerful, open, and customizable advertising technology platform.

Ad is served in <100 milliseconds

300x250

AUCTION REQUEST

AD RESPONSE BID: $2.50

ADVERTISER 1

BID: $3.25

ADVERTISER 2

BID: $4.10

ADVERTISER 3

APPNEXUS OPTIMIZATION

WINNING BID

Evolution of AppNexus

PEOPLE 350 430 20

AD REQUESTS 45B 39B 100M FROM

MYSQL, HADOOP/HBASE, AEROSPIKE, NETEZZA, VERTICA

5000+ SERVERS

OF DATA EVERY DAY 38+ TB

UPTIME 99.99%

Evolution of AppNexus

ENGINEERING HQ IN NYC

ENG OFFICES IN PORTLAND & SF

Data-Driven Decisioning (D3)

DATA PIPELINE

D3 PROCESSING

Bidder Bidder Bidder BIDDERS

Python at AppNexus Python enables us to scale our team and rapidly iterate and prototype technologies.

CLUSTER 1PB NODES ACROSS SEVERAL CLUSTERS 862

40B BILLION LOG RECORDS DAILY

5.6B BILLION LOG RECORDS/HOUR AT PEAK

Hadoop enables us to do aggregations for reporting and other data pipeline jobs

Hadoop at AppNexus

Data modeling today

Task Task Task Task logs logs logs logs VERTICA CACHE

HADOOP

DATA SERVICES

Σ DATA DRIVEN DECISIONING

BIG DATA: TBS/HOUR MEDIUM DATA: GBS/HOUR

To enable the next generation of data modeling, we need to leverage our Hadoop cluster

What are we trying to do

Access the data on Hadoop

Continue to use Python to model

à No consensus on the best solution

So we conducted our own research to evaluate integration options

The budget problem

We have thousands of bidders buying billions of ads per hour in real-time auctions.

We need to create a model that can manipulate how our bidders spend their budgets and purchase ads.

Data modeling today

Task Task Task Task logs logs logs logs VERTICA CACHE

HADOOP

DATA SERVICES

Σ DATA DRIVEN DECISIONING

BIG DATA: TBS/HOUR MEDIUM DATA: GBS/HOUR

DATA DRIVEN DECISIONING

Test problem: Budget aggregation

SCENARIO: Each auction creates a row in a log.

timestamp, auction_id, object_type, object_id, method, value

We need to aggregate and model to update bidders.

Method: Budget aggregation

STEP 1: De-duplicate records where

KEY: object_type, object_id, method, auction_id

STEP 2: Aggregate value where

KEY: object_type, object_id, method

HARDWARE

•  300 GB of log data

•  5 nodes running Scientific Linux 6.3 (Carbon)

•  Intel Xeon CPU @ 2.13 GHz, 4 cores

•  2 TB Disk

•  CDH4

•  45 map, 35 reduce tasks at a time

Research: Potential solutions

1. Native Java

2. Streaming ‒ no framework

3. mrjob

4. Happy / Jython / PyCascading

5.  Pig + Jython UDF 6. Pydoop

7. Disco

8.  Hadoopy / dumbo 9. Hipy

evaluating Hadoop

Effectively ORM for Hive

similar to mrjob

prohibitive installation

Research: Criteria

1. Usability

2. Performance

3. Versatility / Flexibility

Research: Native Java

Benchmark for comparison, using new Hadoop Java API

BudgetAgg.java Mapper class

BudgetAgg.java Reducer class

Research: Native Java

USABILITY: ›  Not straightforward for analysts to implement, launch, or tweak

PERFORMANCE: ›  Fastest implementation. ›  Can further enhance by overriding comparators for grouping and sorting

Research: Native Java

VERSATILITY / FLEXIBILITY:

›  Ability to customize pretty much everything

›  Custom Partitioner, Comparator, Grouping Comparator in our implementation

›  Can use complex objects as keys or values

Research: Streaming

Supplies an executable to Hadoop that reads from stdin and writes to stdout

mapper.py reducer.py

Research: Streaming

USABILITY: ›  Key/value detection has to be done by the user ›  Still, straightforward for relatively simple jobs

hadoop jar /usr/lib/hadoop-0.23.0-mr1-cdh4b1/contrib/streaming/hadoop-*streaming*.jar \ -D stream.num.map.output.key.fields=4 \ -D num.key.fields.for.partition=3 \ -D mapred.reduce.tasks=35 \ -file mapper.py \ -mapper mapper.py \ -file reducer.py \ -reducer reducer_nongroup.py \ -partitioner org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner \ -input /logs/log_budget/v002/2013/03/06/19/ -output bidder_logs/streaming_output

Research: Streaming

PERFORMANCE: ›  ~50% slower than Java

VERSATILITY / FLEXIBILITY: ›  Inputs in reducer are iterated line-by-line ›  Straightforward to get de-duplication and agg to work in a single step

Research: mrjob

Open-source Python framework that wraps Hadoop Streaming

USABILITY: ›  “Simplified Java” ›  Great docs, actively developed python budget_agg.py -r hadoop --hadoop-bin /usr/bin/hadoop \

--jobconf stream.num.map.output.key.fields=4 \

--jobconf num.key.fields.for.partition=3 \

--jobconf mapred.reduce.tasks=35 \

--partitioner org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner \

-o hdfs:///user/apando/budget_logs/mrjob_output \

hdfs:///logs/log_budget/v002/2013/03/06/19/

Research: mrjob

PERFORMANCE: ›  Not much slower than Streaming if only using RawValueProtocol

Research: mrjob

PERFORMANCE: ›  Involving objects or multiple steps slow it down a lot

VERSATILITY / FLEXIBILITY:

›  Can define Input /Internal / Output protocols

Research: Happy / Jython

HAPPY: ›  Full access to Java MapReduce API ›  Happy project is deprecated

›  Depends on Hadoop 0.17

JYTHON: ›  Doesn’t work easily out of the box

›  Relies on deprecated Jython compiler in Jython 2.2 ›  Limited to Jython implementation of Python

›  Numpy/SciPy and Pandas unavailable

Research: PyCascading

Python wrapper around Cascading framework for data processing workflow.

Uses Jython as high level language for defining workflows.

Research: PyCascading

USABILITY: ›  Relatively new project ›  Cascading API is simple and intuitive ›  Job Planner abstracts details of MapReduce

PERFORMANCE: ›  Abstraction makes performance tuning challenging ›  Does not support Combiner operation ›  Dev time was fast, runtime was slow

Research: PyCascading

VERSATILITY / FLEXIBILITY: ›  Allows Jython UDFs ›  Rich set of built-in functions: GroupBy, Join, Merge

Research: Pig

Provides a high-level language for data analysis which is compiled into a sequence of MapReduce operations.

USABILITY:

Research: Pig

USABILITY: ›  Powerful debugging and optimization tools (e.g. explain, illustrate)

›  Automatically optimizes MapReduce operations: ›  Applies Combiner operations where applicable ›  Reorders and conflates data flow for efficiency

Research: Pig

PERFORMANCE: ›  Pig compiler produces performant code ›  Complex operations might require manual optimization ›  Budget Aggregation require the implementation of a User Defined Function in Jython to eliminate unnecessary MapReduce step

Research: Pig

VERSATILITY / FLEXIBILITY: USING PIG + JYTHON UDF

›  PigLatin is expressive and can capture most use cases

›  Define custom data operations in Jython called UDFs

›  UDFs can implement custom loaders, partitioners, and other advanced features

Research: Summary

0 50 100 150 200 250 300

Java

Streaming

MRJob

PyCascading

Pig

Running Time (minutes), Lines of Code

Running Time / Lines of Code for Implementations

Lines of Code

Running Time

Research: Recommendations

•  Pig and PyCascading enable complex pipelines to be expressed simply

•  Pig is more mature and the most viable option for ad-hoc analysis

QUESTIONS

??????? ??:::::::?? ??:::::::::::? ?:::::????:::::? ?::::? ?::::? ?::::? ?::::? ?????? ?::::? ?::::? ?::::? ?::::? ?::::? ?::::? ?::::? ??::?? ???? ??? ??:?? ???

??????? ??:::::::?? ??:::::::::::? ?:::::????:::::? ?::::? ?::::? ?::::? ?::::? ?????? ?::::? ?::::? ?::::? ?::::? ?::::? ?::::? ?::::? ??::?? ???? ??? ??:?? ???

pydata@appnexus.com