View
384
Download
0
Category
Preview:
Citation preview
Apache ApexIntro to ApexIngestion and Dimensions Compute for a customer use-case
Devendra Tagaredevendrat@datatorrent.com@devtagare9h July 2016
What is Apex• Platform and runtime engine that enables development of scalable
and fault-tolerant distributed applications • Hadoop native• Process streaming or batch big data• High throughput and low latency• Library of commonly needed business logic• Write any custom business logic in your application
Applications on Apex• Distributed processing
• Application logic broken into components called operators that run in a distributed fashion across your cluster
• Scalable• Operators can be scaled up or down at runtime according to the load and SLA
• Fault tolerant• Automatically recover from node outages without having to reprocess from beginning• State is preserved• Long running applications
• Operators• Use library to build applications quickly• Write your own in Java using the API
• Operational insight – DataTorrent RTS• See how each operator is performing and even record data
Apex Stack Overview
Apex Operator Library - Malhar
Native Hadoop Integration
• YARN is the resource manager
• HDFS used for storing any persistent state
Application Development Model
A Stream is a sequence of data tuplesA typical Operator takes one or more input streams, performs computations & emits one or more output streams
• Each Operator is YOUR custom business logic in java, or built-in operator from our open source library• Operator has many instances that run in parallel and each instance is single-threaded
Directed Acyclic Graph (DAG) is made up of operators and streams
Directed Acyclic Graph (DAG)
Filtered
Stream
Output StreamTuple Tuple
Filtered Stream
Enriched Stream
Enriched
Stream
er
Operator
er
Operator
er
Operator
er
Operator
er
Operator
er
Operator
Advanced Windowing Support
Application window Sliding window and tumbling window Checkpoint window No artificial latency
Application in Java
Partitioning and unificationNxM PartitionsUnifier
0 1 2 3
Logical DAG
0 1 2
1
1 Unifier
1
20
Logical Diagram
Physical Diagram with operator 1 with 3 partitions
0
Unifier
1a
1b
1c
2a
2b
Unifier 3
Physical DAG with (1a, 1b, 1c) and (2a, 2b): No bottleneck
Unifier
Unifier0
1a
1b
1c
2a
2b
Unifier 3
Physical DAG with (1a, 1b, 1c) and (2a, 2b): Bottleneck on intermediate Unifier
Advanced Partitioning
0
1a
1b
2 3 4Unifier
Physical DAG
0 4
3a2a1a
1b 2b 3b
Unifier
Physical DAG with Parallel Partition
Parallel Partition
Container
uopr
uopr1
uopr2
uopr3
uopr4
uopr1
uopr2
uopr3
uopr4
dopr
dopr
doprunifier
unifier
unifier
unifier
Container
Container
NIC
NIC
NIC
NIC
NIC
Container
NIC
Logical Plan
Execution Plan, for N = 4; M = 1
Execution Plan, for N = 4; M = 1, K = 2 with cascading unifiers
Cascading Unifiers
0 1 2 3 4
Logical DAG
Dynamic Partitioning
• Partitioning change while application is runningᵒ Change number of partitions at runtime based on statsᵒ Determine initial number of partitions dynamically
• Kafka operators scale according to number of kafka partitionsᵒ Supports re-distribution of state when number of partitions changeᵒ API for custom scaler or partitioner
2b
2c
3
2a
2d
1b
1a1a 2a
1b 2b
3
1a 2b
1b 2c 3b
2a
2d
3a
Unifiers not shown
• Ingest from Kafka and S3• Parse, Filter and Enrich• Dimensional compute for key performance indicators• Reporting of critical metrics around campaign monetization• Aggregate counters & reporting on top N metrics• Low latency querying using Kafka in pub-sub model
Use Case ...
Screenshots - Demo UI
Proprietary and Confidential
Scale
• 6 geographically distributed data centers• Combination of co-located & AWS based DC’s.• > 5 PB under the data management• 22 TB / day data generated from auction & client logs• heterogeneous data log formats• North of 15 Bn impressions / day.• Average data inflow of 200K events/s
• Ad server log events consumed as Avro encoded, Snappy compressed files
from S3. New files uploaded every 10-20 minutes.
• Data may arrive in S3 out of order (time stamps).
• Event size is about 2KB uncompressed, only subset of fields retrieved for
aggregation.
• Aggregates kept in memory (checkpointed) with expiration policy, query
processing against in memory data.
• Front-end integration through Kafka based query protocol for realtime
dashboard components.
Initial Requirements
Proprietary and Confidential
Apache Apex
AdServer
REST proxy
REST proxy
Real-time architecture- Powered By Apex
Kafka Cluster
S3Reader S3Reader
Filter Operator Filter Operator
Dimensions Aggregator
Dimensions Aggregator
Dimensions Store
Query Query Result
Kafka Cluster
Auction Logs
Middleware
Auction Logs
Filtered Events Filtered Events
Aggregates
Query from MW
Query Query Results
S3 S3 Client logsAuction Logs
Architecture 1.0 - Batch Reads + Streaming Aggregations
• Unstable S3 client libraries– Unpredictable hangs and Corrupted data– On Hang, Master kills the container and restart reading of file from different container– Corrupt files caused containers to kill – application configurable retry mechanism and skip bad
files– Limited read consumption throughput – 1 reader/file
• Out of Order data– Some timestamp in future and past
• Spike in load when new files are added followed by period of inactivity• Memory Requirement for Store
– Cardinality Estimation for incoming data
Challenges
Proprietary and Confidential
Apache Apex
REST proxy
Real-time architecture- Powered By Apex
Client logs
Kafka Input
(Auction logs)
ETL operator
Filter Operator Filter Operator
Dimensions Aggregator
Dimensions Aggregator
Dimensions Store/HDHT
Query Query Result
Kafka Cluster
Auction LogsKafka Cluster
Middleware
AdServerREST proxy
Kafka Cluster
Auction Logs
Client logs
Kafka Messages
Decompress & Flatten
Filtered Events Filtered Events
Aggregates
Query from MW
Query Query Results
S3
S3Reader
Kafka Input
(Auction logs)Auction Logs
Architecture 2.0 - Batch + Streaming
Challenges• Complex Logical DAG• Kafka Operator Issues
– Dynamic Partitioning– Memory Configuration– Offset snapshotting to ensure exactly once semantics
• Resource Allocation– More memory requirement for Store (Large number of Unifiers)
• Harder Debugging (More number of components)– GB(s) of container logs– Difficult to locate the sequence of failure
• More of data transferred over wire within cluster• Limit Kafka read rate
Proprietary and Confidential
User Browser
AdServer
REST proxy
REST proxy
Real-time architecture- Powered By Apex
Kafka Cluster
Client logs
Kafka Input
(Auction logs)
Kafka Input
(Client logs)
CDN(Caching of
logs)
ETL operator ETL operator
Filter Operator Filter Operator
Dimensions Aggregator
Dimensions Aggregator
Dimensions Store
Query Query Result
Kafka Cluster
Auction Logs
Client logs
Middleware
Auction Logs
Client logs
Kafka Messages Kafka Messages
Decompress & Flatten
Decompress & Flatten
Filtered Events Filtered Events
Aggregates
Query from MW
Query Query Results
Architecture 3.0 - Streaming
Operational Architecture
Proprietary and Confidential
Application Configuration
• 64 Kafka Input operators reading from 6 geographically distributed DC’s
• Under 40 seconds end to end latency - from ad-serving to visualization
• 32 instances of in-memory distribute store
• 64 aggregators
• 1.2 TB memory footprint @ peak load
• In-memory store was later replaced by HDHT for fault tolerance
Proprietary and Confidential
Learning’s
• DAG – sizing, locality & partitioning (Benchmark)• Memory sizing for the store or other memory heavy operators.• Cardinality estimation for incoming data is critical.• Upstream operators tend to require more memory than down-stream
operators for high velocity reads.• Back pressure from down-stream failures due to skew in velocity of events
& upstream failures .. Buffer Server sizing is critical.• For end to end exactly once its necessary to understand the external
systems semantics & delivery guarantees.• Think fault tolerance & recovery before starting implementation.
Proprietary and Confidential
Before And After
5 Hours + 20 Minute
• No real-time processing system in place. • Publishers and buyers could only rely on a
batch processing system for gathering relevant data
• Outdated data, not relevant to current time
• Current data being pushed to a waiting queue
• Cumbersome batch-processing lifecycle
• No visualization for reports• No glimpse into everyday
happenings, translating to lost decisions or untimely decision making scenarios
Before Scenario After Scenario
• Phase 1• With DataTorrent RTS (built on
Apache Apex), Dev team put together the first real time analytics platform
• This enabled Reporting of critical metrics around campaign monetization
• Reuse of batch ingestion mechanism for the impression data, shared with other pipelines (S3)
~ 30 seconds
No Real-time Batch + Real-time
• Phase 2• Reduce end-to-end latency
through real-time ingestion of impression data from Kafka
• Results available much sooner to the user
• Balances load (no more batch ingestion spikes), reduces resource consumption
• Handles ever growing traffic with more efficient resource utilization.
Real-time Streaming
Proprietary and Confidential
Operators used
S3 reader (File Input Operator)• Recursively reading the contents of a S3 bucket based on a partitioning pattern• Inclusion & exclusion support• Fault tolerance (replay and idempotent)• Throughput of over 12K reads/second for event size of 1.2 KB each
Kafka Input Operator• Ability to consume from multiple Kafka clusters • Offset management support• Fault tolerant reads• Support for idempotent & exactly once semantics• Controlled reads for managing back-pressure
POJO Enrichment Operator• takes a POJO as input and does a look-up in a store for given key• supports caching• stores are pluggable• App builder ready
Proprietary and Confidential
Operators used (cont …)Parser • Specify JSON schema• Emits a POJO based on the output schema• No user code required
Dimension Store• Distributed in-memory store• Supports re-aggregation of events• Partitioning of aggregates per view• Low latency query support with a pub/sub model using Kafka
HDHT• HDFS backed embedded key-value store• Fault tolerant, random read & write• Durability in-case of cold restarts
Dimensional Model - Key Concepts
Metrics : pieces of information we want to collect statistics about.
Dimensions : variables which can impact our measures.
Combinations : set of dimensions for which one or metric would be aggregated.They are sub-sets of dimensions.
Aggregations : the aggregate function eg.. SUM, TOPN, Standard deviation.
Example :Dimensions - campaignId, advertiserId, timeMetrics - Cost, revenue, clicks, impressionsAggregate functions -SUM,AM etc..
Combinations :1. campaignId x time - cost,revenue2. advertiser - revenue, impressions3. campaignId x advertiser x time - revenue, clicks, impressions
How to aggregate on the combinations ?
Dimensional Model
Dimensions Schema
{"keys":[{"name":"campaignId","type":"integer"}, {"name":"adId","type":"integer"}, {"name":"creativeId","type":"integer"}, {"name":"publisherId","type":"integer"}, {"name":"adOrderId","type":"integer"}], "timeBuckets":["1h","1d"], "values": [{"name":"impressions","type":"integer","aggregators":["SUM"]}, {"name":"clicks","type":"integer","aggregators":["SUM"]}, {"name":"revenue","type":"integer"}], "dimensions": [{"combination":["campaignId","adId"]}, {"combination":["creativeId","campaignId"]}, {"combination":["campaignId"]}, {"combination":["publisherId","adOrderId","campaignId"],"additionalValues":["revenue:SUM"]}]}
Proprietary and Confidential
More Use-cases
• Real-time MonitoringAlerts on deal tracking & monetizationCampaign & deal health• Real-time LearningUsing the lost bid insights for price recommendations.• Allocation EngineFeedback to ad serving for guaranteed delivery & line item pacing
Data Processing Pipeline ExampleApp Builder
Monitoring ConsoleLogical View
Monitoring ConsolePhysical View
Real-Time DashboardsReal Time Visualization
Q&A
Resources• http://apex.apache.org/• Learn more: http://apex.apache.org/docs.html • Subscribe - http://apex.apache.org/community.html• Download - http://apex.apache.org/downloads.html• Follow @ApacheApex - https://twitter.com/apacheapex• Meetups – http://www.meetup.com/pro/apacheapex/• More examples: https://github.com/DataTorrent/examples• Slideshare:
http://www.slideshare.net/ApacheApex/presentations• https://www.youtube.com/results?search_query=apache+apex• Free Enterprise License for Startups -
https://www.datatorrent.com/product/startup-accelerator/
Recommended