Program analysis for fun and profit or How I spent my summer
vacation
Ariel RabkinPrinceton
[email protected] and Degradation in
JetStream: Streaming Analytics in the Wide AreaWork done with
Matvey Arye, Siddhartha Sen, Vivek S. Pai, and Michael J.
Freedman
1Todays Analytics Architectures2
Backhaul is inefficient and inflexible
MillWheel (Google)Storm
Stream-processing systems have become a fundamental building
block of modern large-scale software systems. This talk describes
how to apply this paradigm to the wide area, where bandwidth is
unreliable and scarce.2Tomorrows Architecture: JetStream3
Backhaul is inefficient and inflexibleGoal: optimize use of WAN
links by exposing them to streaming
system.JetStreamStream-processing systems have become a fundamental
building block of modern large-scale software systems. This talk
describes how to apply this paradigm to the wide area, where
bandwidth is unreliable and scarce.3Backhaul is Intrinsically
Inefficient4Time [two days]
Bandwidth
AvailableBuyers remorse: wasted bandwidthAnalysts remorse:
system overload ormissing dataNeeded for backhaulStream Processing
Basics5Filtering (count > 100)Sampling (drop 90% of data)Image
Compression
Quantiles (95th percentile)Query stored dataSite ASome Operators
in JetStream:Stream OperatorsInput DataStream OperatorsInput
DataStream OperatorsStream OperatorsSite BStream OperatorSite CThe
JetStream SystemWhat: Streaming with aggregation and degradation as
first-class primitivesWhere: Storage and processing at edgeWhy:
Maximize goodput using aggregation and degradationHow: Data cubes
and feedback control6An Example Query7How popular is every
URL?RequestsRequestsCDN RequestsRequestsRequestsCDN
RequestsMechanism 1: Storage with Aggregation 8RequestsRequestsCDN
RequestsRequestsRequestsCDN RequestsEvery minute, compute request
counts by URLLocalAggregation and StorageLocalAggregation and
StorageMechanism 2: Adaptive Degradation9RequestsRequestsCDN
RequestsRequestsRequestsCDN RequestsEvery minute, compute request
counts by URLLocalAggregation and StorageLocalAggregation and
StorageAdjustable FilteringAdjustable FilteringRequirements for
Storage Abstraction10Update-able (locally and incrementally)
DataDataMerged Representation+=DataDataMerge-able (without
accuracy penalty)
Data size is reducible (with predictable accuracy cost)
Stored Data+=DataThe Data Cube ModelAggregation used
for:UpdatesRoll-upsMerging cubesSummarizing cubes11Counts by
URL12:0012:0112:02www.mysite.com/a350www.mysite.com/b020www.yoursite.com54www.her-site.com812Cube:
A multidimensional array, indexed by a set of dimensions, whose
cells hold aggregates.Cubes have aggregation function: Agg( , )We
adapted this abstraction from OLAP databases. Unlike OLAP, we are
just keeping a primary-key index, so inserts are cheap and
efficient.11Cubes can be Rolled Up12Counts by
URL12:0012:0112:02www.mysite.com/a350www.mysite.com/b020www.yoursite.com54www.her-site.com812Cube:
A multidimensional array, indexed by a set of dimensions, whose
cells hold aggregates.Counts by
URL*www.mysite.com/a8www.mysite.com/b2www.yoursite.com9www.her-site.com20Counts
by URL12:0012:0112:02*162312Cubes Unify Storage and
Aggregation13Stored DataUpdateUpdateUpdateUpdate sent
downstreamStanding QueryOne-off query
Feedback controlDegradation: The Big Picture14Local
DataDataflowOperatorsSummarized or ApproximatedDataLevel of
degradation auto-tuned to match bandwidth.Challenge: Supporting
mergeability and flexible
policiesNetworkDataflowOperatorsMergeability Imposes
ConstraintsInsight: Degradation may be discontinuous01 - 1011 -
20Every 1021 - 3001 - 30Every 30??01 - 0506 - 1011 - 1516 - 2021 -
25Every 526 - 3001 - 0607 - 1213 - 1819 - 24Every 625 -
3015??????02 - 0607 - 1112 - 1617 - 2122 - 26Every 527 - 31This is
an consequence of the problem space, not our design. If you have
fidelity changes over time, you have to deal with this sort of
thing. Part of the challenge for our design is to compensate for
these difficulties and shield the user.15There Are Many Ways to
Degrade Data16Can coarsen a dimensionCan drop low-rank values
Coarsening Does Not Always Help17
Degradations Have Trade-offs18NameFixed BW SavingsFixed Accuracy
costParameterDim. CoarseningUsually noYesDimension ScaleDrop values
(locally)YesNoCut-offDrop values (globally)No, multi-round
protocolYesCut-offAudiovisual downsamplingYesYesSample
rateHistogram CoarseningYes
Yes
Number of BucketsThe cube data model gives a nice intuition
here: transforming aggregates can have predictable bandwidth and
accuracy results; transforming dimensions wont.18
A Simple Idea that Does Not WorkWe have sensors that report
congestion.Have operators read sensor and adjust
themselves?19Coarsening OperatorIncoming
dataNetworkSampledDataSending 4x too muchIncrease aggregation
period up to 10 sec. If insufficient, use sampling
Challenge: Composite PoliciesChaos if two operators are
simultaneously responding to the same sensor20Coarsening
OperatorIncoming dataNetworkSampling OperatorSending 4x too
muchInterfacing with Operators21Shrinking data by 50%Possible
levels: [0%, 50%, 75%, 95%, ]Go to level 75%
Coarsening OperatorIncoming dataNetworkSampling
OperatorControllerSending 4x too much21Experimental Setup2280 nodes
on VICCI testbed at three sites (Seattle, Atlanta, and
Germany)Policy: Drop data if insufficient BW
Princeton
DO NOT Say son of planelab -- ke22
Without Degradation23Drop BW
Median Latency95th percentile latencyMaximum latency
Degradation Keeps Latency Bounded24Bandwidth Shaping
Median Latency95th percentile latency
Showing maximum latencies25Median Latency95th percentile
latencyMaximum LatencyProgramming Ease26ScenarioLines of codeSlow
requests5Requests by URL5Bandwidth by node15Bad referrers16Latency
and size quantiles25Success by domain30Top 10 domains by
period40Big Requests97Coded up eight queries based on operational
experience with coral. Some were simple, like the requests by URL.
Some were hard, like pulling back full logs on the requests above a
quantile.26Conclusions and Future WorkUseful to embed aggregation
and degradation abstractions in streaming systems.Aggregation can
be unified with storage.System must accommodate degradation
semantics.Open questions: How to guide users to the right
degradation policy? How to embed abstractions in higher-level
language?
27Backup slides start here2829
Data Cube
Raw byte stringse.g. key-value storeDatabase tablesMerge +
AggregateHigh-level APIPredictable
performanceArbitraryJoinsXXXXXXClarify Api level: A short SQL
statement can compile to a longish string of MR programs. The
reverse is not true2930
Data Cube
Raw byte stringse.g. key-value storeDatabase tablesMerge +
AggregateHigh-level APIPredictable
performanceArbitraryJoinsXXXXXXClarify Api level: A short SQL
statement can compile to a longish string of MR programs. The
reverse is not true3031
Data Cube
Raw byte stringse.g. key-value storeDatabase tablesMerge +
AggregateHigh-level APIPredictable
performanceArbitraryJoinsXXXXXXClarify Api level: A short SQL
statement can compile to a longish string of MR programs. The
reverse is not true31Comparing Multiple Degradations32
Introducing a ControllerIntroduce a controller for each network
connection that determines which degradations to applyDegradation
policies for each controllerPolicy no longer constrained by
operator topology33Coarsening OperatorIncoming dataNetworkSampling
OperatorControllerDrop 75% of data!Sending 4x too much? ? ? ? ? ?
What Happens During Overload?34Time [one day]Bandwidth
Needed for backhaulAvailableTimeLatency
Queue size grows without bound!Related WorkBorealis: Shows how
to do reliable streamingSparkStreaming, MillWheel, Storm: Show how
to manage recovery and scalability within datacenter. Do not
address unreliable or constrained links.BlinkDB: Degrades data for
performance; assumes ability to precompute samples.35System
Architecture36compute resources (several sites)Control planeData
plane
Query graphCoordinatorDaemonOptimized queryJetStream APIPlanner
Libraryworker nodestream sourceAn Example Query 37File Read
OperatorParse Log FileLocal StorageFile Read OperatorParse Log
FileLocalStorageQueryEvery 10 sCentral StorageQueryEvery 10 sSite
ASite BSite CJetStream is Straightforward to Program38ScenarioLines
of codeBW savingsBig Requests9722xSlow requests524xRequests by
URL5351xSuccess by domain30445xLatency and size quantiles25715xTop
10 domains by period402300xBad referrers1618600xBandwidth by
node1549800xThe JetStream Vision39Time [two days]
Bandwidth
AvailableNeeded for backhaulJetStream lets programs adapt to
shortages and backfill later.Need new programming abstractions.Used
by JetStreamContributions40
Storage abstraction that allows flexible aggregation inside the
streaming systemDynamic adjustment of data quality to match
bandwidthhttp://qeisecurity.com/wp-content/uploads/2009/12/Camera_WallMt_Panasonic.jpg40Bandwidth
CostsAmazon EC2 bandwidth: $0.05 per GB
Wireless broadband: $2 per GB
Cell phone broadband (ATT/Verizon): $6 per GB(Other providers
are similar)
Satellite Bandwidth $200 - $460 per GBMay drop to
~$2041http://aws.amazon.com/s3/pricing/http://www.rogers.com/web/Rogers.portal?_nfpb=true&_pageLabel=Homehttp://groundcontrol.com/FleetBroadband_Service_Plans.htm
http://gigaom.com/2012/09/19/why-your-in-flight-wi-fi-is-slow-and-expensive-its-all-about-the-pipe/41
