CONTROL Overview

CONTROL groupJoe Hellerstein, Ron Avnur, Christian

Hidber, Bruce Lo, Chris Olston, Vijayshankar Raman, Tali Roth, Kirk

Wylie, UC BerkeleyPeter Haas, IBM Almaden

Context (wild assertions)

• Value from information– The pressing problem in CS (?) (!!)

• “Point” querying and data management is a solved problem– at least for traditional data (business data,

documents)

• “Big picture” analysis still hard

Data Analysis c. 1998

• Complex: people using many tools– SQL Aggregation (Decision Support Sys, OLAP)– AI-style WYGIWIGY systems (e.g. Data Mining, IR)

• Both are Black Boxes– Users must iterate to get what they want– batch processing (big picture = big wait)

• We are failing important users!– Decision support is for decision-makers!– Black box is the world’s worst UI

Black Box Begone!

• Black boxes are bad– cannot be observed while running– cannot be controlled while running

• These tools can be very slow– exacerbates previous problems

• Thesis:– there will always be slow computer

programs, usually data-intensive– fundamental issue: looking into the box...

Crystal Balls

• Allow users to observe processing– as opposed to “lucite watches”

• Allow users to predict future• Ideally, allow users to change

future– online control of processing

• The CONTROL Project:– online delivery, estimation, and control for

data-intensive processes

Performance Regime for CONTROL

• Online performance:– Maximize 1st derivative of the “mirth index”

Time

100%

CONTROLTraditional

Examples

• Online Aggregation– Informix Dynamic Server

• Enhanced by UCB students with Control algorithms• Lots of algorithmics, many fussy end-to-end system

issues [Avnur, Hellerstein, Raman DMKD ’00]– IBM has ongoing project to do this in DB2– IBM buys Informix (4/01)

• Online Visualization– Visual enumeration & aggregation

• Interactive data cleaning & analysis– Potter’s Wheel ABC– Online “enumeration” and discrepancy detection

Example: Online Aggregation

SELECT AVG(gpa) FROM studentsGROUP BY college

Example: Online Data Visualization

• In Tioga DataSplash

Visual Transformation Shot

Scalable Spreadsheets

Decision-Support in DBMSs

• Aggregation queries– compute a set of qualifying records– partition the set into groups– compute aggregation functions on the

groups– e.g.:

Select college, AVG(grade)From ENROLLGroup By college;

Interactive Decision Support?• Precomputation

– the typical “OLAP” approach (a.k.a. Data Cubes)– doesn’t scale, no ad hoc analysis– blindingly fast when it works

• Sampling– makes real people nervous?– no ad hoc precision

• sample in advance• can’t vary stats requirements

– per-query granularity only

Online Aggregation• Think “progressive” sampling

– a la images in a web browser– good estimates quickly, improve over time

• Shift in performance goals– online mirth index

• Shift in the science– UI emphasis drives system design– leads to different data delivery, result estimation– motivates online control

Not everything can be CONTROLed• “needle in haystack” scenarios

– the nemesis of any sampling approach– e.g. highly selective queries, MIN, MAX,

MEDIAN

• not useless, though– unlike presampling, users can get some info

(e.g. max-so-far)

• we advocate a mixed approach– explore the big picture with online processing– when you drill down to the needles, or want

full precision, go batch-style– can do both in parallel

New Techniques

• Online Reordering– gives control of group delivery rates– applicable outside the RDBMS setting

• Ripple Join family of join algorithms– comes in naïve, block & hash

• Statistical estimators & confidence intervals– for single-table & multi-table queries– for AVG, SUM, COUNT, STDEV– Leave it to Peter

• Visual estimators & analysis

ST

RS

T

R

Online Reordering

• users perceive data being processed over time– prioritize processing for “interesting” tuples– interest based on user-specified preferences

• reorder dataflow so that interesting tuples go first

• encapsulate reordering as pipelined dataflow operator

• online aggregation– for SQL aggregate queries, give gradually improving

estimates – with confidence intervals

– allow users to speed up estimate refinement for groups of interest

– prioritize for processing at a per-group granularity

SELECT AVG(gpa) FROM studentsGROUP BY college

Context: an application of reordering

Framework for Online Reordering

• want no delay in processing in general, reordering can only be best-effort

• typically process/consume slower than produce– exploit throughput difference to reorder

• two aspects– mechanism for best-effort reordering– reordering policy

acddbadb...

abcdabc..

reorder consumeproduceprocess

f(t)

user interest

network xfer.

Juggle mechanism for reordering

• two threads -- prefetch from input -- spool/enrich from auxiliary side disk

• juggle data between buffer and side disk

– keep buffer full of “interesting” items

– getNext chooses best item currently on buffer

• getNext, enrich/spool decisions -- based on reordering policy

• side disk management– hash index, populated in a way that postpones random I/O

buffer

spoolprefetch enrich

getNext

side diskproduce

process/consume

Reordering policies

• quality of feedback for a prefix t1t2

…tk

QOF(UP(t1), UP(t2

), … UP(tk )), UP = user preference

– determined by application

• goodness of reordering: dQOF/dt

• implication for juggle mechanism – process gets item from buffer that increases QOF the most

– juggle tries to maintain buffer with such items

time

QOFGOAL: “good” permutation of

items t1…tn to t1…tn

QOF in Online Aggregation

• avg weighted confidence interval

• preference acts as weight on confidence interval(Recall from Central Limit Theorem that sample mean’s confidence

interval half-width is proportional to /n. Conservative (Hoeffding) confidence intervals also have a n in the denominator. So…)

QOF = UPi / ni , ni = number of tuples processed from group I

process pulls items from group with max UPi / nini

desired ratio of group i tuples on buffer = UPi2/3/ UPj

2/3

– juggle tries to maintain this by enrich/spool

Other QOF functions

• rate of processing (for a group) preference – QOF = (ni - nUPi)

2 (variance from ideal proportions)

process pulls items from group with max (nUPi - ni )

desired ratio of group i tuples in buffer = UPi

Results: Reordering in Online Aggregation

• implemented in Informix UDO server• experiments with modified TPC-D queries• questions:

– how much throughput difference is needed for reordering– can we reorder handle skewed data

• one stress test: skew, very small proc. cost

– index-only join– 5 orderpriorities, zipf distribution

A B C D E 1 1/2 1/3 1/4 1/5

consume

process

scan

juggle

indexSELECT AVG(o_totalprice), o_orderpriority

FROM order WHERE exists ( SELECT * FROM lineitem WHERE l_orderkey = o_orderkey)GROUP BY o_orderpriority

Performance results A B C D E

initial preferences 1 1 1 5 3

after T1 1 1 3.5 0.5 1

time

# t

up

les

pro

cess

ed

• 3 times faster for interesting groups• 2% completion time overhead

E

C A

con

fid

en

ce

inte

rval

time

Ripple Joins

• Good confidence intervals for joins of samples– Vs. samples of joins! – Requires “Cross-Product CLT”

• Progressively Refining join:– ever-larger rectangles in R S– we can update confidence intervals at “corners”– comes in loop, index and hash flavors

• Benefits:– sample from both relations simultaneously– “animation rate”:

• Goal for the next “corner”, determines an optimization problem based on observations so far

• Old-fashioned systems are one extreme– adaptively tune “aspect ratio” for next “corner”

• sample from higher-variance relation faster – intimate relationship between delivery and estimation

Traditional

R

S

Ripple

R

S

Haas & Hellerstein, SIGMOD 99

Aspect Ratios

• Consider an extreme example:

• In general, to get to the next corner:– Need a cost model parameterized by relation

• Different for block and hash

– “Benefit”: change in confidence interval– An online linear optimization problem

• Arguments about estimates converging quickly, stabilizing…

Fussy Implementation Details• How to implement as an iterator? Issues:

– Need cursors on all inputs (as usual)– Need to maintain aspect ratios– Need to maintain current “inner” & cursor

• I.e. the relation currently being scanned– Need to know current sampling step

• To know how far to scan current “inner”– Need to know “starter” for next step

• Determines length of scan (see pic), end of sampling step• And pass that role along at EOF

Ensuring Aspect Ratios

Ripple Join Performance

• Too lazy to fetch graphs, but…– Typical orders of magnitude benefit vs.

batch…

CONTROL Lessons

• Dream about UIs, work on systems– User needs drive systems design!

• Systems and statistics intertwine– “what unlike things must meet and mate”

• Art, Herman Melville

• Sloppy, adaptive systems a promising direction

Questions

• Where else do these lessons apply?– Outside of data analysis, manipulation

• Systems people think a lot about interfaces (APIs)…– Encapsulation, narrow interfaces …– In the CONTROL regime, how do you design these

APIs and build systems?

• Ubiquitous computing:– Is it about portable computing and point

access/delivery?– Or sensors/actuators, dataflow, big-picture queries?

More?

• CONTROL: http://control.cs.berkeley.edu– Overview: IEEE Computer, 8/99

• Telegraph: http://db.cs.berkeley.edu/telegraph

Backup slides

• The following slides may be used to answer questions...

Sampling

• Much is known here– Olken’s thesis– DB Sampling literature– more recent work by Peter Haas

• Progressive random sampling– can use a randomized access method (watch

dups!)– can maintain file in random order– can verify statistically that values are

independent of order as stored

Estimators & Confidence Intervals

• Conservative Confidence Intervals– Extensions of Hoeffding’s inequality– Appropriate early on, give wide intervals

• Large-Sample Confidence Intervals– Use Central Limit Theorem– Appropriate after “a while” (~dozens of tuples)– linear memory consumption– tight bounds

• Deterministic Intervals– only useful in “the endgame”