1

Map-Reduce and Datalog Implementation

Distributed File SystemsMap-Reduce

Join Implementations

2

Humongous Data Problems

We are seeing new applications for very large data operations. Web operations, e.g., PageRank. Social network data. Collaborative filtering of commercial

data. Result: new infrastructure.

Distributed file systems. Map-reduce/Hadoop/Hive/Pig,…

3

Role of Datalog

Many operations are remarkably simple.

Example: suggest new friends in a social network by looking for violations of transitivity:

suggest(X,Y) :- friend(X,Z) & friend(Z,Y)

& NOT friend(X,Y)

4

Scale of the Problem

FaceBook has 250 million subscribers, each with about 300 friends.

Self join of friends with itself could have 22.5 trillion tuples. But because of “locality,” the size

would be less by a factor of perhaps 10–100.

5

Distributed File Systems

To deal with computations of this size, companies use large collections of commodity servers. Both for storage and for computing.

• Often the same servers.

Files are stored in chunks, typically 64MB.

Chunks are replicated, typically 3 times.

6

Cluster Computing

Racks of compute nodes, interconnected, e.g., by gigabit Ethernet.

New element: computations involve so much work, that a node failure is common.

Map-reduce (Hadoop) is a framework for dealing effectively with node failure, as well as simplifying certain calculations.

7

Map-Reduce

You write two functions, Map and Reduce.

Several Map tasks and Reduce tasks implement these functions.

Each Map task gets one or more chunks of input data from a distributed file system.

8

Map-Reduce – (2)

Map tasks turn input into a list of key-value pairs. But “keys” are not unique.

A master controller assigns each key, and all output from Map tasks with that key, to one of the Reduce tasks.

Reduce tasks apply some operation to the values associated with one key.

9

Graph of Map and Reduce Tasks

Map

Map

Map

.

.

.

Input

Reduce

Reduce

Reduce

.

.

Output

10

Example: Join by Map-Reduce

Answer(X,Y) :- r(X,Z) & s(Z,Y) Map takes each tuple from r, say

r(x,z), and produces the key-value pair [z, (r,x)].

From tuple s(z,y), Map produces key-value pair [z, (s,y)].

Thus, all tuples r(x,z) and s(z,y) go to the same Reduce task.

11

Join by Map-Reduce – (2)

The Reduce tasks perform a standard join on all the r- and s-tuples they receive.

Output is the union of the results of all Reduce tasks.

12

Coping With Failures

Because Every Map and Reduce task receives

all its input at the beginning, Every Map and Reduce task finishes

by handing its complete output to the master controller.

Any task at a failed node can be restarted without affecting any other task.

13

Multiway Join Via Map-Reduce

From Afrati/Ullman in EDBT-2010. Useful for Datalog evaluation because:

Bodies often have more than two subgoals. Seminaive evaluation can involve a complex

expression with many relations and their increments (next talk).

Normal procedure is to take a cascade of two-way joins.

14

Multiway Join – (2)

Sometimes, it is more efficient to replicate tuples to several Reduce tasks.

1. When relations have large fan-out. Examples: “friends” or links on the Web.

2. Star joins. Join of a large fact table with smaller

dimension tables.

Intuition: wins when intermediate joins would be large.

15

Multiway Join – (3)

Assume k Reduce tasks. Certain variables get shares of a hash

function that maps to k buckets. Product of the shares = k.

If variable X has share x, then each X-value is hashed to one of x hash keys.

Hash key of a Reduce task = vector of hash values for each variable with a share.

16

Example: Multiway Join

Answer(W,X,Y,Z) :- r(W,X) & s(X,Y) & t(Y,Z)

Only X and Y get a share, say xy = k. Theorem: never give a share to a

dominated variable.• = variable that appears only where some other

variable also appears.

Tuple s(a,b) goes only to Reduce task [h(a), h’(b)].

17

Example: Multiway Join – (2)

Answer(W,X,Y,Z) :- r(W,X) & s(X,Y) & t(Y,Z)

However, tuple r(a,b) must go to all Reduce tasks [h(b), n] where n is any of y different hash values.

Similarly, tuple t(a,b) must go to all Reduce tasks [m, h’(a)], where m is any of x different hash values.

18

Example: Multiway Join – (3)

Answer(W,X,Y,Z) :- r(W,X) & s(X,Y) & t(Y,Z)

To minimize the number of tuples transmitted, pick:

x = k|r|/|t| y = k|t|/|r| Intuition: costs distributing tuples of r

and t are the same.

19

Summary of Afrati/Ullman EDBT-2010

It is possible to find the optimum shares for variables for any join.

Usually, the process is a straightforward Lagrangean analysis.

In pathological cases, an exponential search appears necessary.

Constraining to positive integers and adjusting the product add complexity.