Data Parallelism Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy &...

Data Parallelism

Companion slides forThe Art of Multiprocessor Programming

by Maurice Herlihy & Nir Shavit

Ever Wonder …

Art of Multiprocessor Programming 2

When did the term multicore become popular?

A multi-core processor is a single computing component with two or more independent actual central processing units, which are the units that read and execute program instructions.)

wikipedia

Let’s Ask Google Ngram!

usage of multicore in books by publication year

This part since 2000 is obvious …

multicore cablemulticore fiber

but we digress …

WordCount

alpha 8

easy to do sequentially …what about in parallel?

bravo 3charlie 9

zulu 1

MapReduce

chapter 1 chapter 2 chapter k

split text among mapping threads

Map Phase

chapter 1 chapter 2 chapter k

must count words!

a mapping thread per chapter

Map Phase

chapter 1

alpha 9juliet 2,alpha 1tango 4

…each mapper thread produces a stream …of key-value pairs …

key: wordvalue: local count

Mapper Class

abstract class Mapper<IN, K, V> extends RecursiveTask<Map<K, V>> { IN input; public void setInput(IN anInput) { input = anInput; }}

Mapper Class

input: document fragment

Mapper Class

key: individual word

Mapper Class

value: local count

Mapper Class

a task that runs in parallel with other tasks

Mapper Class

produces a map: word count

Mapper Class

initialize input: which document fragment?

WordCount Mapperclass WordCountMapper extends mapreduce.Mapper< List<String>, String, Long > { … }

WordCount Mapperclass WordCountMapper extends mapreduce.Mapper< List<String>, String, Long > { … }document fragment is list of words

map each word …

map each word …to its count in the fragment

WordCount MapperMap<String,Long> compute() { Map<String,Long> map = new HashMap<>(); for (String word : input) { map.merge(word, 1L, (x, y) -> x + y); } return map; } }

the compute() method constructs the local word count

create a map to hold the output

examine each word in the document fragment

WordCount MapperMap<String,Long> compute() { Map<String,Long> map = new HashMap<>(); for (String word : input) { map.merge(word, 1L, (x, y) -> x + y); } return map; } } increment that word’s

count in the map

when the local count is complete, return the map

Reduce Phase

alpha 4bravo 2

…zulu 1

a reducer thread merges mapper outputs

alpha 2juliet 1tango 1

alpha 1foxtrot 1papa 1tango 1

alpha 1oscar 1,bravo

alpha 3bravo 2

…zulu 1

Reduce Phase

the reducer task produces a stream …of key-value pairs …

key: word value: word count

Reducer Classabstract class Reducer<K, V, OUT> extends RecursiveTask<OUT> { K key; List<V> valueList; public void setInput( K aKey, List<V> aList) { key = aKey; valueList = aList; }}

each reducer is givena single key (word)

and a list of associated values (word count per fragment)

It produces a single summary value(the total count for that word)

WordCount

normalizing document wordcountgives a fingerprint vector

Document Fingerprint

a fingerprint is a point in a high-dimensional space

Clustering

romance novels

tango lyrics

Usenix Procedings

k-means

Find k clusters from raw data

k-means

Find k clusters from raw data

each vector closer to those in same cluster …

than in different clusters.

MapReduce

thread 1 thread 2 thread k

split points among mapping threads

k-means

Reducer picks k “centers” at random

Reduce Phase

reducer sends key-value pair to mappers

key: cluster number

value: center point

Mappers

Each mapper uses centers to assign each vector to a cluster

Mappers

Each mapper uses centers to assign each vector to a cluster

Mappers

p0 2p1 1p2 1p3 0

mapper sends key-value stream to reducerkey: point

value: cluster ID

C0 = {…}C1 = {…}C2 = {…}

Back at the Reducer

The reducer merges the streams ….and assembles clusters …

Back at the Reducer

The reducer computes new centersbased on new clusters …

0 c0’1 c1’2 c2’

Once is Not Enough

reducer sends new centers to mappers

process ends when centers become stable

0 c0’1 c1’2 c2’

To Recaptulate

We saw two problems …

wordcount & k-means …

with similar parallel solutions

Map part is parallel …

Reduce part is sequential.

abstraction

Map Function

(k1, v1) list(k2, v2)

doc, contents word, count

cluster ID, center point, cluster ID

Reduce Function

(k2, list(v2)) list(v2)

word, list of counts count

cluster ID, list of points new cluster center

Example

Distributed Grep

line of document

Reduce:

copy line to display

Example

URL Access Frequency

(URL, local count)

Reduce:

(URL, total count)

Example

Reverse web link graph

(target link, source page)

Reduce:

(target link, list of source pages)

Other Examples

histogram

matrix multiplication

PageRank

Betweenness centrality

Distributed MapReduce

Google, Hadoop, etc…

Communication by message

Fault-tolerance important

Multicore MapReduce

Phoenix, Phoenix++, Metis …

Communication by shared memory objects

Fault-tolerance unimportant

key-value layout

memory allocation

mechanism overhead

cache pressure

static vs dynamic

Part Two

Data Streams59

Streams

sequence of transformations

transformation

source

consumer

no relation toI/O streams

sometimes in parallel

Streams

transformation

source

consumer

transformations given bymathematical functions

Streams

transformation

source

consumer

transformation createsnew stream

no modifications or side-effects

correctness easier?

Functional Programming

functions map old state to new state

old state never changed

no complex side-effects

elegant, easier proofs of correctness

Oh, Really?

“Functional languages are unnatural to use; but so are knives and forks, diplomatic protocols, double-entry bookkeeping, and a host of other things modern civilization has found useful.”

Jim Morris, 1982

esthetically pleasing

only works on those who understand Haiku

Karate

esthetically pleasing

works even on those who do not understand Karate

Jim Morris’s Question

Is functional programming more like Haiku or Karate?

1981: Haiku Today: Karate

Laziness

1,2,3,…

No computation until absolutely necessary

add 1 to each element: no computation

double each element: no computation

2(xi+1) is terminal

Laziness

1,2,3,…

collect in Listmove to container is terminal

Laziness

1,2,3,…

collect in ListLaziness permits optimizations

x 2(x+1)

Laziness

1 1 2 3 5 8 12 20 32 …

Laziness permits infinite streams …

Stream<Integer> fib = new FibStream();

Unbounded Random Stream

Stream<Double> randomDoubleStream() { return Stream.generate( () -> random.nextDouble() );}

Unbounded Random Stream

Stream<Double> randomDoubleStream() { return Stream.generate( () -> random.nextDouble() );}Unbounded stream of double-precision random numbers

Random Stream

Stream that generates new elements on the fly

Random Stream

function to call when generating new element

example of Java lambda expression (anon method)

WordCountList<String> readFile(String fileName) { … return reader .lines() .map(String::toLowerCase) .flatMap(s -> pattern.splitAsStream(s)) .collect(Collectors.toList());}

puts each word from the document into a List

open the file, create a FileReader

how a stream program looks

each line creates a new stream

turn the FileReader into a stream of lines, each line a string

map creates a new stream by applying a function to each stream element

here, converts to lower case

flatMap replaces one stream element with multiple stream elements

here, splits line into words

collect puts stream elements in a container

here, in a List

terminal operation

No loops

No conditionals

No mutable objects

WordCountMap<String,Long> map = text .stream() .collect( Collectors.groupingBy( Function.identity(), Collectors.counting()));

now let’s count the words

start with list of words

turn List into a stream

put stream into a container (Map)word count

each element’s key is that element

each element’s value is the number of times it appears

No loops

No conditionals

No mutable objects

k-Means

class Point { Point(double x, double y) {…} Point plus(Point other) {…} Point scale(double x) {…} static Point barycenter( List<Point> cluster ) {…}}

k-Means

class Point { Point(double x, double y) {…} Point plus(Point other) {…} Point scale(double x) {…} static Point barycenter( List<Point> cluster ) {…}}

stream-based!

BaryCenter

Stream Barycenter

Point barycenter(List<Point> cluster){ double numPoints = cluster.size(); Optional<Point> sum = cluster .stream() .reduce(Point::plus); return sum.get() .scale(1 / numPoints); }

Stream Barycenter

cluster size

Stream Barycenter

turn List into Stream

Reduce

stream.reduce(+):

(a,b) a+b

(a,b,c) (a+b)+c

etc. …terminal operation

k-Means

Point barycenter(List<Point> cluster){ double numPoints = cluster.size(); Optional<Point> sum = cluster .stream() .reduce(Point::plus); return sum.get() .scale(1 / numPoints); } sum points in cluster

Optional because sum might be empty!

k-Means

extract sum and divide by # points

k-Means

List<Point> points = readFile("cluster.dat");centers = randomDistinctCenters(points);double convergence = 1.0; while (convergence > EPSILON) { … }

k-Means

read points from file

k-Means

pick random centers

k-Means

keep going until centers are stable

Compute New Clusters

while (convergence > EPSILON) { Map<Integer,List<Point>> clusters = points .stream() .collect( Collectors.groupingBy( p -> closestCenter(centers, p) ) );}

turn list of points into a Stream

Compute New Clusters

while (convergence > EPSILON) { Map<Integer,List<Point>> clusters = points .stream() .collect( Collectors.groupingBy( p -> closestCenter(centers, p) ) );}

put each point in a mapkey is closest centervalue is list of points with that center

k-Means Con’t

while (convergence > EPSILON) { ... Map<Integer, Point> newCenters = clusters .entrySet() .stream() .collect( Collectors.toMap( e -> e.getKey(), e -> Point.barycenter(e.getValue()) ) ); convergence = distance(centers, newCenters); centers = newCenters;}

compute new centers map: ID center

k-Means Con’t

turn map into a Stream of (ID, point) pairs

k-Means Con’t

turn stream into a mapID barycenter

k-Means Con’t

new Key is still the cluster ID

k-Means Con’t

new value is the barycenter computed earlier

k-Means Con’t

If centers have moved, start again with the new centers

Functional k-Means

many fewer lines of code

easier to read (really!)

easier to reason

easier to optimize

Parallelism?Arrays.asList("Arlington", "Berkeley", "Clarendon", "Dartmouth", "Exeter") .stream() .forEach(s -> printf("%s\n", s));

So far, Streams are sequential

make List of strings and turn them into a stream

forEach applies a method to each element (not functional)

ArlingtonBerkeleyClarendonDartmouthExeter

prints

Parallelism?Arrays.asList("Arlington", "Berkeley", "Clarendon", "Dartmouth", "Exeter") .parallelStream() .forEach(s -> printf("%s\n", s));

turn List into a parallel stream

Parallelism?Arrays.asList("Arlington", "Berkeley", "Clarendon", "Dartmouth", "Exeter") .parallelStream() .forEach(s -> printf("%s\n", s));

ArlingtonBerkeleyClarendonDartmouthExeter

prints

Dartmouth ArlingtonClarendonBerkeleyExeter

Dartmouth BerkeleyExeterArlingtonClarendon

Parallelism?Arrays.asList("Arlington", "Berkeley", "Clarendon", "Dartmouth", "Exeter") .stream() .parallel() .forEach(s -> printf("%s\n", s));

can turn stream into a parallel stream

Pitfallslist.stream().forEach( s -> list.add(0));

lambda (function) must not modify source!

Pitfallssource.parallelStream() .forEach( s -> target.add(s));

exception if target not thread-safeorder added is non-deterministic

Conclusions

Streams support

functional programming

data parallelism

compiler optimizations

Data Parallelism Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy &...

Documents

Linked Lists: Lazy and Non-Blocking Synchronization Based on the companion slides for The Art of Multiprocessor Programming (Maurice Herlihy & Nir Shavit)

Spin Locks Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit

Art of Multiprocessor Programming 1 Transactional Memory Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit

Art of Multiprocessor Programming1 Concurrent Queues Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit Modified

Art of Multiprocessor Programming1 Concurrent Hashing Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit Modified

Lecture 6-2 : Concurrent Queues and Stacks Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit

Transactional Memory Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit TexPoint fonts used in EMF. Read the TexPoint

Multiprocessor Architecture Basics Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit Ahmed Khademzadeh Azad University

Hashing and Natural Parallism Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit

Replication and Consistency - TU Kaiserslautern · The Art of Multiprocessor Programming by Maurice Herlihy and Nir Shavit Synchronization Algorithms and Concurrent Programming by

Mutual Exclusion Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit

The Relative Power of Synchronization Operations Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit

Linked Lists: Locking, Lock- Free, and Beyond … Based on the companion slides for The Art of Multiprocessor Programming (Maurice Herlihy & Nir Shavit)

Introduction Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit

Introduction to Multiprocessor Synchronization Maurice Herlihy TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAA

Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit Concurrent Skip Lists

SkipLists and Balanced Search The Art Of MultiProcessor Programming Maurice Herlihy & Nir Shavit Chapter 14 Avi Kozokin

Monitors and Blocking Synchronization Dalia Cohn Alperovich Based on “The Art of Multiprocessor Programming” by Herlihy & Shavit, chapter 8

Universality of Consensus Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit

Barrier Synchronization Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit