Parallel Sorting

Sathish Vadhiyar

Sorting

Sorting n keys over p processors Sort and move the keys to the

appropriate processor so that every key on processor k is larger than every key on processor k-1

The number of keys on any processor should not be larger than (n/p + thres)

Communication-intensive due to large migration of data between processors

Bitonic Sort

One of the traditional algorithms for parallel sorting

Follows a divide-and-conquer algorithm

Also has nice properties – only a pair of processors communicate at each stage

Can be mapped efficiently to hypercube and mesh networks

Bitonic Sequence Rearranges a bitonic sequence into a sorted

sequence Bitonic sequence – sequence of elements

(a0,a1,a2,…,an-1) such that

Or there exists a cyclic shift of indices satisfying the above

E.g.: (1,2,4,7,6,0) or (8,9,2,1,0,4)

a0 ai an-1

Using bitonic sequence for sorting

Let s = (a0,a1,…,an-1) be a bitonic sequence such that a0<=a1<=…<=an/2-1 and an/2>=an/2+1>=…>=an-1

ConsiderS1 = (min(a0,an/2),min(a1,an/2+1),….,min(an/2-1,an-1)) and

S2 = (max(a0,an/2),max(a1,an/2+1),….,max(an/2-1,an-1))

Both are bitonic sequencesEvery element of s1 is smaller than s2

Using bitonic sequence for sorting Thus, initial problem of rearranging a bitonic

sequence of size n is reduced to problem of rearranging two smaller bitonic sequences and concatenating the results

This operation of splitting is bitonic split This is done recursively until the size is 1 at

which point the sequence is sorted; number of splits is logn

This procedure of sorting a bitonic sequence using bitonic splits is called bitonic merge

Bitonic Merging Network

3

5

8

9

10

12

14

20

95

90

60

40

35

23

18

0

++

++

+++

++

++

+++

++

3

5

8

9

10

12

14

0

95

90

60

40

35

23

18

20

++

++

++

++

++

++

++

++

3

5

8

0

10

12

14

9

35

23

18

20

95

90

60

40

+

++

++

++

++

++

++

++

+

3

0

8

5

10

9

14

12

18

20

35

23

60

40

95

90

+++++++++++++++

+

0

3

5

8

9

10

12

14

18

20

23

35

40

60

90

95

Takes a bitonic sequence and outputs sorted order; contains logn columnsA bitonic merging network with n inputs denoted as BM[n]

+

Sorting unordered n elements

By repeatedly merging bitonic sequences of increasing length

+

+

+

+

-

-

-

-

BM[4]

+

+

+

+

-

-

-

BM[2]

BM[2]

BM[2]

BM[2]

BM[2]

BM[2]

BM[2]

BM[2]

BM[4]

BM[4]

BM[4]

BM[8]

BM[8]

BM[16]

•An unsorted sequence can be viewed as a concactenation of bitonic sequences of size two•Each stage merges adjancent bitonic sequences into increasing and decreasing order•Forming a larger bitonic sequence

Bitonic Sort

Eventually obtain a bitonic sequence of size n which can be merged into a sorted sequence

Figure 9.8 in your book Total number of stages, d(n) =

d(n/2)+logn = O(log2n) Total time complexity = O(nlog2n)

Parallel Bitonic SortMapping to a Hypercube Imagine N processes (one element per

process). Each process id can be mapped to the

corresponding node number of the hypercube. Communications between processes for

compare-exchange operations will always be neighborhood communications

In the ith step of the final stage, processes communicate along the (d-(i-1))th dimension

Figure 9.9 in the book

Parallel Bitonic SortMapping to a Mesh

Connectivity of a mesh is lower than that of hypercube

One mapping is row-major shuffled mapping

Processes that do frequent compare-exchanges are located closeby

0 1 4 5

2 3 6 7

8 9 12 13

10 11 14 15

Mesh..

For example, processes that perform compare-exchange during every stage of bitonic sort are neighbors

0 1 4 5

2 3 6 7

8 9 12 13

10 11 14 15

Block of Elements per ProcessGeneral

3

5

8

9

10

12

14

20

95

90

60

40

35

23

18

0

++

++

+++

++

++

+++

++

3

5

8

9

10

12

14

0

95

90

60

40

35

23

18

20

++

++

++

++

++

++

++

++

3

5

8

0

10

12

14

9

35

23

18

20

95

90

60

40

+

++

++

++

++

++

++

++

+

3

0

8

5

10

9

14

12

18

20

35

23

60

40

95

90

+++++++++++++++

+

0

3

5

8

9

10

12

14

18

20

23

35

40

60

90

95

General..

For a given stage, a process communicates with only one other process

Communications are for only logP steps

In a given step i, the communicating process is determined by the ith bit

Drawbacks

Bitonic sort moves data between pairs of processes

Moves data O(logP) times Bottleneck for large P

Sample Sort

Sample Sort

A sample of data of size s is collected from each processor; then samples are combined on a single processor

The processor produces p-1 splitters from the sp-sized sample; broadcasts the splitters to others

Using the splitters, processors send each key to the correct final destination

Parallel Sorting by Regular Sampling (PSRS)

1. Each processor sorts its local data2. Each processor selects a sample vector of

size p-1; kth element is (n/p * (k+1)/p)3. Samples are sent and merge-sorted on

processor 04. Processor 0 defines a vector of p-1

splitters starting from p/2 element; i.e., kth element is p(k+1/2); broadcasts to the other processors

PSRS

5. Each processor sends local data to correct destination processors based on splitters; all-to-all exchange

6. Each processor merges the data chunk it receives

Step 5

Each processor finds where each of the p-1 pivots divides its list, using a binary search

i.e., finds the index of the largest element number larger than the jth pivot

At this point, each processor has p sorted sublists with the property that each element in sublist i is greater than each element in sublist i-1 in any processor

Step 6

Each processor i performs a p-way merge-sort to merge the ith sublists of p processors

Example

Example Continued

Analysis The first phase of local sorting takes

O((n/p)log(n/p)) 2nd phase:

Sorting p(p-1) elements in processor 0 – O(p2logp2) Each processor performs p-1 binary searches of n/p

elements – plog(n/p)

3rd phase: Each processor merges (p-1) sublists Size of data merged by any processor is no more

than 2n/p (proof) Complexity of this merge sort 2(n/p)logp

Summing up: O((n/p)logn)

Analysis

1st phase – no communication 2nd phase – p(p-1) data collected; p-1

data broadcast 3rd phase: Each processor sends (p-1)

sublists to other p-1 processors; processors work on the sublists independently

Analysis

Not scalable for large number of processors

Merging of p(p-1) elements done on one processor; 16384 processors require 16 GB memory

Sorting by Random Sampling

An interesting alternative; random sample is flexible in size and collected randomly from each processor’s local data

Advantage A random sampling can be retrieved

before local sorting; overlap between sorting and splitter calculation

Sources/References

On the versatility of parallel sorting by regular sampling. Li et al. Parallel Computing. 1993.

Parallel Sorting by regular sampling. Shi and Schaeffer. JPDC 1992.

Highly scalable parallel sorting. Solomonic and Kale. IPDPS 2010.

END

Bitonic Sort - Compare-splits When dealing with a block of elements per process,

instead of compare-exchange, use compare-split i.e, each process sorts its local elementsl then each

process in a pair sends all its elements to the receiving process

Both processes do the rearrangement with all the elements

The process then sends only the necessary elements in the rearranged order to the other process

Reduces data communication latencies

Block of elements and Compare Splits Think of blocks as elements Problem of sorting p blocks is identical to

performing bitonic sort on the p blocks using compare-split operations

log2P steps At the end, all n elements are sorted since

compare-splits preserve the initial order in each block

n/p elements assigned to each process are sorted initially using a fast sequential algorithm

Block of Elements per ProcessHypercube and Mesh Similar to one element per process case, but

now we have p blocks of size n/p, and compare exchanges are replaced by compare-splits

Each compare-split takes O(n/p) computation and O(n/p) communication time

For hypercube, the complexity is: O(n/p log(n/p)) for sorting O(n/p log2p) for computation O(n/p log2p) for communication

Histogram Sort

Another splitter-based method Histogram also determines a set of p-1

splitters It achieves this task by taking an iterative

approach rather than one big sample A processor broadcasts k (> p-1) initial

splitter guesses called a probe The initial guesses are spaced evenly

over data range

Histogram SortSteps

1. Each processor sorts local data2. Creates a histogram based on local data

and splitter guesses3. Reduction sums up histograms4. A processor analyzes which splitter guesses

were satisfactory (in terms of load)5. If unsatisfactory splitters, the , processor

broadcasts a new probe, go to step 2; else proceed to next steps

Histogram SortSteps

6. Each processor sends local data to appropriate processors – all-to-all exchange

7. Each processor merges the data chunk it receives

Merits: Only moves the actual data once Deals with uneven distributions

Probe Determination

Should be efficient – done on one processor

The processor keeps track of bounds for all splitters Ideal location of a splitter i is (i+1)n/p When a histogram arrives, the splitter

guesses are scanned

Probe Determination A splitter can either

Be a success – its location is within some threshold of the ideal location

Or not – update the desired splitter to narrow the range for the next guess

Size of a generated probe depends on how many splitters are yet to be resolved

Any interval containing s unachieved splitters is subdivided with sxk/u guess where u is the total number of unachieved splitters and k is the number of newly generated splitters

Merging and all-to-all overlap

For merging p arrays at the end Iterate through all arrays simultaneously Merge using a binary tree

In the first case, we need all the arrays to have arrived

In the second case, we can start as soon as two arrays arrive

Hence this merging can be overlapped with all-to-all

Radix Sort

During every step, the algorithm puts every key in a bucket corresponding to the value of some subset of the key’s bits

A k-bit radix sort looks at k bits every iteration

Easy to parallelize – assign some subset of buckets to each processor

Lad balance – assign variable number of buckets to each processor

Radix Sort – Load Balancing

Each processor counts how many of its keys will go to each bucket

Sum up these histograms with reductions

Once a processor receives this combined histogram, it can adaptively assign buckets

Radix Sort - Analysis

Requires multiple iterations of costly all-to-all

Cache efficiency is low – any given key can move to any bucket irrespective of the destination of the previously indexed key

Affects communication as well