Basic Block Scheduling

Utilize parallelism at the instruction level (ILP)

Time spent in loop execution dominates total execution time

It is a technique that reforms the loop, so as to achieve an overlapped iteration execution

Process Overview

Parallelize a single operation or the whole loop?

More parallelism achievable if we consider the entire loop

Construct Instructions that contain operations from different iterations of the initial loop

Construct a flat schedule, and repeat it over the time taking into account resource and dependence constraints.

Techniques

Software pipelining restructures loops in order to achieve overlapping of various iterations in time

Although this optimization does not create massive amounts of parallelism, it is desirable

There exist two main methods for software pipelining: kernel recognition and modulo scheduling

Modulo Scheduling

We will focus on modulo scheduling technique (it is incorporated in commercial compilers)

We try to select a schedule for one loop iteration and then repeat the schedule

No unrolling applied

Terminology (Dependences)

To make a legal schedule, it is important to know which operations must follow other operations

A conflict exists if two operations cannot execute, at the same time, but it does not matter which one executes first (resource/hardware constraintsresource/hardware constraints)

A dependence exists between two operations if interchanging their order changes the result (data/control data/control dependencesdependences)

Terminology (Data Dependence Graph)

Represent operations as nodes and dependences between operations as directed arcs

Loop carried arcs show relationships between operations of different iterations (may turn DDG into cyclic graph)

Loop independent arcs represent a must follow relationship among operations of the same iteration

Assign arc weights with the form of a (difdif, , minmin) dependence pair.

dif value indicates the number of iterations the dependence spans

min time which is the time to elapse between consecutive execution of the dependent operations

Value (min/dif) is called the slope of the schedule

Terminology (Resource Reservation Table)

Construct Resource Reservation Table

Terminology (Loop Types)

Doall loop in which iterations can proceed in parallel. Those type of loops lead to massive parallelism and are easy to schedule

Doacross loop in which synchronization is needed between operations of various iterations

Doall Loop Example

dif=0, no loop-carried dependences min=1, loop-independent dependences Construct a valid flat schedule. Then, repeat it

Doacross Loop Example (dif=1)

for (i=1; i<=n; i++)

O1: a[i + 1] = a[i] + 1

O2: b[i] = a[i + 1] /2

O3: c[i] = b[i] + 3

O4: d[i] = c[i]

dif=1 for Operation1 (loop-carried dependences exist)

min=1, loop-independent dependences

Construct a valid flat schedule. Then, repeat it

However, repetition is not easy We should take into account

that dif=1 for O1. Each Iteration should start with

one slot delay A legal scheduled has been

achieved

Doacross Loop Example (dif=2)

for (i=1; i<=n; i++)

O1: a[i + 2] = a[i] + 1

O2: b[i] = a[i + 2] /2

O3: c[i] = b[i] + 3

O4: d[i] = c[i]

dif=2 for Operation1 (loop-carried dependences exist)

min=1, loop-independent dependences

Each second Iteration should now start with one slot delay from the previous

This is because dif=2, dependence is deeper (that is less restrictive)

Comparison

In our first example where dif=1 and min=1, the kernel is found in the 4th time slot and is equal to 4 3 2 1. Instructions before and after the kernel are defined as the prelude and postlude of the schedule, respectively

In the second example the loop carried dependence is between iterations that are two apart. This is a less restrictive constraint, so iterations are overlapped more. Indeed, the kernel now is 4 4 3 3 2 2 1 1

Main Idea

Let’s combine all these concepts (data dependence graph, resource reservation tables, schedule, loop types, arcs, flat schedule) in some simple examples

Don’t forget that the main idea behind software pipelining (incl. modulo scheduling) is that the body of a loop can be reformed so as to start one loop iteration before previous iterations have finished

Another Loop Example

for (i=1; i<=n; i++)

O1: a[i] = i * i

O2: b[i] = a[i] * b[i– 1]

O3: c[i] = b[i] / n

O4: d[i] = b[i] % n

O1 is always scheduled in the first time step.

Thus the distance between O1 and the rest of the operations increases in successive iterations.

cyclic pattern (such as those achievable in other examples) never forms.

Initiation Interval

So far we have described the first step of modulo scheduling procedure, that is analysis of DDG for a loop to identify all kinds of dependences

Second step is to try identify the minimum number of instructions required between initiating execution of successive loop iterations

Specifically, the delay between iterations of the new loop is called the Initiation Interval (II)

a) Resource Constrained II b) Dependence Constrained II

Resource Constrained IIres

The resource usage imposes a lower bound on the initiation interval (IIres). For each resource, compute the schedule length necessary to accommodate uses of that resource.

If we have a DDG and 4 available resources, we

try to calculate the maximum usage for every maximum usage for every resourceresource

Example

Resource 2 is required 4 times.

e.g. 2 can only be executed 4 cycles after its previous execution

Suppose that the Flat schedule is as shown.

We repeat it with 4 time slots delay

Methods for computing IIdep

Modulo scheduling is all about calculating the lower bound for the initiation interval

We will present two techniques to compute the dependence constrained II (the calculation of IIres is straightforward)

1) Shortest Path Algorithm 2) Iterative Shortest Path

1) Shortest Path Algorithm

This method uses transitive closure of a graph which is a reachability relationship

Let θ be a cyclic path from a node to itself, minθ be the sum of the min times on the arcs that constitute the cycle and difθ be the sum of dif times on the arcs

So the time between execution of a node and itself depends on II, e.g. time elapsed between execution of α node and another copy difθ iterations away is II * difθ

The maximum min/dif for all cycles is the IIdep

Let’s see the effect of II on cyclic times in this figure

II must be large enough so that II * difθ >= minθ

Repeating Flat schedule

Calculate IIdep

II * difθ >= minθ

0 >= minθ - II * difθ

0 >= minθ - IIdep * difθ

dif

II cyclesdep

minmax )(

Therefore, we select II:

II = max(IIdep, IIres)?

Shortest Path Algorithm Example

IIdep = max([6/2],[4/1],[6/3])

Transitive closure of the graphTransitive closure of the graph

2) Iterative Shortest Path2) Iterative Shortest Path

Simplify the previous method by recomputing the transitive closure of the graph for each possible IIfor each possible II

Use the term of distance (Mdistance (Mabab) between two nodes

In the flat schedule (relative scheduling of each operation of the original iteration, something like list scheduling), the distance between two nodes a, b joint by an arc whose weight is (dif, min) is given by:

Ma,b = min – II * dif

We want to compute the minimum distance two nodes must be separated, but this information is dependent on the initiation intervaldependent on the initiation interval

Distance Distance MMa,ba,b

Effect of Effect of IIII on node precedence on node precedence

Procedure to find II Construct a matrix M where each entry Mi,j represents

the min time between two subsequent nodes i and j This computation gives the earliest time node j can be

placed with respect to node i in the flat schedule

Matrix M

Estimate that II=2

Procedure to find II The next step is to

compute matrix M2 which represents the minimum time difference between nodes of length two

Continue by calculating matrix M3

and so on. Finally, we compute

Γ(Μ) as following 132 ...)( nMMMMM

Matrix M

Matrix M2

Example (II=1)

Matrix M

Matrix Γ(M)

Example (II=2)

Matrix M

Matrix M2Matrix Γ(M)

Example (II=3)

Matrix M

Matrix Γ(M)

Final Result

Γ(Μ) represents the maximum distance between each pair of nodes considering paths of all lengths

A legal II will produce a closure matrix in which entriesentries on the main diagonal are non-positivemain diagonal are non-positive

Positive values on the diagonal is an indication of a small initiation interval

Negative values across the diagonal indicate an adequate estimate of II

Plus or minus

Drawback of this method is that before we are able to construct the matrix M, we should estimate II

However, this technique allows us to tell if the estimate for II is large enough or need to iteratively try larger II

Why use “modulo” term in the first place? Initially, we have the flat schedule F, consisting

of location F1, F2….

Kernel K is formed by overlapping copies of F offset by II

Modulo scheduling results when all operations from locations in the flat schedule that have the same value modulo II are executed simultaneously

Operations from (Fi: i mod II = 1) execute togetherOperations from (Fi: i mod II = 0) execute together

Flat schedule with II=2 Modulo scheduling

Example 1DO I=1, 100

a[I] = b[I-1] + 5;

b[I] = a[I] • I; // mul->2clocks

c[I] = a[I-1] • b[I];

d[I] = c[I];

ENDDO

S1

S2

S3

S4

(0,1)

(1,2)

(0,2)

(1,1)

(0,2)

Graph of example 1

Example 1PRODUCE THE FLAT SCHEDULE S1 and S2 are strongly connected Which one should be placed earlier in the Flat

schedule? S3 and S4 should be placed after S1 and S2

(S3 should precede S4) Eliminate all loop-carried dependences Loop-Independent arcs determine the

sequence of nodes in the flat schedule Flat schedule is therefore:

S1

S2

S3

S4

(0,1)

(1,2)

(0,2)

(1,1)

(0,2)

S1

S2

S3

S4

(0,1)

(0,2)

(0,2)

S1S2

S3

S4

t0:t1:t2:t3:t4:t5:

Example 1

COMPUTE II Using the method of “Shortest Path

Algorithm” Find Strongly connected components Calculate the transitive closure of the

graph

S1

S2

S3

S4

(0,1)

(1,2)

(0,2)

(1,1)

(0,2)

S1

S2

(0,1)

(1,2)

Table I Transitive closure

  Destination Node

Source Node 1 2

1 (1.3) (0.1)

2 (1.2) (1.3)

II = max (3/1, 3/1) = 3.

Example 1

Execution schedule Each blue box

represents the kernel of the pipeline

S1S2

S3

S4

t0:t1:t2:t3:t4:t5:t6:t7:t8:t9:

t10:t11:t12:t13:t14:t15:t16:t17:

kernelS1S2

S3

S4

S1S2

S3

S4

S1S2

S3

S4

S1S2

S3

S4

6-30.5

6E

Worst case scenario: E=0 when II=length of the flat schedule => no overlapping between adjacent operations

length of flat schedule - IIExploitation

length of flat schedule

EXAMPLE 2

DO I=1, 100

S1: a[I] = b[I-1] + 5;

S2: b[I] = a[I] • I;

S3: c[I] = a[I-1] • b[I];

S4: d[I] = c[I] + e[I-2];

S5: e[I] = d[I] • f[I-1];

S6: f[I] = d[I] • 4;

ENDDO

S1

S2

S3

S4

(0,1)

(1,2)

(0,1)(1,1)

(0,2)

S5

(1,2)

(0,1)

S6

(2,2)

(0,1)

EXAMPLE 2 PRODUCE THE FLAT SCHEDULE Eliminate all loop-carried dependences Loop-Independent arcs determine the

sequence of nodes in the flat schedule Flat schedule is therefore:

S1S2

S3

S4S5,6

t0:t1:t2:t3:t4:t5:t6:

S1

S2

S3

S4

(0,1)

(1,2)

(0,1)(1,1)

(0,2)

S5

(1,2)

(0,1)

S6

(2,2)

(0,1)

S1

S2

S3

S4

(0,1)

(0,1)

(0,2)

S5

(0,1)

S6

(0,1)

EXAMPLE 2

Source Node Transitive Closure

  4 5 6

4 (2,3),(2,5) (0,1),(0,3) (0,1) 

5  (2,2)  (2,3),(2,5   (2,3)

6   (2,4)  (0,2)  (2,5) 

S1

S2

S3

S4

(0,1)

(1,2)

(0,1)(1,1)

(0,2)

S5

(1,2)

(0,1)

S6

(2,2)

(0,1)

COMPUTE II using the method of “Shortest Path Algorithm”

Find Strongly connected components Initiation Interval for the first graph is II=3 Calculate the transitive closure of the

second graph

S1

S2

S4

(0,1)

(1,2)

S5

(1,2)

(0,1)

S6

(2,2)

(0,1)

II = max (3/2, 5/2) = 2.5

IItotal = max (2.5, 3) = 3

EXAMPLE 2

Execution code of example 2 Kernel in blue box

length of flat schedule - IIExploitation

length of flat schedule

S1S2

S3

S4S5,6

t0:t1:t2:t3:t4:t5:t6:t7:t8:t9:

t10:t11:t12:t13:t14:t15:

kernel

S1S2

S3

S4S5,6

S1S2

S3

S4S5,6

S1S2

S3

S4S5,6

Prologue

Epilogue

7-30.5714

7E

DO I=1, 100

S1: a[I] = b[I-3] 5; mul 3clks

S2: b[I] = a[I] ; sqrt 4clks

S3: c[I] = a[I-2] b[I-1]; mul 3clks

S4: d[I] = c[I] + 5; add 1clk

S5: e

[I] = d[I-1] + c[I-1]; add 1clk

ENDDO

EXAMPLE 3

S1

S2

S3

S4

S5

(0,2)

(3,4)

(2,2)

(1,4)

(0,3)

(1,3)

(1,1)

Nodes S1, S2 comprise a strongly connected component. II is therefore:

4 2 62

3 0 3II

EXAMPLE 3

S1

S2

S3

S4

S5

(0,2)

(3,4)

(2,2)

(1,4)

(0,3)

(1,3)

(1,1)

Produce the Flat schedule.• Eliminate all loop-carried dependences

• There is not a loop-independent arc across all nodes

• In this case, flat schedule cannot be produced just by following the loop-independent arc

• We need a global method to generate the flat schedule.

• The pre-mentioned method does not always work

• Introduce “Modulo scheduling via hierarchical reduction”

S1

S2

S3

S4

S5

(0,2)

(0,3)

Modulo Scheduling Via Hierarchical Reduction

modify the DDG so as to schedule the strongly connected components of the graph first

strongly connected components of a graph can be found using Tarjan’s algorithm

afterwards, schedule the acyclic DDG

Modulo Scheduling Via Hierarchical Reduction

compute the upper and low bounds where each node can be placed in the flat schedule using Equations below

Its in an iterative method We begin with II=1 trying to find a legal schedule If it is not possible, II is incremented until all nodes are

placed in the flat schedule in legal positions

)(

),(cosmax)( )(

v

vutv

upper

IINulow

),(cos)(),(min()(

),(cos)(),(max()(

vutvuu

uvtvuuII

upperupper

IIlowlow

Equations to initialize low and upper bounds

Equations to update low and upper bounds

CostII(v, u) stands for the cost (measured by dif, min values) in order node v to reach node u. We need thus the cost matrix for the strongly connected nodes (i.e. the transitive closure).

EXAMPLE 4 DO I=1,100S1: a(i) = c(i-1) + d(i-3);S2: b(i) = a(i) • 5; S3: c(i) = b(i-2) • d(i-1); S4: d(i) = c(i) + i;S5: e(i) = d(i);S6: f(i) = d(i-1) • i;S7: g(i) = f(i-1); ENDDO

S1

S7

S6

S5

S4

S3

S2

(0,1)

(1,2)

(3,1)

(2,2)

(1,1)

(0,2)

(0,1) (1,1)

(1,2)

S1

S4

S3

S2

(0,1)

(1,2)

(3,1)

(2,2)

(1,1)

(0,2)

Find strongly connected components Compute the transitive closure

Destination Node

Source Node 1 2 3 4

1 (3,5) , (5,6) (0,1) (2,3) (2,5)

2 (3,4) , (5,5) (3,5) , (5,6) (2,2) (2,4)

3 (1,2) , (3,3) (1,3) , (3,4) (3,5) , (1,3) , (5,6) (0,2)

4 (3,1) , (2,3) (3,2) , (2,4) (1,1), (5,4) (1,3), (5,6)

5 6 3max( , , ) 3

3 5 1II

EXAMPLE 4

Destination Node

Source Node 1 2 3 4

1 (3,5) , (5,6) (0,1) (2,3) (2,5)

2 (3,4) , (5,5) (3,5) , (5,6) (2,2) (2,4)

3 (1,2) , (3,3) (1,3) , (3,4) (3,5) , (1,3) , (5,6) (0,2)

4 (3,1) , (2,3) (3,2) , (2,4) (1,1), (5,4) (1,3), (5,6)

)(

),(cosmax)( )(

v

vutv

upper

IINulow

Compute the Simplified Transitive Closure by keeping the values that give the maximum distance Initialize the upper and low bounds for the nodes in the strongly connected component

Simplified Transitive Closure

where costII(u,v) = M

IIab(min, dif u→v)

( ) max( ( ), ( ) cos ( , )

( ) min( ( ), ( ) cos ( , )

IIlow low

IIupper upper

u u v t v u

u u v t u v

where σ(v) is the time slot in F where the scheduled node has been placed

EXAMPLE 4 (Initialize nodes)

Destination Node

Source Node 1 2 3 4

1 (3,5) , (5,6) (0,1) (2,3) (2,5)

2 (3,4) , (5,5) (3,5) , (5,6) (2,2) (2,4)

3 (1,2) , (3,3) (1,3) , (3,4) (3,5) , (1,3) , (5,6) (0,2)

4 (3,1) , (2,3) (3,2) , (2,4) (1,1), (5,4) (1,3), (5,6)

Initialize the upper and low bounds for the nodes in the strongly connected component

Simplified Transitive Closure

3 3 3

3 3 3

(1) max(cos (2,1),cos (3,1),cos (4,1))

max( (3,4), (1,2), (2,3))

max(4 3 3,2 1 3,3 2 3) 1

low

ab ab ab

t t t

M M M

3 3 3

3 3 3

(2) max(cos (1,2),cos (3,2),cos (4,2))

max( (0,1), (1,3), (2, 4))

max(1,3 1 3,4 2 3) 1

low

ab ab ab

t t t

M M M

3 3 3

3 3 3

(3) max(cos (1,3),cos (2,3),cos (4,3))

max( (2,3), (2,2), (1,1))

max(3 2 3,2 2 3,1 1 3) 2

low

ab ab ab

t t t

M M M

3 3 3

3 3 3

(4) max(cos (1,4),cos (2,4),cos (3,4))

max( (2,5), (2, 4), (0, 2)

max(5 2 3,4 2 3,2 0 3) 2

low

ab ab ab

t t t

M M M

EXAMPLE 4 (Schedule the first node)

S1:[ -1,∞ ]S2:[ 1 ,∞ ]S3:[ -2,∞ ]S4:[ 2 ,∞ ]

Node S3 has the lowest low bound so it is scheduled first. It is placed in time slot 0 (t0). Afterwards, we need to update low and upper bounds for the rest nodes

S3t0:t1:t2:

EXAMPLE 4 (Update nodes S1, S2, S4 )3

3

(1) max( (1), (3) cos (3,1))

max( 1,0 (1,2))

max(0,2 1 3) 0

low low

ab

t

M

3

3

(1) min( , (3) cos (1,3))

min( ,0 (2,3))

min( ,3) 3

upper

ab

t

M

3

3

(2) max( (2), (3) cos (3,2))

max(1,0 (1,3))

max(1,0) 1

low low

ab

t

M

3

3

(2) min( , (3) cos (2,3))

min( ,0 (2,2))

min( ,4) 4

upper

ab

t

M

3

3

(4) max( (4), (3) cos (3,4))

max(2,0 (0,2))

max(2,2) 2

low low

ab

t

M

3

3

(4) min( , (3) cos (4,3))

min( ,0 (1,1))

min( , 2) 2

upper

ab

t

M

S3

S4

t0:t1:t2:

S1:[ 0, 3 ]S2:[ 1 ,4 ]S4:[ 2 ,2 ]

Node S2 has the lowest upper bound, so it is scheduled first and is placed in the position indicated by the value of low bound (i.e. t2).Then, we need to update nodes S1, S2.

EXAMPLE 4 (Update nodes S1, S2 )

S1,3S2S4

t0:t1:t2:

S1:[0,3]S2:[1,4]

Node S3 has the lowest upper bound, so it is scheduled first and is placed in the position indicated by the value of low bound (i.e. t0).

3

3

(1) max( (1), (4) cos (4,1))

max(0,2 (2,3))

max(0,2 3) 0

low low

ab

t

M

3

3

(1) min( (1), (4) cos (1,4))

min(3,2 (2,5))

min(3,3) 3

upper upper

ab

t

M

3

3

(2) max( (2), (4) cos (4,2))

max(1,2 (2,4))

max(1,0) 1

low low

ab

t

M

3

3

(2) min( (2), (4) cos (2,4))

min(4,2 (2,4))

min(4,4) 4

upper upper

ab

t

M

S1,3

S4

t0:t1:t2:

Finally, node S2 is placed in time slot 1 (t1).

EXAMPLE 4 (Condensed Graph )

The acyclic graph is then easy to be scheduled, using the equation below, where σ(α) is the position in the flat schedule of the node on which node n depends. Min and dif values are the sum of mins and difs across the arcs that connect the corresponding nodes. S7

S6

S5

S1234

(0,1) (1,1)

(1,2)

After scheduling the strongly connected component, the new (condensed) graph is shown below

(( , ,min) )( ) max ( ( ) min * )low a n dif Scheduledn a II dif

4(6) ( ( ) min * ) (2 1 3 1) 0low S II dif

4(5) ( ( ) min * ) (2 1 3 0) 3low S II dif 4(7) ( ( ) min * ) (2 3 3 2) 1low S II dif

EXAMPLE 4 (Condensed Graph )

The acyclic graph is then easy to be scheduled, using the equation below, where σ(α) is the position in the flat schedule of the node on which node n depends. Min and dif values are the sum of mins and difs across the arcs that connect the corresponding nodes. S7

S6

S5

S1234

(0,1) (1,1)

(1,2)

After scheduling the strongly connected component, the new (condensed) graph is shown below

(( , ,min) )( ) max ( ( ) min * )low a n dif Scheduledn a II dif

4(6) ( ( ) min * ) (2 1 3 1) 0low S II dif

4(5) ( ( ) min * ) (2 1 3 0) 3low S II dif 4(7) ( ( ) min * ) (2 3 3 2) 1low S II dif

S3,1,6,7S2S4S5

t0:t1:t2:t3:

EXAMPLE 4 (Execution Schedule )

The execution schedule is then

S3,1,6,7S2S4S5

t0:t1:t2:t3:t4:t5:t6:t7:t8:t9:

S3,1,6,7S2S4S5 S3,1,6,7

S2S4S5

II=3