Embed Size (px)
Citation preview
On the Correct Sizing on Meshes Through an Effective Congestion
Management Strategy
P. J. García1, J. Flich2, J. Duato2, I. Johnson3, F. J. Quiles1, F. Naven3
2Technical University of Valencia
Valencia, Spain
3Xyratex
Havant, UK
1University of Castilla-La Mancha
Albacete, Spain
Euro-Par 2005 30 August - 2 September Lisboa, Portugal
Tit
le:
On
th
e C
orr
ect
Siz
ing
on
Mesh
es
Th
rou
gh
an
Eff
ect
ive C
on
gest
ion
Man
ag
em
en
t S
trate
gyy
Con
fere
nce
: Eu
ro-P
ar
20
05
30
Au
gu
st-2
Sep
tem
ber
Lisb
oa, Port
ug
al
2
Outline
• Introduction
• Congestion and HOL blocking
• Why HOL blocking affects network sizing?
• HOL blocking elimination techniques
• RECN
• Performance evaluation
• Conclusions
Tit
le:
On
th
e C
orr
ect
Siz
ing
on
Mesh
es
Th
rou
gh
an
Eff
ect
ive C
on
gest
ion
Man
ag
em
en
t S
trate
gyy
Con
fere
nce
: Eu
ro-P
ar
20
05
30
Au
gu
st-2
Sep
tem
ber
Lisb
oa, Port
ug
al
3
Introduction
PC Clusters:• Alternative to massive parallel computers• Current use:
– High Performance Computing Systems (HPC)– Internet Servers– Storage Area Network (SANs)
• Usually based on high-speed interconnection networks
High-speed interconnection networks:• Myrinet, Infiniband, Quadrics, Advanced Switching…• Main features: High bandwidth, Low latencies• Additional features: Lossless networks, Flexible topology • Network performance may be affected by congestion
Tit
le:
On
th
e C
orr
ect
Siz
ing
on
Mesh
es
Th
rou
gh
an
Eff
ect
ive C
on
gest
ion
Man
ag
em
en
t S
trate
gyy
Con
fere
nce
: Eu
ro-P
ar
20
05
30
Au
gu
st-2
Sep
tem
ber
Lisb
oa, Port
ug
al
5
Congestion and HOL Blocking
Networkcontention
Tit
le:
On
th
e C
orr
ect
Siz
ing
on
Mesh
es
Th
rou
gh
an
Eff
ect
ive C
on
gest
ion
Man
ag
em
en
t S
trate
gyy
Con
fere
nce
: Eu
ro-P
ar
20
05
30
Au
gu
st-2
Sep
tem
ber
Lisb
oa, Port
ug
al
6
Congestion and HOL Blocking
Persistentnetworkcontention
Tit
le:
On
th
e C
orr
ect
Siz
ing
on
Mesh
es
Th
rou
gh
an
Eff
ect
ive C
on
gest
ion
Man
ag
em
en
t S
trate
gyy
Con
fere
nce
: Eu
ro-P
ar
20
05
30
Au
gu
st-2
Sep
tem
ber
Lisb
oa, Port
ug
al
7
Congestion and HOL Blocking
Persistentnetworkcontention
Flow control
Tit
le:
On
th
e C
orr
ect
Siz
ing
on
Mesh
es
Th
rou
gh
an
Eff
ect
ive C
on
gest
ion
Man
ag
em
en
t S
trate
gyy
Con
fere
nce
: Eu
ro-P
ar
20
05
30
Au
gu
st-2
Sep
tem
ber
Lisb
oa, Port
ug
al
8
Congestion and HOL Blocking
Persistentnetworkcontention
Congestionpropagates
Tit
le:
On
th
e C
orr
ect
Siz
ing
on
Mesh
es
Th
rou
gh
an
Eff
ect
ive C
on
gest
ion
Man
ag
em
en
t S
trate
gyy
Con
fere
nce
: Eu
ro-P
ar
20
05
30
Au
gu
st-2
Sep
tem
ber
Lisb
oa, Port
ug
al
9
Congestion and HOL Blocking
• Congestion introduces HOL blocking, and this may degrade network performance dramatically
33%
33%
HOL 33%
33%100%
33%
33%
33%
100%
Tit
le:
On
th
e C
orr
ect
Siz
ing
on
Mesh
es
Th
rou
gh
an
Eff
ect
ive C
on
gest
ion
Man
ag
em
en
t S
trate
gyy
Con
fere
nce
: Eu
ro-P
ar
20
05
30
Au
gu
st-2
Sep
tem
ber
Lisb
oa, Port
ug
al
10
Why HOL blocking affects network sizing?
Network size restricted by:• Required system bandwidth: Network offered
bandwidth must meet the system traffic conditions• Components cost: Recent interconnects (Myrinet,
InfiniBand, ASI) are expensive compared to processors• Power consumption: As network size increases,
higher power consumption, higher heat dissipation• Other constraints: Topology, Links per switch, etc.
Even if the network is correctly sized, HOL blocking may prevent to reach the expected performance
Tit
le:
On
th
e C
orr
ect
Siz
ing
on
Mesh
es
Th
rou
gh
an
Eff
ect
ive C
on
gest
ion
Man
ag
em
en
t S
trate
gyy
Con
fere
nce
: Eu
ro-P
ar
20
05
30
Au
gu
st-2
Sep
tem
ber
Lisb
oa, Port
ug
al
11
Why HOL blocking affects network sizing?
Example 1: Reducing cost and consumption• A solution is to reduce the number of network components• Link utilization increases
• Low link utilization
High link utilization
• High cost and consumptionHigh congestion and HOL
blocking probability
Tit
le:
On
th
e C
orr
ect
Siz
ing
on
Mesh
es
Th
rou
gh
an
Eff
ect
ive C
on
gest
ion
Man
ag
em
en
t S
trate
gyy
Con
fere
nce
: Eu
ro-P
ar
20
05
30
Au
gu
st-2
Sep
tem
ber
Lisb
oa, Port
ug
al
12
Why HOL blocking affects network sizing?
Example 2: Increasing network bandwidth• A solution is to add as network components as necessary• Cost, consumption and length of routes increase
Greater length of routes• Small length of routes
Greater HOL blocking probability when congested
• Low offered bandwidth
Tit
le:
On
th
e C
orr
ect
Siz
ing
on
Mesh
es
Th
rou
gh
an
Eff
ect
ive C
on
gest
ion
Man
ag
em
en
t S
trate
gyy
Con
fere
nce
: Eu
ro-P
ar
20
05
30
Au
gu
st-2
Sep
tem
ber
Lisb
oa, Port
ug
al
13
HOL blocking elimination/reduction techniques
• DAMQs and Virtual Channels • not efficient for multihop networks
• VOQ (Virtual Output Queueing)• VOQ at switch level scales but does not eliminate HOL blocking• VOQ at network level: A separate queue at every input port for every
destination• Number of required resources scales at least quadratically with
network size !!!
• Credit Flow Controlled ATM• References congestion to network output only• Consumes large number of buffers: A separate queue at every
output port for every destination
Tit
le:
On
th
e C
orr
ect
Siz
ing
on
Mesh
es
Th
rou
gh
an
Eff
ect
ive C
on
gest
ion
Man
ag
em
en
t S
trate
gyy
Con
fere
nce
: Eu
ro-P
ar
20
05
30
Au
gu
st-2
Sep
tem
ber
Lisb
oa, Port
ug
al
14
RECN: Regional Explicit Congestion Notification
• RECN is a new efficient and scalable congestion management technique
• Basic ideas:• The real problem is not the congestion, but its negative effects
(HOL blocking)• By eliminating HOL blocking, congestion becomes harmless• Non-congested flows do not introduce significant HOL blocking
• HOL blocking elimination: • Packets belonging to congested flows are stored in specific Set
Aside Queues (SAQs)• Packets belonging to non-congested flows are stored in a
“common” queue
• Implementation requirements:• Deterministic source routing• A reduced number of SAQs per port, controlled by a CAM
Tit
le:
On
th
e C
orr
ect
Siz
ing
on
Mesh
es
Th
rou
gh
an
Eff
ect
ive C
on
gest
ion
Man
ag
em
en
t S
trate
gyy
Con
fere
nce
: Eu
ro-P
ar
20
05
30
Au
gu
st-2
Sep
tem
ber
Lisb
oa, Port
ug
al
15
How RECN works
• RECN basic procedure:• Congested points are detected in any egress or ingress
switch port of the network• The routes to detected congested points are
progressively notified to ingress and egress ports crossed by congested flows
• After receiving a notification, a port allocates a SAQ for the detected congested point
• A packet arriving to a port will be stored in a SAQ if it will pass through the congested point associated to that SAQ
• A packet arriving at a port will be stored in the “common” queue if its route does not match any SAQ
• SAQs can be deallocated, and later allocated for other congested points
Tit
le:
On
th
e C
orr
ect
Siz
ing
on
Mesh
es
Th
rou
gh
an
Eff
ect
ive C
on
gest
ion
Man
ag
em
en
t S
trate
gyy
Con
fere
nce
: Eu
ro-P
ar
20
05
30
Au
gu
st-2
Sep
tem
ber
Lisb
oa, Port
ug
al
16
A congestion point forms
How RECN Works
Tit
le:
On
th
e C
orr
ect
Siz
ing
on
Mesh
es
Th
rou
gh
an
Eff
ect
ive C
on
gest
ion
Man
ag
em
en
t S
trate
gyy
Con
fere
nce
: Eu
ro-P
ar
20
05
30
Au
gu
st-2
Sep
tem
ber
Lisb
oa, Port
ug
al
17
How RECN Works
Cold queue fills over a threshold
Tit
le:
On
th
e C
orr
ect
Siz
ing
on
Mesh
es
Th
rou
gh
an
Eff
ect
ive C
on
gest
ion
Man
ag
em
en
t S
trate
gyy
Con
fere
nce
: Eu
ro-P
ar
20
05
30
Au
gu
st-2
Sep
tem
ber
Lisb
oa, Port
ug
al
19
How RECN Works
Internal notification to each input port
sending packets to the output port
Tit
le:
On
th
e C
orr
ect
Siz
ing
on
Mesh
es
Th
rou
gh
an
Eff
ect
ive C
on
gest
ion
Man
ag
em
en
t S
trate
gyy
Con
fere
nce
: Eu
ro-P
ar
20
05
30
Au
gu
st-2
Sep
tem
ber
Lisb
oa, Port
ug
al
21
How RECN Works
Input ports allocate a new SAQ for
packets addressed tothe congested output port
Tit
le:
On
th
e C
orr
ect
Siz
ing
on
Mesh
es
Th
rou
gh
an
Eff
ect
ive C
on
gest
ion
Man
ag
em
en
t S
trate
gyy
Con
fere
nce
: Eu
ro-P
ar
20
05
30
Au
gu
st-2
Sep
tem
ber
Lisb
oa, Port
ug
al
23
How RECN Works
Notification sent whenthe SAQ fills
over a threshold
Tit
le:
On
th
e C
orr
ect
Siz
ing
on
Mesh
es
Th
rou
gh
an
Eff
ect
ive C
on
gest
ion
Man
ag
em
en
t S
trate
gyy
Con
fere
nce
: Eu
ro-P
ar
20
05
30
Au
gu
st-2
Sep
tem
ber
Lisb
oa, Port
ug
al
25
How RECN Works
A new SAQ allocatedfor the congested port
at each output port
Tit
le:
On
th
e C
orr
ect
Siz
ing
on
Mesh
es
Th
rou
gh
an
Eff
ect
ive C
on
gest
ion
Man
ag
em
en
t S
trate
gyy
Con
fere
nce
: Eu
ro-P
ar
20
05
30
Au
gu
st-2
Sep
tem
ber
Lisb
oa, Port
ug
al
26
How RECN Works
Internal notification when the SAQ fills over
A threshold
Tit
le:
On
th
e C
orr
ect
Siz
ing
on
Mesh
es
Th
rou
gh
an
Eff
ect
ive C
on
gest
ion
Man
ag
em
en
t S
trate
gyy
Con
fere
nce
: Eu
ro-P
ar
20
05
30
Au
gu
st-2
Sep
tem
ber
Lisb
oa, Port
ug
al
27
How RECN Works
The input port allocatesA new SAQ
Tit
le:
On
th
e C
orr
ect
Siz
ing
on
Mesh
es
Th
rou
gh
an
Eff
ect
ive C
on
gest
ion
Man
ag
em
en
t S
trate
gyy
Con
fere
nce
: Eu
ro-P
ar
20
05
30
Au
gu
st-2
Sep
tem
ber
Lisb
oa, Port
ug
al
28
How RECN Works
At the end, the congestion tree builds and is mapped
entirely onto SAQs
Tit
le:
On
th
e C
orr
ect
Siz
ing
on
Mesh
es
Th
rou
gh
an
Eff
ect
ive C
on
gest
ion
Man
ag
em
en
t S
trate
gyy
Con
fere
nce
: Eu
ro-P
ar
20
05
30
Au
gu
st-2
Sep
tem
ber
Lisb
oa, Port
ug
al
29
Performance Evaluation
•Two network-sizing scenarios considered:• Network cost and consumption reduction:
– Network is downsized, keeping constant the number of total system endnodes
• Network bandwidth increase: – Network size is increased, keeping constant the
number of endnodes per switch
•Evaluation based on simulation results•Evaluation metric:
• Network relative throughput when using: – RECN– VOQ at network level (VOQnet)– VOQ at switch level (VOQsw)
Tit
le:
On
th
e C
orr
ect
Siz
ing
on
Mesh
es
Th
rou
gh
an
Eff
ect
ive C
on
gest
ion
Man
ag
em
en
t S
trate
gyy
Con
fere
nce
: Eu
ro-P
ar
20
05
30
Au
gu
st-2
Sep
tem
ber
Lisb
oa, Port
ug
al
30
Simulation Model
•Simulation assumptions:• Mesh topologies• Deterministic routing (X-Y)• 128 KB memories at ingress/egress ports• Multiplexed crossbar (BW=12 Gbps)• Serial full-duplex pipelined links (BW=8 Gbps)• 64-byte packets• Credit-based and Xon-Xoff (for SAQs) flow
control• Maximum of 8 SAQs at ingress/egress ports
(RECN)
Tit
le:
On
th
e C
orr
ect
Siz
ing
on
Mesh
es
Th
rou
gh
an
Eff
ect
ive C
on
gest
ion
Man
ag
em
en
t S
trate
gyy
Con
fere
nce
: Eu
ro-P
ar
20
05
30
Au
gu
st-2
Sep
tem
ber
Lisb
oa, Port
ug
al
31
Network Configurations
• Five different mesh-based network configurations:
Network configuration
Mesh
topologySwitches Endnodes per switch
#1 16 x 16 256 1
#2 8 x 8 64 4
#3 4 x 4 16 16
#4 8 x 8 64 1
#5 4 x 4 16 1
Tit
le:
On
th
e C
orr
ect
Siz
ing
on
Mesh
es
Th
rou
gh
an
Eff
ect
ive C
on
gest
ion
Man
ag
em
en
t S
trate
gyy
Con
fere
nce
: Eu
ro-P
ar
20
05
30
Au
gu
st-2
Sep
tem
ber
Lisb
oa, Port
ug
al
32
Traffic Load
• Three different synthetic traffic patterns:
Normal traffic Congestion tree
Traffic case #Sources Destination #Sources Destination
#1 100% Random - -
#2 87.5% Random 12.5% hot-spot
#3 75% Random 25% hot-spot
Tit
le:
On
th
e C
orr
ect
Siz
ing
on
Mesh
es
Th
rou
gh
an
Eff
ect
ive C
on
gest
ion
Man
ag
em
en
t S
trate
gyy
Con
fere
nce
: Eu
ro-P
ar
20
05
30
Au
gu
st-2
Sep
tem
ber
Lisb
oa, Port
ug
al
33
Network Cost and Consumption Reduction
•Relative throughput:• Traffic case 1• 256-endnodes networks
(Network configurations 1,2,3)
16 x 16 switches network (Conf. 1)
8 x 8 switches network (Conf. 2)
4 x 4 switches network (Conf. 3)
Tit
le:
On
th
e C
orr
ect
Siz
ing
on
Mesh
es
Th
rou
gh
an
Eff
ect
ive C
on
gest
ion
Man
ag
em
en
t S
trate
gyy
Con
fere
nce
: Eu
ro-P
ar
20
05
30
Au
gu
st-2
Sep
tem
ber
Lisb
oa, Port
ug
al
34
Network Cost and Consumption Reduction
•Relative throughput:• Traffic case 2• 256-endnodes networks
(Network configurations 1,2,3)
16 x 16 switches network (Conf. 1)
8 x 8 switches network (Conf. 2)
4 x 4 switches network (Conf. 3)
Tit
le:
On
th
e C
orr
ect
Siz
ing
on
Mesh
es
Th
rou
gh
an
Eff
ect
ive C
on
gest
ion
Man
ag
em
en
t S
trate
gyy
Con
fere
nce
: Eu
ro-P
ar
20
05
30
Au
gu
st-2
Sep
tem
ber
Lisb
oa, Port
ug
al
35
Network Cost and Consumption Reduction
•Relative throughput:• Traffic case 3• 256-endnodes networks
(Network configurations 1,2,3)
16 x 16 switches network (Conf. 1)
8 x 8 switches network (Conf. 2)
4 x 4 switches network (Conf. 3)
Tit
le:
On
th
e C
orr
ect
Siz
ing
on
Mesh
es
Th
rou
gh
an
Eff
ect
ive C
on
gest
ion
Man
ag
em
en
t S
trate
gyy
Con
fere
nce
: Eu
ro-P
ar
20
05
30
Au
gu
st-2
Sep
tem
ber
Lisb
oa, Port
ug
al
36
Network Bandwidth Increase
•Relative throughput:• Traffic case 1• 1 endnode/switch networks
(Network configurations 1,4,5)
16 x 16 switches network (Conf. 1)
8 x 8 switches network (Conf. 4)
4 x 4 switches network (Conf. 5)
Tit
le:
On
th
e C
orr
ect
Siz
ing
on
Mesh
es
Th
rou
gh
an
Eff
ect
ive C
on
gest
ion
Man
ag
em
en
t S
trate
gyy
Con
fere
nce
: Eu
ro-P
ar
20
05
30
Au
gu
st-2
Sep
tem
ber
Lisb
oa, Port
ug
al
37
Network Bandwidth Increase
•Relative throughput:• Traffic case 2• 1 endnode/switch networks
(Network configurations 1,4,5)
16 x 16 switches network (Conf. 1)
8 x 8 switches network (Conf. 4)
4 x 4 switches network (Conf. 5)
Tit
le:
On
th
e C
orr
ect
Siz
ing
on
Mesh
es
Th
rou
gh
an
Eff
ect
ive C
on
gest
ion
Man
ag
em
en
t S
trate
gyy
Con
fere
nce
: Eu
ro-P
ar
20
05
30
Au
gu
st-2
Sep
tem
ber
Lisb
oa, Port
ug
al
38
Network Bandwidth Increase
•Relative throughput:• Traffic case 3• 1 endnode/switch networks
(Network configurations 1,4,5)
16 x 16 switches network (Conf. 1)
8 x 8 switches network (Conf. 4)
4 x 4 switches network (Conf. 5)
Tit
le:
On
th
e C
orr
ect
Siz
ing
on
Mesh
es
Th
rou
gh
an
Eff
ect
ive C
on
gest
ion
Man
ag
em
en
t S
trate
gyy
Con
fere
nce
: Eu
ro-P
ar
20
05
30
Au
gu
st-2
Sep
tem
ber
Lisb
oa, Port
ug
al
39
Network Bandwidth Increase
•Maximum number of SAQs
used by RECN:• Traffic case 3• 1 endnode/switch networks
(Network configurations 1,4,5)
16 x 16 switches network (Conf. 1)
8 x 8 switches network (Conf. 4)
4 x 4 switches network (Conf. 5)
Tit
le:
On
th
e C
orr
ect
Siz
ing
on
Mesh
es
Th
rou
gh
an
Eff
ect
ive C
on
gest
ion
Man
ag
em
en
t S
trate
gyy
Con
fere
nce
: Eu
ro-P
ar
20
05
30
Au
gu
st-2
Sep
tem
ber
Lisb
oa, Port
ug
al
40
Conclusions
• HOL blocking may affect the performance of networks dimensioned with different restrictions
• We have analyzed the importance of using an efficient HOL blocking elimination strategy
• We have shown that RECN allows to size the network in any way while keeping network performance at the expected maximum
• RECN only requires a small number of SAQs for a wide range of network sizes, so it is a scalable strategy
On the Correct Sizing on Meshes Through an Effective Congestion
Management Strategy
P. J. García1, J. Flich2, J. Duato2, I. Johnson3, F. J. Quiles1, F. Naven3
2Technical University of Valencia
Valencia, Spain
3Xyratex
Havant, UK
1University of Castilla-La Mancha
Albacete, Spain
Euro-Par 2005 30 August - 2 September Lisboa, Portugal
