Embed Size (px)
Citation preview
IEEE P
roof
IEEE TRANSACTIONS ON GREEN COMMUNICATIONS AND NETWORKING 1
Performance Evaluation of a Power-Efficient andRobust 60 GHz Wireless Server-to-Server
Datacenter NetworkSayed Ashraf Mamun, Senior Member, IEEE, Sree Gowrishankar Umamaheswaran, Student Member, IEEE,
Amlan Ganguly , Member, IEEE, Minseok Kwon, Member, IEEE, and Andres Kwasinski, Senior Member, IEEE
Abstract—Datacenters have become the digital backbone of the1
modern world. To support the growing demands on bandwidth,2
datacenter networks (DCNs) consume an increasing amount of3
power. Additionally, the complex cabling in traditional data-4
centers poses design and maintenance challenges and increases5
the energy cost of the cooling infrastructure by obstructing the6
flow of chilled air. In this paper, we present a wireless server-7
to-server datacenter network architecture using millimeter-wave8
links to eliminate the need for power-hungry switching fabric9
of traditional fat-tree based datacenter networks. The server-to-10
server wireless datacenter network (S2S-WiDCN) architecture11
requires line-of-sight (LoS) between servers to establish direct12
communication links. However, in the presence of an obstruction13
such as an IT technician, the LoS may be blocked. To address14
this issue, we propose a novel obstruction-aware adaptive rout-15
ing algorithm for S2S-WiDCN. We evaluate the performance of16
S2S-WiDCN in terms of traffic flow completion duration and17
throughput, with different kinds of datacenter traffic as well18
as in presence of obstruction to LoS links. We also compare19
the performance and power efficiency of S2S-WiDCN with the20
traditional fat tree DCN as well as a top-of-rack (ToR)-to-ToR21
wireless DCN.22
Index Terms—Datacenter network, millimeter wave communi-23
cation, OFDM, server-to-server, wireless communication.24
I. INTRODUCTION25
DATACENTERS have become an essential part of com-26
puting resources as they provide large storage banks27
and processing power for cloud services. According to28
a 2016 report from Lawrence Berkeley National Laboratory,29
it is projected that the power consumption of all data-30
centers in the USA is going to be 73 billion KWh per31
year by 2020 and the power consumption of the Datacenter32
Manuscript received February 2, 2018; revised May 2, 2018; acceptedMay 14, 2018. This work was supported by the U.S. National ScienceFoundation CAREER under Grant CNS-1553264. The associate editor coor-dinating the review of this paper and approving it for publication wasE. Ayanoglu. (Corresponding author: Amlan Ganguly.)
S. A. Mamun is with the Golisano College of Computing and InformationSciences, Rochester Institute of Technology, Rochester, NY 14623 USA(e-mail: [email protected]).
S. G. Umamaheswaran, A. Ganguly, and A. Kwasinski are with theDepartment of Computer Engineering, Rochester Institute of Technology,Rochester, NY 14623 USA (e-mail: [email protected]; [email protected];[email protected]).
M. Kwon is with the Department of Computer Science, Rochester Instituteof Technology, Rochester, NY 14623 USA (e-mail: [email protected]).
Digital Object Identifier 10.1109/TGCN.2018.2838525
Network (DCN) will continuously increase to support the 33
increasing demand for bandwidth [1]. Keeping this high- 34
fidelity network active, often constructed from legacy switch- 35
ing fabrics, consumes 10 to 50% of the total IT power 36
depending on server utilization [2]. This high-power consump- 37
tion warrants immediate attention to improve efficiency and 38
power consumption of datacenters in general and DCNs in 39
particular. 40
Traditionally, DCNs are interconnected with fat tree topolo- 41
gies using wired links over multiple hierarchical levels of 42
aggregation. These tree-based networks exhibit inherent limita- 43
tions in scalability and oversubscription as they rely on copper 44
and optical cable-based links and a hierarchical topology [3]. 45
Moreover, wired network technologies require power-hungry 46
switches and create large bundles of cables, causing design 47
overheads and maintenance challenges and obstructions to the 48
flow of chilled air for cooling [4]. Inefficient cooling only 49
exacerbates the energy efficiency problem especially for small 50
and mid-size datacenters with hundred to a few thousands of 51
individual servers in educational institutions and private enter- 52
prises. This is because these are designed and deployed in an 53
ad-hoc manner often leading to structural and functional het- 54
erogeneity making regular systematic design impossible. To 55
address these common design issues faced by wired datacen- 56
ters, wireless datacenter architectures are being investigated 57
as a promising alternative. The capability of the unlicensed 58
60GHz wireless band to deliver very high communication rates 59
has led to the development and approval of the IEEE 802.11ad 60
wireless local area network (WLAN) standard [5]. Therefore, 61
recently proposed designs leverage newly developed tech- 62
nologies in the unlicensed 60GHz wireless band for wireless 63
DCNs [6], [7]. 64
Advancements in the 60GHz technologies allow the 65
transceivers to consume low power, in the range of a few 66
Watts [8], [9] and establish multi-gigabit communication 67
channels [10]. Directional horn antennas, which have been 68
used in wireless DCNs [11] as well as more recently devel- 69
oped phased arrays of antennas in the 60GHz bands [12] 70
are capable of providing high directional gains and beam 71
steering capability between wireless transceivers. Using such 72
antennas, the 60GHz channels can exhibit spatial reusability, 73
allowing multiple concurrent links reusing frequency bands 74
to be formed within the same datacenter. The low power 75
consumption combined with the ability to form concurrent 76
2473-2400 c© 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
IEEE P
roof
2 IEEE TRANSACTIONS ON GREEN COMMUNICATIONS AND NETWORKING
Fig. 1. Server-to-server wireless datacenter network (S2S-WiDCN) showingsome horizontal wireless paths (red) and one of the vertical wireless commu-nication planes (yellow). All servers are capable of communicating via theirrespective horizontal line and vertical plane.AQ1
multi-gigabit channels makes these transceivers ideal for use77
in power-efficient wireless DCNs.78
In this paper, we present a server-to-server wireless DCN79
architecture called S2S-WiDCN based on the 60GHz wire-80
less technology for a small or medium-size datacenter.81
Through direct server-to-server wireless links using directional82
antenna arrays the power-hungry switching fabric of traditional83
DCNs are eliminated, resulting in significant power savings in84
data transfer [13]. The communication between servers in the85
wireless DCN are achieved along horizontal lines and vertical86
planes as shown in Fig. 1, which shows only a few horizontal87
red lines and a single yellow plane for clarity. The horizon-88
tal communication lines are used for data transfer between89
rows, whereas, the yellow planes are used for transmissions90
within the same row. However, the presence of any obstruc-91
tion in the datacenter aisles such as an IT technician may result92
in blocking of the horizontal Line-of-Sight (LoS) lines caus-93
ing a failure in data transmission. Therefore, in this paper we94
propose a novel adaptive routing mechanism to recover from95
such obstructions. We compare the performance as well as96
power savings of the proposed server-to-server wireless DCN97
(S2S-WiDCN) to that of a conventional hierarchical fat tree-98
based DCN. We have considered various kinds of datacenter99
traffic for these evaluations, which are typically encountered100
in index-search/query-response and multimedia/video applica-101
tions. In this way, we demonstrate that S2S-WiDCN is able102
to sustain and provide performance comparable to the con-103
ventional counterpart at five to seventeen times lower power104
consumption. The novel contributions of this paper are:105
• We design the S2S-WiDCN architecture with an106
obstruction-avoidance adaptive routing for server-to-107
server LoS communication using 60GHz wireless links.108
• We evaluate the performance of S2S-WiDCN with differ-109
ent kinds of traffic patterns depending on different types110
of applications.111
• We evaluate the performance of S2S-WiDCN in presence112
of obstructions to LoS paths.113
• We modeling and estimate the power consumption114
of S2S-WiDCN and compare it to traditional tree-115
based DCNs.116
The rest of the paper is organized as follows. Section II 117
presents the related works regarding both novel wired 118
and wireless datacenter networks. Section III discusses 119
the topology, communication protocol and an obstruction- 120
avoidance routing. In Section IV we present the evaluation 121
of S2S-WiDCN. The conclusions are discussed in Section V. 122
II. RELATED WORK 123
Various designs have been proposed to address DCN design 124
issues such as energy consumption, cabling complexity, scala- 125
bility, and over-subscription. One popular topology used today 126
in datacenter networks is a fat-tree topology [14]. In this topol- 127
ogy, servers are connected through a hierarchy of access, 128
aggregate and core layer switches. The drawbacks of fat-tree 129
networks include congestion or oversubscription at the upper 130
levels of the hierarchy [11]. Several alternative DCN architec- 131
tures have been proposed such as, BCube [15], Dcell [16], 132
DOS [17], VL2 [18], FireFly [19] and Helios [20]. Optical 133
interconnects were explored as well, to improve the perfor- 134
mance even further [21]. However, these innovations still rely 135
on copper or optical cables and do not mitigate the challenges 136
due to high power consumption, design and maintenance of 137
a DCN with physical links. 138
To alleviate the issues of DCNs with power hun- 139
gry switching fabrics and bundles of cables wireless dat- 140
acenters with mm-wave inter-rack links are envisioned 141
in [6], [11], [22], and [23]. Most of the recent works on wire- 142
less datacenters propose interconnecting entire racks of servers 143
as units with 60GHz wireless links primarily in order to uti- 144
lize the commodity Ethernet switching between servers inside 145
individual racks [6]. Phased antenna arrays or directional horn 146
antennas are used to establish wireless links between Top- 147
of-Racks (ToRs) in the entire datacenter [23], [24], and [25]. 148
LoS communication paths are necessary between the antennas 149
for reliable communication in a wireless datacenter [23]. Paths 150
through metal frames and racks will have increased losses 151
due to obstructions. Hence, reflectors on ceilings in the form 152
of metallic mirrors or signal relays can be mounted to form 153
paths where direct LoS does not exist [26]. Cylindrical [27] 154
or polygonal [28] arrangements of servers are proposed to 155
create LoS wireless links between servers. This however, 156
requires non-traditional cylindrical arrangement of servers 157
having implications on cooling, server density and scalabil- 158
ity of the DCN which are not well known at this point. Our 159
work is fundamentally different from the existing wireless 160
DCN architectures as we propose LoS based server-to-server 161
wireless connectivity while maintaining regular rectangular 162
arrangement of server racks. 163
III. WIRELESS DATACENTER NETWORK ARCHITECTURE 164
In this section, we discuss the architecture of S2S- 165
WiDCN. We describe the design methodologies, the 166
adopted antenna technology, and finally its communication 167
protocols. 168
A. Wireless Datacenter Network Topology 169
In S2S-WiDCN, the datacenter racks are laid out in the tra- 170
ditional rectangular pattern, adjacent to one another with aisles 171
IEEE P
roof
MAMUN et al.: PERFORMANCE EVALUATION OF POWER-EFFICIENT AND ROBUST 60 GHz WIRELESS S2S DCN 3
Fig. 2. Single server showing two antenna arrays and the WiFi controlmodule. Inset: each antenna array.
running between rows of racks. In order to avoid obstruction to172
the wireless communication links, we establish wireless links173
only along horizontal lines and vertical planes to communi-174
cate between any two servers in the three-dimensional space as175
shown in Fig. 1. To achieve this, each server will be equipped176
with two high gain 60 GHz antenna arrays [12]. We propose177
attaching one of the antennas on the top of the server to enable178
the communication in the horizontal direction and other one179
on the back or front of the server projecting out from the rack180
as shown in Fig. 2 to enable communication in the vertical181
plane. To avoid interference and obstructions from the rack182
frames, communication in the horizontal plane are restricted183
only to a single line between horizontally aligned servers.184
Datacenters are typically arranged in hot aisle/cold aisle lay-185
out where servers in adjacent rows are either face-to-face or186
back-to-back [29]. To minimize the interference between two187
neighboring rows of racks, servers will have the provision to188
connect the antenna for vertical plane communication either189
on the back or on the front side. This will ensure that no190
two separate vertical planes will exist in a single aisle and191
hence eliminate interference between vertical planes. Using192
the beam-steering capability of the antenna array, LoS links193
between communicating servers can be established with the194
help of a control interface discussed in Section III-B. Each195
server is assigned a unique ID according to its geometric196
location in order to help determine the beam-steering angles.197
The angles are precomputed depending on the location of the198
communicating servers. All the metallic surfaces and walls of199
the datacenter should be coated with anti-reflecting material200
cover [30] to eliminate the multipath propagation of a signal.201
Such anti-reflection material with low reflection coefficients202
are relatively easily available and only add another additional203
layer to the building infrastructure without significant change204
in building design.205
The proposed design can be adapted for racks having doors206
with few minor modifications. Traditional doors for datacen-207
ter racks comes in two varieties- perforated metal sheet with208
breathable mesh design or acrylic glass with metal frame. For209
the perforated metal doors, we propose a series of rectan-210
gular opening areas as shown in Fig. 3(a), aligned with the211
(a) (b)
Fig. 3. Creating LoS between servers in presence of (a) metal rack doors inhorizontal and vertical planes and (b) acrylic glass door in vertical plane.
top antenna in all the racks. This will ensure horizontal LoS 212
between top antennas of the servers. The metal doors are per- 213
forated to aid in cooling air circulation, and the rectangular 214
openings foster air conditioning further. By contrast, for the 215
doors with acrylic glass, the doors are designed to contain 216
the chilled air implementing air conditioning which is differ- 217
ent compared to datacenters with metal doors. This material 218
is relatively transparent to 60GHz wireless band (compared to 219
actual glass) with only 1.02 dB/cm of path loss through it [31]. 220
While each door’s glass is thinner than 1cm, in case of paths 221
through many doors, the link-budget analysis should take into 222
account this loss while designing the wireless datacenter with 223
this type of door. Therefore, in case, the number of rows in 224
the datacenter is high, this type of rack is not recommended 225
for the S2S-WiDCN architecture. Furthermore, to use racks 226
with glass door, the top antennas can be mounted on the side 227
panel of the racks with a high quality, low loss 60GHz cable 228
or waveguide rather than on top of the servers. To create LoS 229
between the servers in different racks across multiple rows, 230
a narrow open space is required between adjacent racks in the 231
same row to accommodate the horizontal LoS lines through 232
the sides of the racks. To prevent hot and cold air contam- 233
ination, thermal ducts individually deployed in each rack as 234
envisioned in [32], must be used in such racks to remove the 235
hot air completely from the floor through the racks. 236
To create LoS in the vertical plane in presence of doors 237
in the rack, we propose mounting the antenna arrays on an 238
extended panel as shown in Fig. 3(a) and (b) for metal and 239
acrylic door respectively. The antennas will be connected to 240
the corresponding servers with high-quality, low-loss 60GHz 241
cable or waveguide fitted to the frame which can be coupled 242
to the servers. This will ensure that movements of the doors 243
will not affect the antenna alignment. 244
B. Antenna Technology for the Wireless Datacenter 245
Each server in S2S-WiDCN is equipped with a wireless 246
module consisting of a transceiver and two accompany- 247
ing antenna arrays [12]. This particular array [12] is fabri- 248
cated using semiconductor lithography techniques on a single 249
wafer and hence, is extremely compact with a size of only 250
10.5mm×3.3mm. As the radiation pattern suggests, the array 251
provides high directional gain of 9dBi in the forward and back- 252
ward directions. Moreover, by adjusting the relative phase of 253
IEEE P
roof
4 IEEE TRANSACTIONS ON GREEN COMMUNICATIONS AND NETWORKING
the antenna elements by activating various feed paths, beam-254
steering can be accomplished over an angle of 60 degrees. As255
horizontal communication happens in a single straight line,256
no beam-steering is required in the antenna arrays on top of257
the servers. However, as the range of beam steering angle258
is 60 degrees for this particular array, 6 antenna arrays are259
required to cover the entire 360 degree panorama in the verti-260
cal plane. Only one out of the 6 arrays will need to be signaled261
at any given point of time to establish a single link involv-262
ing that server. Electronic beam-steering for the antenna array263
has negligible latency compared to mechanically steered horn264
antennas used in earlier wireless DCNs [25], [26]. Moreover,265
the antenna array being extremely compact requires very tiny266
space on top of each server to enable LoS communication in267
the horizontal direction. The effect of these spaces on the ver-268
tical server density in the datacenter racks is discussed and269
quantified in the next section.270
This beam-steering of the transmitting and receiving anten-271
nas is controlled by using a separate control interface using272
IEEE 802.11 2.4/5 GHz ISM bands. Although the data rates273
sustained by the IEEE 802.11 2.4/5 GHz bands are much274
lower than the 60GHz bands, it is sufficient for the short275
control packets. Moreover, the isotropic antennas in the IEEE276
802.11 2.4/5 GHz modules do not require any antenna steering277
before the control messages can be transmitted. When a traffic278
flow between a pair of servers is created, a short control or279
header packet for the flow will be sent over the IEEE 802.11280
2.4/5 GHz ISM band to enable communicating servers to steer281
their antenna beams towards each other when required. The282
details of the steering are discussed in Section III-D.283
C. Wireless Communication Protocols284
Establishing connections between servers require reliable285
wireless 60GHz physical and Medium Access Control (MAC)286
layer protocols. The IEEE 802.11ad standard [5] is designed287
for 60GHz wireless LANs. This standard defines a physical288
layer protocol that supports beam-forming, and also supports289
extremely high data rates in both a single carrier (SC) and290
Orthogonal Frequency Division Multiplexing (OFDM) mode291
of operation with maximum achievable data rates are 4.62Gbps292
and 6.76Gbps respectively. Motivated by these high data rates,293
we adopt IEEE 802.11ad as the 60GHz physical layer protocol294
for wireless datacenters. IEEE 802.11ad MAC layer protocol295
incorporates a Carrier Sense Multiple Access (CSMA) mecha-296
nism for on-demand establishment of wireless links depending297
upon the traffic flow requirements. The MAC layer protocol298
establishes as many non-interfering links as possible, greedily299
on a first-come first-serve basis until all traffic flow demands300
are met or all the available OFDM channels are exhausted. The301
IEEE 802.11ad standard only allows wireless links to be estab-302
lished where a bit error rate (BER) of 3×10−7 or lower can303
be achieved considering the signal to interference plus noise304
ratio (SINR) and the corresponding data rates to be sustained305
by the wireless link. Once a flow is found not to be feasible306
due to interference with already-existing flows in any of the307
OFDM channels, the flow is no longer considered serviced and308
that demand is left incomplete. We evaluate the performance309
(a) (b)
(c) (d)
Fig. 4. Possible communication paths between servers situated in (a) samerack, (b) same vertical plane, (c) same horizontal line, and (d) differenthorizontal lines and vertical planes.
of the MAC layer protocol through a comparison and analy- 310
sis against similar-sized wired networks in our case studies in 311
the next section. We use TCP as the transport layer protocol 312
for reliable packet delivery for its widespread use and well- 313
known characteristics in datacenter networks as well as in the 314
Internet. 315
The feasibility of 60GHz communication in datacen- 316
ters is established with physical channel measurements 317
in [11] and [23]. However, the effect of temperature on the 318
60GHz channel has not been considered. The 60GHz chan- 319
nel is known to be affected by molecular absorption, which 320
in turn, is affected by the temperature. Due to variation in 321
temperature in a datacenter between hot and cold airflows, the 322
wireless path loss may vary. However, the recommended range 323
of temperature variation in a datacenter according to ASHRAE 324
thermal guideline is between 18 to 27 degree Celsius [33]. 325
The path loss varies by roughly 2dB/km for this range of 326
temperature [34]. Therefore, for typical data center dimensions 327
in the range of few meters, the variation is negligible. 328
D. Adaptive Routing Protocol for S2S-WiDCN 329
In this section, we propose an adaptive routing protocol for 330
S2S-WiDCN, which is capable of routing traffic flows even in 331
the presence of obstruction of the LoS between two servers 332
due to reasons such as presence of human beings along the 333
datacenter aisles. First, we describe the default routing mech- 334
anism followed by our proposed method to make the default 335
routing adaptive for robustness against obstruction of LoS. 336
1) Default Routing Mechanism: For the default routing 337
mechanism, we develop a Horizontal-First routing described 338
in this subsection. The server arrangement plays a vital role 339
in the design of this routing protocol. For the purpose of 340
the Horizontal-First routing algorithm the servers are con- 341
sidered to be arranged in a 3D Cartesian coordinate system 342
IEEE P
roof
MAMUN et al.: PERFORMANCE EVALUATION OF POWER-EFFICIENT AND ROBUST 60 GHz WIRELESS S2S DCN 5
Fig. 5. Pseudocode for Horizontal-First routing protocol.
with each server having a unique 3D coordinate as shown343
in Fig. 4. In this coordinate system, X-axis runs along rows,344
Y-axis runs along columns and Z-axis runs along racks as345
shown in Fig. 4(a). Server-to-server communication in a data-346
center can be broadly classified into two types, i.e., inter-rack347
and intra-rack communication based on the location of the348
source and destination servers. All the intra-rack communica-349
tions are completed in one hop in the vertical plane as shown350
in Fig. 4(a) whereas inter-rack communication depends on the351
relative position of the source and destination servers. There352
are three possible scenarios for inter-rack communication:353
• Both the source and destination servers are located in354
the same vertical plane (the same row or the same Y355
coordinate) as shown in Fig. 4(b). In this case a direct356
single hop link will be established between the source357
and destination for data transfer.358
• Both the source and destination servers are in the same359
column with same height above the ground (the same X360
and the same Z coordinates, but different Y coordinates).361
In this case a single hop direct link along a horizontal line362
will be established between the source and destination for363
communication as shown in Fig. 4(c).364
• The source and destination servers are in different row365
and column (different X and Y coordinates, may or may366
not have the same Z coordinate). In this case a 2-hop link367
will be established for communication using an interme-368
diate server as shown in Fig. 4(d). The intermediate server369
is the one that is in the same column and height from the370
source server, but in the row of the final destination (the371
same X and Z coordinates as that of the source and the372
same Y coordinate as the destination). As the data trav-373
els along the horizontal line first, the adopted routing374
protocol is referred to as Horizontal-First routing. In the375
proposed topology, every server is capable of working as376
a potential intermediate node.377
The pseudocode of the Horizontal-First routing strategy for378
these various conditions is shown in Fig. 5. Control infor-379
mation in the form of a control packet with instructions for380
intermediate and destination servers to steer their antennas381
Fig. 6. Possible communication paths between servers if obstruction isdetected.
in the correct directions is sent over a separate IEEE 802.11 382
2.4/5 GHz ISM band. Each server is equipped with an IEEE 383
802.11 2.4/5 GHz transceiver. As the radiation pattern has 384
main lobes in both forward and backward directions, steer- 385
ing is not required for the horizontal linear communication 386
a shown in Fig. 2. For communications in the vertical planes, 387
the server, which is ready to send data, first sends a control 388
packet to the receiving server while simultaneously steer- 389
ing its antenna array towards the receiver. Upon receipt of 390
this control message, wireless module at the receiver chooses 391
the antenna array in the correct sector out of the set of 392
6 and steers that array towards the sending server by acti- 393
vating the correct phase differences (paths connecting the 394
elements). The IEEE 802.11 2.4/5 GHz ISM band is also used 395
for sending the acknowledgments to enable the CSMA-based 396
MAC for the 60GHz links using the IEEE802.11ad proto- 397
cols. In order to provide access to the Internet with necessary 398
bandwidth, we envision gateway functionalities to be hosted at 399
multiple server locations within the rectangular arrangement in 400
the wireless DCN. These gateways will therefore be connected 401
directly or indirectly, to all the servers and will also need to 402
run firewall and security functionalities as per the requirement 403
of the datacenter. 404
2) Obstruction-Avoidance Routing Mechanism: In some 405
scenarios, the LoS necessary for the Horizontal-First routing 406
can be obstructed. For example, when a human technician or 407
any other obstacle is in front of an aisle it can potentially 408
obstruct the horizontal server-to-server LoS communication 409
between all servers in the aligned racks as shown in Fig. 6. 410
This will not only affect servers of the rows directly adjacent 411
to the human obstruction but also servers in racks of all rows 412
that use those horizontal paths for inter-row communication. 413
We propose an Obstruction-Avoidance adaptive routing mech- 414
anism to address this failure model and to successfully route 415
traffic flows in presence of such obstructions between spe- 416
cific racks. In the adaptive routing, all servers start sending 417
packets following the default Horizontal-First routing strat- 418
egy outlined earlier. We utilize the CSMA acknowledgment 419
mechanism to detect a failed transmission after several trials 420
IEEE P
roof
6 IEEE TRANSACTIONS ON GREEN COMMUNICATIONS AND NETWORKING
Fig. 7. Pseudocode for routing protocol if obstruction is detected.
according to the IEEE 802.11ad MAC. Then a retransmis-421
sion is attempted again using the adaptive routing algorithm422
as described in Fig. 7.423
In this adaptive routing strategy, after detecting a failed424
transmission, the sender determines the route of the next trans-425
mission attempt. If the destination server is in another rack426
in the same row, the sender retransmits the flow using the427
default Horizontal-First routing algorithm. This is because428
the failed transmission did not happen because of the hori-429
zontal LoS obstruction from the technician as that LoS link430
was not used in the first transmission attempt. The transmis-431
sion happens over the back/front vertical plane, which is not432
obstructed by the failure model under consideration. However,433
if the destination is in another row, instead of adopting the434
Horizontal-First approach, a Vertical-First routing approach435
is adopted where, a server in the same row but a differ-436
ent rack is chosen at random and the path is established to437
that server using the back/front vertical plane. Control packets438
are sent over the IEEE 802.11 2.4/5GHz ISM control plane439
to establish the links using beam-steering. From that other440
server, again the default Horizontal-First routing is adopted to441
reach the final destination. Such a path is shown in Fig. 6.442
If the randomly chosen intermediate server for Obstruction-443
Avoidance routing is also obstructed by another technician,444
the Obstruction-Avoidance routing approach can be repeated445
again till the Horizontal-First routing is successful to transfer446
packets to the destination row. In this way, this adaptive rout-447
ing mechanism can be extended to an obstruction model with448
multiple technicians obstructing multiple racks in the datacen-449
ter. The performance of S2S-WiDCN in presence of such an450
obstruction will be degraded for the obstructed flows. In the451
Fig. 8. Top view of datacenter layout floor plan.
next section, we describe the performance of S2S-WiDCN in 452
presence of various flow traffic patterns and obstructions. 453
IV. MODELING, RESULTS AND ANALYSIS 454
In this section, we present our modeling, results and corre- 455
sponding analysis of S2S-WiDCN. We first demonstrate that 456
it can sustain comparable performance compared to that of 457
conventional DCNs with network-level simulations in terms 458
of communications between servers within a datacenter. Next, 459
we present the estimates of power consumption to highlight 460
the main benefit of S2S-WiDCN. 461
A. Simulation Platform 462
We use the Network Simulator-3 (NS-3) suite [35] to evalu- 463
ate S2S-WiDCN. NS-3 supports the characteristics of wireless 464
propagation as well as network-level communications. It is 465
important to simulate both the propagation and network-level 466
communication characteristics accurately in order to obtain 467
credible performance results. We use a modified version of 468
NS-3 extended with features of wireless datacenter including 469
the 60GHz band and the IEEE 802.11ad standard as discussed 470
in [11]. This extension incorporates interference modeling, bit 471
error rates, and directional antenna modeling. The accuracy of 472
these parameters is verified with physical layer measurements 473
of prototype 60GHz hardware [11]. Additionally, we introduce 474
criteria for wireless link selection to enable many concurrent 475
links and modify the IEEE 802.11ad physical layer to allow 476
multiple OFDM channels. 477
We have considered two datacenter sizes in this paper to 478
represent datacenters belonging to two different classes. The 479
first one, with a total of 800 servers, is a small-sized datacen- 480
ter representing those in an educational institution. The second 481
one is a mid-sized datacenter and has 1600 servers represent- 482
ing those in private enterprises [36], [37]. In both cases, the 483
servers are arranged in a 20×8 array of racks as shown in 484
Fig. 8. There are 10 racks arranged in a single row and two 485
columns of 8 rows, totaling 160 racks. Each rack occupies an 486
IEEE P
roof
MAMUN et al.: PERFORMANCE EVALUATION OF POWER-EFFICIENT AND ROBUST 60 GHz WIRELESS S2S DCN 7
area of 0.6m×0.9m and is 2m high. Adjacent rows are sepa-487
rated by 1m and the width of the central aisle is 2m. Each rack488
contains 5 and 10 servers for the 800 and 1600 server data-489
centers respectively. In our simulations, the racks are assumed490
to be without any front or back side door.491
To account for the latency required to set up the 60GHz492
communication links using the exchange of control informa-493
tion over the IEEE 802.11 2.4/5 GHz ISM band and beam-494
steering, we run a conservative simulation using NS-3 with the495
packet size of 200 bytes representing control packets of 60GHz496
S2S-WiDCN datacenter with beam-steering information. Each497
new flow according to the flow arrival process discussed in498
Section IV-B (1) in the DCN is considered to generate a con-499
trol packet. The flow arrival process is considered to be the500
same as described in Section IV-B (1). The simulation showed501
that a single wireless access-point can sustain the demand for502
the control packet traffic with an average latency of 266µs in503
a system with 240 servers. As the maximum number of servers504
in a single vertical plane is 200, a single access point per505
row should be enough to server the requirement. This latency506
overhead is considered in the evaluation of S2S-WiDCN507
next.508
B. Performance Evaluation and Analysis509
Here we present the simulation results of S2S-WiDCN510
along with a comparative analysis with respect to existing511
DCNs in terms of flow completion duration and through-512
put. The throughput is defined as the average rate of bit513
transferred per second over the DCN. We compare S2S-514
WiDCN with a traditional wired fat-tree based DCN and515
a ToR-ToR wireless DCN. In the traditional wired fat-tree516
based DCN, we have considered 3 hierarchical layers consist-517
ing of 160 access, 2 aggregate and 2 core layer switches similar518
to the architecture evaluated in [22]. Each traditional DCN519
link between the access, aggregate and core level switches520
is considered to have a channel bandwidth of 10Gbps. The521
intra-rack communication in the fat-tree based DCN occurs522
through the ToR switch, which has 1.0Gbps direct links to523
each server in its rack. Although the proposed wireless DCN524
is a direct server-to-server network, we have not compared525
it with a wired server-to-server all-to-all DCN because such526
a network is not practical and will have extremely high degree527
of connectivity at each server. In the ToR-to-ToR wireless528
DCN (ToR-WiDCN) the intra-rack communication is man-529
aged in a traditional manner, same as in the conventional wired530
DCN. The inter-rack communication is done with ToR-to-ToR531
60GHz wireless links using the same physical and MAC layers532
as in S2S-WiDCN. We use the simulation platform described533
in Section IV-A and the datacenter traffic model discussed534
below to evaluate these DCNs.535
1) Datacenter Traffic Model: S2S-WiDCN is at first evalu-536
ated with a set of traffic flows based on application demands.537
These demands reflect real traffic within the network over538
a period of time. The application demands include informa-539
tion specifying the flow arrival time, identity of the source540
and destination, the flow size, and the data rate at which the541
traffic is generated. Real datacenter traffic for different classes542
TABLE IPARAMETERS FOR INDEX/QUERY BASED TRAFFIC GENERATION
Fig. 9. CDF of flow transmission rates of the (a) index/query based trafficfor small size (b) index/query based traffic for medium size (c) multimediatraffic for small size DCN.
of datacenters such as educational (small), private (medium) 543
and corporate (large) running typical query/response based 544
applications like map-reduce and index-search are measured 545
in [36]. Using these measured traffic flows, a Poisson shot- 546
noise based model to synthesize datacenter traffic is proposed 547
and verified in [38]. According to [38], the new flow arrival 548
time, the flow duration and the injection rate for each appli- 549
cation follow a Poisson, Pareto and, Gaussian distribution 550
respectively. The new flow arrival time is generated using 551
a Poisson distribution with an average flow arrival rate. The 552
average flow arrival rate is considered to be 1000 flows/second 553
for the small-sized and 3000 flows/second for the mid-sized 554
datacenter [36]. The flow injection rate and the flow dura- 555
tion are independent parameters. The flow duration refers to 556
the time required to inject the flow into the DCN and is 557
different from the flow completion duration which is a per- 558
formance metric. The flow size is a product of the injection 559
rate and duration and therefore depends on both. In our evalu- 560
ations, we have considered a skewed Gaussian distribution for 561
the injection rate to have a mean of 1.0kBps such that 90% 562
of the traffic rates are less than 10kBps and the remaining 563
10% can be as high as several MBps as per the traffic model 564
from [38]. Application flow duration is generated following 565
an independent Pareto distribution having a minimum dura- 566
tion of 10 microseconds [36]. The characteristic parameters 567
for these distributions are summarized in Table I. As custom- 568
ary for TCP traffic, we have considered the size of all packets 569
in the generated flows equal to the maximum transmission unit 570
(1500 bytes). The CDFs of the transmission rates of both of 571
these two traffics are shown in Fig. 9 (a) and Fig. 9 (b). 572
According to [36] and [38], in a DCN, around 80% of the 573
total traffic stays within the same rack. Only 20% commu- 574
nication takes place between the servers situated in different 575
racks. For each new flow in our simulations, a random desti- 576
nation was chosen such that 80% of the destinations belonged 577
to the same rack as the flow source. The simulations were con- 578
ducted such that no new flows were allowed to be injected after 579
IEEE P
roof
8 IEEE TRANSACTIONS ON GREEN COMMUNICATIONS AND NETWORKING
Fig. 10. Average flow completion duration for index/query based traffic.
20 seconds but the simulations were run until the completion580
of all the established flows. Next, we analyze the performance581
of the DCNs in the presence of this traffic pattern.582
2) Flow Completion Duration: Here, we estimate the flow583
completion duration of the applications in the different DCNs.584
The flow completion duration for both small and medium sizes585
of all 3 different DCNs are shown in Fig. 10. It can be seen586
that the average completion duration of S2S-WiDCN is lower587
than that of the wired network for both sizes because of fewer588
number of hops involved, resulting in lower time of flight589
and switching overheads. For the wired network, two servers590
even in a single rack need to go through the access layer591
switch to communicate, requiring at least two hops. On the592
other hand, in the wireless architecture, those two servers can593
communicate directly using a single hop. As the major por-594
tion of communications in datacenters is intra-rack [38], the595
reduction in delay of intra-rack flows in the wireless DCN is596
likely to reduce the overall average flow completion duration.597
The beam-steering latency is 266µs for exchange of control598
information over the control plane is considered while com-599
puting the flow completion duration of S2S-WiDCN as shown600
in Fig. 10.601
Fig. 10 also captures the minimum and maximum flow602
completion durations for all the DCNs. The minimum flow603
completion duration in case of S2S-WiDCN is higher than that604
of the fat-tree based DCN due to the beam-steering latency.605
This effect is also seen in the ToR-WiDCN. However, as seen606
in [38] only a very small fraction of the flows have such short607
flow durations.608
3) Throughput: Fig. 11 shows the average throughput along609
with the minimum and maximum for all flows in each610
DCN. As we can see for both sizes S2S-WiDCN provides the611
same throughput as that of the wired traditional fat-tree based612
DCN. Even the ToR-WiDCN is capable of achieving a sim-613
ilar performance. All the DCNs achieve throughputs which614
closely match the injection rate of the flows. This is because,615
in the traditional wired network, the available bandwidth per616
link is 1.0Gbps and that available for OFDM wireless chan-617
nels is 0.563Gbps. We have considered 3 sub-carriers in the618
60GHz band each with maximum OFDM rates of 6.76Gbps,619
which are in turn, split into 12 sub-channels each, to cater to620
Fig. 11. Average throughput of different datacenter networks for index/querybased traffic.
all the application flows injected into the wireless DCNs. So 621
the physical bandwidths of both wired and wireless channels 622
are much higher than the average injection rates encountered 623
in these scenarios. Moreover, we find that in S2S-WiDCN the 624
throughput is higher than that in both the traditional wired 625
DCN and ToR-WiDCN. This is because the lower number 626
of hops in S2S-WiDCN implies that the flow will encounter 627
fewer intermediate nodes resulting in a reduced likelihood of 628
being congested. Therefore, for the type of application consid- 629
ered here, S2S-WiDCN performs better than the fat-tree based 630
DCN. Although the flow arrival rate increases with the num- 631
ber of servers, its impact on performance is marginal as the 632
flow transmission rates of most of the flows are less than the 633
60GHz OFDM channel capacity. 634
4) Evaluation With Different Traffic Patterns: In this sub- 635
section, we further evaluate S2S-WiDCN with a different 636
traffic scenario that can be encountered in multimedia or 637
video hosting/streaming servers. This kind of application is 638
significantly different from query/index search applications 639
primarily from two perspectives. First, this kind of traffic gen- 640
erally has a higher data rate requirement. Multimedia/video 641
streaming servers hosting applications are seen to have average 642
data rates of 100Mbps [39]. Second, these applications typi- 643
cally experience bursty flow arrivals [40]. In order to evaluate 644
S2S-WiDCN with multimedia/video, traffic we adopt a bursty 645
flow arrival rate with a high average data rate of 100Mbps 646
based on [39]. Unlike the query based traffic where the flow 647
arrival process is assumed to be a Poisson process, the bursty 648
flow arrival is modeled as a fractal process. The entire sim- 649
ulation duration is divided into windows of 30ms and each 650
window is randomly chosen to be either in ON or OFF phase. 651
New flow arrivals are allowed only in the ON phase. The new 652
flows have an arrival rate such that the overall average flow 653
arrival rate is the same as that of the query-based traffic for 654
the simulation duration. The details of this traffic are listed in 655
Table II. The CDF of number of concurrent flow arrivals within 656
a window size of 30ms for both the query-based traffic and the 657
bursty multi-media traffic is shown in Fig. 12. It can be seen 658
that in the Poisson arrival process typical in query/response 659
applications, the number of concurrent flows is never higher 660
IEEE P
roof
MAMUN et al.: PERFORMANCE EVALUATION OF POWER-EFFICIENT AND ROBUST 60 GHz WIRELESS S2S DCN 9
TABLE IIPARAMETERS FOR VIDEO/MULTIMEDIA TRAFFIC GENERATION
(a) (b)
Fig. 12. Distribution of number of concurrent connections for (a) query-basedtraffic (b) bursty multimedia traffic.
than 50 whereas, it can be as high as 250 in the bursty flow661
arrival pattern. The bursty traffic pattern coupled with the high662
flow rate requirements of this traffic type is therefore, expected663
to stress the DCN more compared to the query/response type664
traffic.665
Fig. 13 shows the average flow completion duration and666
average throughput of a small sized DCN with 800 servers667
for this multimedia/video traffic. We have compared the per-668
formance of S2S-WiDCN with a fat-tree based wired DCN669
with this traffic. Similar to query-based traffic, a few small670
flows with very low flow durations incur a higher flow comple-671
tion duration in S2S-WiDCN as can be seen in the minimum672
of the flow duration range in Fig. 13 (a). This is because673
the beam-steering latency of S2S-WiDCN is higher than the674
flow duration of these very small flows. Moreover, we can see675
that some of the high data rate flows achieve lower through-676
puts in S2S-WiDCN compared to the fat-tree wired DCN as677
shown by the maximum value of the range of throughput of678
S2S-WiDCN in Fig. 13 (b). This is because the maximum679
data rate per OFDM channel that can be supported in S2S-680
WiDCN is 0.563Gbps. As the flow rates follow a Gaussian681
distribution with a mean of 100Mbps, a few flows require682
a data rate higher than 0.563Gbps. The effective throughput683
of these flows are reduced in S2S-WiDCN. However, as can684
be seen from the CDF of flow rates in Fig. 9(c), these flows685
with data rates higher than 0.563Gbps are few in number and686
therefore do not affect the average throughput significantly.687
5) Evaluation in the Presence of LoS Obstruction: In this688
subsection, we evaluate the performance of S2S-WiDCN in the689
presence of an LoS obstruction. For this purpose, a scenario690
is assumed where an IT technician is present in front of the691
column 3 of row 2 in the floorplan shown in Fig. 8. We have692
assumed this obstruction to be stationary within the observed693
window of 20s, which is reasonable as it is a human obstruc-694
tion. Due to the presence of the obstruction, the traffic flows695
(a) (b)
Fig. 13. (a) Average flow completion duration and (b) average throughputof a small sized DCN with 800 servers for this multimedia/video traffic fordifferent datacenter networks.
Fig. 14. Comparison of average flow completion duration in presence ofLoS obstruction.
from all servers in all rows corresponding to the obstructed col- 696
umn, which were supposed to be routed through the horizontal 697
lines in column 3 with the default Horizontal-First routing 698
protocol, need to follow the alternate Obstruction-Avoidance 699
routing mechanism. 700
Fig. 14 shows the flow completion duration of the small- 701
scale S2S-WiDCN with 800 servers with adaptive routing, in 702
the presence of the obstruction. We have evaluated the impact 703
of the obstruction on the overall flow completion duration 704
as well as that of the affected traffic flows only. As 80% 705
of the traffic generated from each server is intra-rack, they 706
use the vertical plane for communication and are unaffected 707
by the presence of a technician. Among the 20% inter-rack 708
traffic, a smaller percentage requires inter-row paths going in 709
the horizontal direction as inter-rack traffic in the same row 710
also uses the vertical plane. The percentage of traffic flows 711
from all rows, whose inter-row traffic is obstructed, is 1.87% 712
of all the flows in S2S-WiDCN. Hence, the LoS obstruction 713
has very little impact on the overall average flow comple- 714
tion duration due to the small percentage of traffic flows that 715
are affected. We have further investigated the impact of the 716
obstruction as a function of the number of retransmission tri- 717
als made before rerouting using the Vertical-First path of the 718
adaptive routing method. As can be seen, in case of a higher 719
number of allowed retransmission attempts before rerouting 720
the flow, the impact of the obstruction is higher. Hence, the 721
number of retransmission attempts can be customized based 722
on the performance demands of the applications. 723
IEEE P
roof
10 IEEE TRANSACTIONS ON GREEN COMMUNICATIONS AND NETWORKING
C. Power Consumption Analysis724
From the previous section it is seen that S2S-WiDCN can725
sustain comparable performance as that of a traditional fat-tree726
DCN. In this section we evaluate its most important benefit,727
which is the reduction in power consumption of S2S-WiDCN728
with respect to the traditional DCN. We discuss the model729
and parameters used in the power estimation followed by the730
results.731
1) Power Model: It is a complex task to estimate the actual732
electrical power consumed by a datacenter. The power con-733
sumption depends on several internal factors such as utilization734
of computing power, the cooling mechanism, and datacenter735
networks. Datacenter power consumption is also affected by736
external parameters like the geographical location, weather,737
temperature, and humidity. Our focus is solely on networking,738
and we only analyze the power consumption involved in net-739
working. In this regard, we assume that the power consumption740
other than networking is identical in all the cases. We esti-741
mate power consumption for wired DCNs using commercially742
available data from Cisco network switches [41], [42], [43].743
Specifically, we use Cisco 7702 for the core-level switches,744
Cisco 9508 at the aggregation level, and Cisco 9372 for745
access-level switches. We also use the data from Silicom746
PE2G2135 for the power consumption of the network inter-747
face cards (NIC) [44]. The power consumption of each core748
and aggregate switches are as follows:749
PCore = PI/O Cards + PFan Tray + PSv (1)750
PAgg = PI/OCards + PFan Tray + PSv + PFabric751
+ PSys Ctrl (2)752
where PI/O Cards ,PFan Tray ,PSv ,PFabric ,PSys Ctrl repre-753
sent the power consumption of the input/output card, fanout754
ports, supervisor controller, cables and system controller755
respectively.756
Then the total power is calculated as follows:757
PTotal = NCorePCore + NAggPAgg+ NAccPAcc758
+ NSPNIC (3)759
where, NCore , NAgg , NAcc , NS are the number of core,760
aggregation, access switches, and the total number of servers,761
respectively; PCore , PAgg , PAcc , PNIC are the power con-762
sumptions of an individual core, aggregation, access switches763
and network interface cards, respectively.764
In S2S-WiDCN, however, no core, aggregate or access layer765
switches are needed, but only antennas, transceivers and NICs766
are required for wireless communication. The power con-767
sumption of the wireless 60GHz transceiver is modeled based768
upon the assessment of emerging 60GHz transceivers such769
as [8] and [9]. The NICs of S2S-WiDCN are equipped with770
two transceivers for horizontal and vertical communication.771
In the traditional DCN, external connections are established772
via the two Cisco7702 switches. To provide equivalent con-773
nectivity in S2S-WiDCN, we employ two servers to work774
as gateways, and their power consumption is modeled as775
that of the Cisco 7702 switch. The power consumption of776
TABLE IIIPOWER CONSUMPTION OF DIFFERENT DCN COMPONENTS
Fig. 15. Total power consumption of various DCN architectures.
communication per server in S2S-WiDCN is calculated as: 777
PWireless = P60GHz Tran + PAntenna + PWifiControl 778
+ PNIC (4) 779
where P60GHz Tran is the power consumption of the 60GHz 780
transceiver, PWifiControl is the power consumption of the 781
802.11 2.4/5 GHz ISM adapter for the control channel, and 782
PAntenna is the power consumed by the antennas for beam- 783
steering. We conservatively adopt P60GHz Tran + PAntenna 784
to be 1.7W [8], [9]. We consider PWifiControl to be 220mW 785
from the datasheet of D-link DWA-171 2.4GHz ISM adapter. 786
Finally, the total power consumption in S2S-WiDCN becomes: 787
PTotal WiDCN = NCorePCore + NSPWireless (5) 788
The power consumption of all the off-the-shelf switching 789
components used in our model is shown in Table III. 790
2) Comparative Analysis of Power Consumption: We posit 791
that the primary advantage of S2S-WiDCN is lower power 792
consumption. To study this more deeply, the total power con- 793
sumption estimated for the typical and maximum cases for 794
all the DCNs is shown in Fig. 15. In the “typical” sce- 795
nario, the average power consumption of every device is used, 796
while their maximum power consumption is considered in the 797
IEEE P
roof
MAMUN et al.: PERFORMANCE EVALUATION OF POWER-EFFICIENT AND ROBUST 60 GHz WIRELESS S2S DCN 11
(a) (b)
Fig. 16. Separation between two adjacent transceivers (a) in horizontal lines(b) in vertical planes.
“maximum” scenario. For small-sized and mid-sized DCNs,798
the result shows eight-fold and five-fold reduction in power799
consumption of S2S-WiDCN compared to the traditional800
fat-tree based DCN topology in the “typical” case. The max-801
imum improvement in power consumption is observed to be802
seventeen-fold for the small-sized DCN in the “maximum” uti-803
lization scenario. The complete elimination of power-hungry804
aggregate and access-layer switches contribute to this drastic805
reduction primarily. Since access-level switches are needed per806
rack in the ToR-WiDCN, its reduction in power consumption807
is not as significant as that of S2S-WiDCN. Therefore, by808
establishing direct links between servers S2S-WiDCN reduces809
the power consumption significantly compared to all the DCNs810
that require higher level switches.811
D. Estimate of the Overheads812
Vertically adjacent servers need to have space between them813
to accommodate the antenna arrays at the top of the servers.814
With the compact size of the antenna of only 10.5mm×3.3mm,815
we envision a separation of 30mm between the servers should816
be enough. As the typical height of a server is 90mm, we817
can reduce the vertical server density to be around 33%. In818
other words, this reduction in server density can increase in819
rack height by 33% to accommodate the same number of820
servers per rack. We anticipate that, this does not have signif-821
icant impact on infrastructure cost. In fact, this type of server822
arrangement will aid in cooling by enabling better chilled air823
circulation around each server.824
As a narrow LoS exists between the servers communicating825
along the same horizontal line, the possibility of interference826
with adjacent receivers decreases as the top and bottom server827
structures do not allow the antenna radiation lobe to reach828
other vertically adjacent receivers as depicted in Fig. 16 (a).829
While multipath transmission may still be caused by diffrac-830
tion around rack structures even though non-reflective coating831
is used on all reflective surfaces, the narrow aperture to estab-832
lish the LoS between the antennas makes it unlikely that833
the multipath non-LoS signals will have significant power.834
Similarly, the wireless communication in a vertical plane also835
does not interfere with the wireless communication with its836
adjacent vertical plane as row of racks will act as a shield837
against the wireless signal of one plane interfering with a dif-838
ferent plane as shown in Fig. 16 (b). Moreover, the half-angle839
of the main radiation lobe is narrow enough to prevent the840
transmission from one vertical plane in reaching another.841
However, in both cases, receivers in the LoS of an active com- 842
munication cannot use the same channel to receive data from 843
another sender. A different OFDM channel is used in such 844
a case to avoid interference. 845
V. CONCLUSION AND FUTURE WORK 846
The challenges in current DCN’s are high design and 847
maintenance cost, huge power consumption, high cabling 848
complexity, hard to keep accurate per-cable information and 849
inefficient cooling. Structured cabling bundle incur significant 850
initial effort and cost to setup and still may cause airflow 851
blockage. All these challenges can be overcome by using the 852
proposed completely wireless server-to-server DCN architec- 853
ture. We observe that S2S-WiDCN provides comparable flow 854
completion duration and throughput to a conventional fat-tree 855
based DCN for query/response and multimedia/video based 856
applications, while reducing the power consumption by five 857
to seventeen times. 858
REFERENCES 859
[1] A. Shehabi et al., “United States data center energy usage 860
report,” Ernest Orlando Lawrence Berkeley Nat. Lab., Berkeley, 861
CA, USA, Rep. LBNL-1005775, 2016. [Online]. Available: 862
https://eta.lbl.gov/sites/default/files/publications/lbnl-1005775_v2.pdf 863
[2] D. Abts, M. R. Marty, P. M. Wells, P. Klausler, and H. Liu, “Energy 864
proportional datacenter networks,” SIGARCH Comput. Archit. News, 865
vol. 38, no. 3, pp. 338–347, Jun. 2010. 866
[3] M. Chen, H. Jin, Y. Wen, and V. C. M. Leung, “Enabling technologies 867
for future data center networking: A primer,” IEEE Netw., vol. 27, no. 4, 868
pp. 8–15, Jul./Aug. 2013. 869
[4] M. Pedram, “Energy-efficient datacenters,” IEEE Trans. Comput.-Aided 870
Design Integr. Circuits Syst., vol. 31, no. 10, pp. 1465–1484, Oct. 2012. 871
[5] IEEE Standards Association, IEEE Standard for Information 872
Technology–Telecommunications and Information Exchange Between 873
Systems–Local and Metropolitan Area Networks–Specific Requirements- 874
Part 11: Wireless LAN Medium Access Control (MAC) and Physical 875
Layer (PHY) Specifications Amendment 3: Enhancements for Very 876
High Throughput in the 60 GHz Band, IEEE Standard 802.11ad-2012, 877
Dec. 2012. 878
[6] E. Baccour, S. Foufou, R. Hamila, and M. Hamdi, “A survey of wireless 879
data center networks,” in Proc. Annu. Conf. Inf. Sci. Syst. (CISS), 2015, 880
Baltimore, MD, USA, pp. 1–6. 881
[7] K. Ranachandran, R. Kokku, R. Mahindra, and S. Rangarajan, “60GHz 882
datacenter networking: Wireless => worryless?” NEC, Tokyo, Japan, 883
Rep., 2008. AQ2884
[8] Analog Devices HMC6300 60 GHz Millimeterwave Transmitter 885
Datasheet. Accessed: Jan. 23, 2018. [Online]. Available: 886
http://www.analog.com/media/en/technical-documentation/data- 887
sheets/HMC6300.pdf 888
[9] Analog Devices HMC6301 60 GHz Millimeterwave Receiver 889
Datasheet. Accessed: Jan. 23, 2018. [Online]. Available: 890
http://www.analog.com/media/en/technical-documentation/data- 891
sheets/HMC6301.pdf 892
[10] B. Floyd et al., “A silicon 60GHz receiver and transmitter chipset for 893
broadband communications,” in IEEE Int. SolidState Circuits Conf. Dig. 894
Tech. Papers (ISSCC), San Francisco, CA, USA, 2006, pp. 649–658. 895
[11] D. Halperin, S. Kandula, J. Padhye, P. Bahl, and D. Wetherall, 896
“Augmenting data center networks with multi-gigabit wireless links,” 897
ACM SIGCOMM Comput. Commun. Rev., vol. 41, no. 4, pp. 38–49, 898
2011. 899
[12] M. Kyrö et al., “5×1 linear antenna array for 60 GHz beam steering 900
applications,” in Proc. 5th Eur. Conf. Antennas Propag. (EuCAP), IEEE, 901
2011, pp. 1258–1262. 902
[13] S. G. Umamaheswaran, S. A. Mamun, A. Ganguly, M. Kwon, and 903
A. Kwasinski, “Reducing power consumption of datacenter networks 904
with 60GHz wireless server-to-server links,” in Proc. GLOBECOM IEEE 905
Glob. Commun. Conf., Singapore, 2017, pp. 1–7. 906
[14] C. E. Leiserson, “Fat-trees: Universal networks for hardware-efficient 907
supercomputing,” IEEE Trans. Comput., vol. C-34, no. 10, pp. 892–901, 908
Oct. 1985. 909
[15] C. Guo et al., “BCube: A high performance, server-centric network 910
architecture for modular data centers,” in Proc. ACM SIGCOMM, 911
vol. 39, 2009, pp. 63–74. 912
[16] C. Guo et al., “DCell: A scalable and fault-tolerant network structure 913
for data centers,” ACM SIGCOMM Comput. Commun. Rev. , vol. 38, 914
no. 4, pp. 75–86, 2008. 915
IEEE P
roof
12 IEEE TRANSACTIONS ON GREEN COMMUNICATIONS AND NETWORKING
[17] X. Ye et al., “DOS: A scalable optical switch for datacenters,” in Proc.916
ACM/IEEE Symp. Archit. Netw. Commun. Syst. (ANCS), La Jolla, CA,917
USA, 2010, pp. 1–12.918
[18] A. Greenberg et al., “VL2: A scalable and flexible data center network,”919
in Proc. ACM SIGCOMM, vol. 39, 2009, pp. 51–62.920
[19] N. Hamedazimi et al., “FireFly: A reconfigurable wireless data cen-921
ter fabric using free-space optics,” SIGCOMM Comput. Commun. Rev.,922
vol. 44, no. 4, pp. 319–330, 2014.923
[20] N. Farrington et al., “Helios: A hybrid electronic/optical switch archi-924
tecture for modular data centers,” in Proc. ACM SIGCOMM, vol. 40,925
pp. 339–350, Oct. 2010.926
[21] C. Kachris and I. Tomkos, “A survey on optical interconnects for data927
centers,” IEEE Commun. Surveys Tuts., vol. 14, no. 4, pp. 1021–1036,928
4th Quart., 2012.929
[22] J.-Y. Shin, E. G. Sirer, H. Weatherspoon, and D. Kirovski, “On the930
feasibility of completely wireless datacenters,” IEEE/ACM Trans. Netw.,931
vol. 21, no. 5, pp. 1666–1679, Oct. 2013.932
[23] M. Z. Zaaimia, R. Touhami, V. A. Fono, L. Talbi, and M. Nedil,933
“60 GHz wireless data center channel measurements: Initial results,” in934
Proc. IEEE Int. Conf. Ultra WideBand (IC UWB), Paris, France, 2014,935
pp. 57–61.936
[24] W. Zhang et al., “3D beamforming for wireless data centers,” in937
Proc. 10th ACM Workshop Hot Topics Netw., Cambridge, MA, USA,938
2011, p. 4.939
[25] S. A. Mamun et al., “An energy-efficient, wireless top-of-rack to top-940
of-rack datacenter network using 60GHz links,” in Proc. IEEE Green941
Comput. Commun. (GreenCom), Exeter, U.K., 2017, pp. 458–465.942
[26] X. Zhou et al., “Mirror mirror on the ceiling: Flexible wireless links943
for data centers,” in Proc. ACM SIGCOMM Comput. Commun. Rev.,944
vol. 42. Helsinki, Finland, 2012, pp. 443–454.945
[27] J. Y. Shin, D. Kirovski, and D. T. Harper, “Data center using wireless946
communication,” U.S. Patent 15/164 635, May 2016.947
[28] H. Vardhan, S.-R. Ryu, B. Banerjee, and R. Prakash, “60GHz wireless948
links in data center networks,” Comput. Netw., vol. 58, pp. 192–205,949
Jan. 2014.950
[29] J. Niemann, K. Brown, and V. Avelar, “Impact of hot and cold aisle951
containment on data center temperature and efficiency,” Schneider Elect.952
Data Center Sci. Center, White Paper, pp. 1–14, 2011.AQ3 953
[30] T. Matsumara et al., “Millimeter-wave broadband anti-reflection coat-954
ings using laser ablation of sub-wavelength structures,” Appl. Opt.,955
vol. 55, no. 13, pp. 3502–3509, Apr. 2016. [Online]. Available:956
https://arxiv.org/pdf/1601.08246957
[31] L. M. Correia and P. O. Frances, “Transmission and isolation of signals958
in buildings at 60 GHz,” in Proc. 6th Int. Symp. Personal Indoor Mobile959
Radio Commun., Toronto, ON, Canada, 1995, pp. 1031–1034.960
[32] SRTHERMDUCT: SmartRack Thermal Duct Kit for SmartRack961
Enclosures. Accessed: Apr. 1, 2018. [Online]. Available:962
https://www.tripplite.com/smartrack-thermal-duct-kit-for-smartrack-963
enclosures∼SRTHERMDUCT964
[33] ASHRAE TC 9.9 Thermal Guidelines for Data Processing Enviroments,965
ASHRAE Tech. Committee, 2011.966
[34] D. S. Makarov, M. Y. Tretyakov, and P. W. Rosenkranz, “60-GHz oxygen967
band: Precise experimental profiles and extended absorption modeling968
in a wide temperature range,” J. Quant. Spectroscopy Radiative Transf.,969
vol. 112, no. 9, pp. 1420–1428, Jun. 2011.970
[35] Network Simulator-3 (NS-3). Accessed: Jan. 23, 2018. [Online].971
Available: https://www.nsnam.org/972
[36] T. Benson, A. Akella, and D. A. Maltz, “Network traffic characteris-973
tics of data centers in the wild,” in Proc. 10th ACM SIGCOMM Conf.974
Internet Meas., New York, NY, USA: ACM, 2010, pp. 267–280.975
[37] “Managing variable data center rack densities,” Reno, NV, USA, Server976
Technol., White Paper, Mar. 2015.977
[38] Y. Han, J.-H. Yoo, and J. W.-K. Hong, “Poisson shot-noise process based978
flow-level traffic matrix generation for data center networks,” in Proc.979
IFIP/IEEE Int. Symp. Integr. Netw. Manag. (IM), Ottawa, ON, Canada,980
2015, pp. 450–457, doi: 10.1109/INM.2015.7140322.981
[39] A. Rao et al., “Network characteristics of video streaming traffic,” in982
Proc. 7th Conf. Emerg. Netw. Exp. Technol. (CoNEXT), New York, NY,983
USA: ACM, 2011, p. 12.984
[40] M. Alizadeh et al., “Data center TCP (DCTCP),” in Proc. ACM985
SIGCOMM Conf. (SIGCOMM), New York, NY, USA: ACM, 2010,986
pp. 63–74.987
[41] Cisco Nexus 7702 Hardware Installation Guide. Accessed: Nov. 27,988
2016. [Online]. Available: http://www.cisco.com/c/en/us/td/docs/989
switches/datacenter/hw/nexus7000/installation/guide/b_n7702_hardware990
_install_guide.pdf991
[42] Cisco Nexus 9508 NX-OS Mode Switch Hardware Installation Guide.992
Accessed: Nov. 27, 2017. [Online]. Available: http://www.cisco.com/c/993
en/us/td/docs/switches/datacenter/nexus9000/hw/n9508_hig/guide/994
b_n9508_nxos-mode_hardware_install_guide.pdf995
[43] Cisco Nexus 9372PX and 9372PX-E NX-OS Mode Switches Hardware996
Installation Guide. Accessed: Nov. 27, 2017. [Online]. Available:997
http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/998
hw/n9372px/guide/b_n9372PX_hardware_install_guide.pdf999
[44] Silicom PE2G2I35 Datasheet. Accessed: Aug. 27, 2017. [Online]. 1000
Available: http://www.silicom-usa.com/wp-content/uploads/2016/08/PE 1001
2G2I35-1G-Server-Adapter.pdf 1002
Sayed Ashraf Mamun (S’XX) AQ4received the B.Sc. 1003
and M.Sc. Engineering degrees from the Department 1004
of Electrical and Electronic Engineering, Bangladesh 1005
University of Engineering and Technology, Dhaka. 1006
He is currently pursuing the Ph.D. degree with 1007
the GCCIS Department, Rochester Institute of 1008
Technology. His current research interests include 1009
wireless networks and communications. 1010
Sree Gowrishankar Umamaheswaran 1011
(S’XX) received the B.S. degree from the Sri 1012
Venkateswara College of Engineering and the M.S. 1013
degree from the Computer Engineering Department, 1014
Rochester Institute of Technology. His research 1015
interests include wired and wireless networking. 1016
Amlan Ganguly (M’XX) received the M.S. and 1017
Ph.D. degrees from the School of Electrical 1018
Engineering and Computer Science, Washington 1019
State University, Pullman, and the B.Tech. (Hons.) 1020
degree in electronics and electrical communica- 1021
tion engineering from the Indian Institute of 1022
Technology, Kharagpur, Kharagpur, India. He is cur- 1023
rently an Associate Professor with the Department 1024
of Computer Engineering, Rochester Institute of 1025
Technology, USA. His research interests include 1026
design of energy-efficeint interconnection infrastruc- 1027
tures for multicore platforms and datacenter networks. 1028
Minseok Kwon received the B.S. degree in com- 1029
puter science and statistics from Seoul National 1030
University, Seoul, South Korea, in 1994, and the 1031
M.S. and Ph.D. degrees in computer science from 1032
Purdue University in 2004 and 2001, respec- 1033
tively. He is currently an Associate Professor with 1034
the Department of Computer Science, Rochester 1035
Institute of Technology, Rochester, NY, USA. His 1036
research interests are in the area of computer net- 1037
works, distributed computing, and network security. 1038
Andres Kwasinski (S’98–M’04–SM’11) received 1039
the Diploma degree in electrical engineering from 1040
the Buenos Aires Institute of Technology, Buenos 1041
Aires, Argentina, and the M.S. and Ph.D. degrees 1042
in electrical and computer engineering from the 1043
University of Maryland, College Park, MD, USA, 1044
in 2000 and 2004, respectively. He is currently 1045
a Professor with the Department of Computer 1046
Engineering, Rochester Institute of Technology, 1047
Rochester, NY, USA. His current areas of research 1048
include cognitive radios and wireless networks, 1049
cross-layer techniques in wireless communications, smart infrastructures and 1050
networking and operation of the Internet-of-Things. 1051
IEEE P
roof
AUTHOR QUERIESAUTHOR PLEASE ANSWER ALL QUERIES
PLEASE NOTE: We cannot accept new source files as corrections for your paper. If possible, please annotate the PDFproof we have sent you with your corrections and upload it via the Author Gateway. Alternatively, you may send usyour corrections in list format. You may also upload revised graphics via the Author Gateway.
AQ1: Note that if you require corrections/changes to tables or figures, you must supply the revised files, as these items arenot edited for you.
AQ2: Please provide the report number for Reference [7].AQ3: Please provide the organization location for References [29] and [33].AQ4: Please provide the missing IEEE membership years for the authors “S. A. Mamun,” “S. G. Umamaheswaran,” and
“A. Ganguly.”
