Embed Size (px)
Citation preview
AT
M a
nd F
ast E
ther
net N
Is f
or U
ser-
Lev
el C
omm
unic
atio
n
AT
M a
nd F
ast E
ther
net
Net
wor
k In
terf
aces
for
Use
r-L
evel
Com
mun
icat
ion
Mat
t W
elsh
Dep
artm
ent
of
Co
mp
ute
r S
cien
ceC
orn
ell U
niv
ersi
tym
dw
@cs
.co
rnel
l.ed
u, h
ttp
://w
ww
.cs.
corn
ell.e
du
/Info
/Pro
ject
s/U
-Net
/
Join
t w
ork
wit
h A
nin
dya
Bas
u a
nd
Th
ors
ten
vo
n E
icke
n
Page
1H
PCA
-3, S
an A
nton
io, 5
Feb
ruar
y 19
97M
att W
elsh
, Cor
nell
Uni
vers
ity
AT
M a
nd F
ast E
ther
net N
Is f
or U
ser-
Lev
el C
omm
unic
atio
n
Use
r-le
vel
Net
wor
k I
nte
rfa
ces:
Mot
iva
tion
Mo
tiva
tio
n 1
: P
erfo
rman
ce
• U
tiliz
e hi
gh-s
peed
net
s
• S
uppo
rt p
aral
lel p
roce
ssin
g on
NoW
’s
• F
iner
com
m g
ranu
larit
y
fi T
hat m
eans
: Low
late
ncy
and
high
ban
dwid
th
Mo
tiva
tio
n 2
: F
lexi
bili
ty
• V
aria
nts
of s
tand
ard
prot
ocol
s
• N
ew c
omm
unic
atio
n se
man
tics
• T
hat m
eans
: Aba
ndon
in-k
erne
l pro
toco
l sta
cks
Pro
po
sed
so
luti
on
: U
ser-
leve
l net
wo
rk a
cces
s
• P
rovi
de m
inim
al in
terf
ace
enab
ling
com
mun
icat
ion
• A
pplic
atio
n im
plem
ents
pro
toco
ls d
irect
ly
• N
eed
to e
nsur
e pr
otec
tion
betw
een
proc
esse
s
Page
2H
PCA
-3, S
an A
nton
io, 5
Feb
ruar
y 19
97M
att W
elsh
, Cor
nell
Uni
vers
ity
AT
M a
nd F
ast E
ther
net N
Is f
or U
ser-
Lev
el C
omm
unic
atio
n
U-N
et: B
asi
c Id
eaT
rad
itio
nal
:
- A
ll co
mm
unic
atio
n vi
a ke
rnel
U-N
et:
- A
pplic
atio
ns s
end/
recv
dire
ctly
via
sim
ple
MU
X in
NI
- K
erne
l onl
y in
volv
ed in
con
nect
ion
set-
up/s
hut-
dow
n
Gen
eric
co
mm
un
icat
ion
arc
hit
ectu
re
- C
an b
e im
plem
ente
d in
har
dwar
e, s
oftw
are,
or
both
Inva
rian
ts:
- O
ff-th
e-sh
elf h
ardw
are
and
softw
are
- N
o co
mpr
omis
e on
pro
tect
ion:
A
can
not i
nspe
ct o
r co
rrup
t B’s
mes
sage
s, A
can
not i
mpe
rson
ate
B
Page
3H
PCA
-3, S
an A
nton
io, 5
Feb
ruar
y 19
97M
att W
elsh
, Cor
nell
Uni
vers
ity
AT
M a
nd F
ast E
ther
net N
Is f
or U
ser-
Lev
el C
omm
unic
atio
n
Insp
ira
tion
: MP
P S
yste
ms
Key
idea
: U
ser-
leve
l acc
ess
to N
I
• E
xam
ples
: T
MC
CM
-5, I
BM
SP
-2, M
eiko
CS
-2
Ad
van
tag
es
• B
ypas
ses
the
kern
el fo
r se
nd/r
ecv
• N
o co
py: D
MA
dire
ct to
/from
use
r m
emor
y
• A
pplic
atio
n-sp
ecifi
c pr
otoc
ols
Sh
ort
com
ing
s
• C
usto
m n
etw
ork
and
NI
• A
ssum
es h
omog
eneo
us n
odes
• R
estr
icts
mul
ti-us
er c
apab
ilitie
s
Page
4H
PCA
-3, S
an A
nton
io, 5
Feb
ruar
y 19
97M
att W
elsh
, Cor
nell
Uni
vers
ity
AT
M a
nd F
ast E
ther
net N
Is f
or U
ser-
Lev
el C
omm
unic
atio
n
Ove
rvie
wS
um
mar
y: E
xplo
re d
esig
n s
pac
e o
f u
ser-
leve
l NI’s
- W
hat i
s th
e ha
rdw
are/
softw
are
trad
eoff?
- D
oes
user
-leve
l com
mun
icat
ion
requ
ire e
xpen
sive
/com
plex
NI’s
?
Ou
r m
eth
od:
Co
mp
arin
g im
ple
men
tati
on
s o
f U
-Net
- N
Is w
ith a
nd w
ithou
t a p
rogr
amm
able
co-
proc
esso
r
- E
xplo
re F
ast E
ther
net a
s an
alte
rnat
ive
to A
TM
for
com
mod
ity in
terc
onne
ct
Det
aile
d p
erfo
rman
ce a
nal
ysis
- C
aref
ul in
stru
men
tatio
n of
U-N
et/F
E im
plem
enta
tion
- M
icro
-ben
chm
arks
for
late
ncy/
band
wid
th p
erfo
rman
ce
Ap
plic
atio
n p
erfo
rman
ce
- S
et o
f par
alle
l ben
chm
arks
mea
sure
d ov
er F
E a
nd A
TM
wor
ksta
tion
clus
ters
Page
5H
PCA
-3, S
an A
nton
io, 5
Feb
ruar
y 19
97M
att W
elsh
, Cor
nell
Uni
vers
ity
AT
M a
nd F
ast E
ther
net N
Is f
or U
ser-
Lev
el C
omm
unic
atio
n
Th
e U
-Net
In
terf
ace
U-N
et E
nd
po
int:
Vir
tual
dev
ice
inte
rfac
e
- M
essa
ge b
uffe
rs a
nd s
end/
recv
/free
que
ues
Tra
nsm
it o
per
atio
n:
- U
ser
cons
truc
ts m
sg in
buf
fer
area
, pus
hes
Tx
desc
ripto
r on
to s
end
queu
e
Rec
eive
op
erat
ion
:
- M
sg a
rriv
es, d
ata
in b
uffe
r fr
om fr
ee q
ueue
, Rx
desc
ripto
r pu
shed
ont
o re
cv q
ueue
Page
6H
PCA
-3, S
an A
nton
io, 5
Feb
ruar
y 19
97M
att W
elsh
, Cor
nell
Uni
vers
ity
AT
M a
nd F
ast E
ther
net N
Is f
or U
ser-
Lev
el C
omm
unic
atio
n
U-N
et A
TM
Im
ple
men
tati
on
• O
rigin
al im
plem
enta
tion
of U
-Net
• P
rogr
amm
able
co-
proc
esso
r, A
TM
as
"obv
ious
cho
ice"
for
inte
rcon
nect
256K
SR
AM
25 M
Hz
i960
FO
RE
PC
A-2
00 A
TM
Inte
rfac
e
U-N
et E
nd
po
int
Use
r ad
dre
ss s
pac
eF
OR
E S
yste
ms
PC
A-2
00
- P
CI b
us 1
55 M
bps
OC
-3 A
TM
NI
- 25
MH
z i9
60, 2
56K
SR
AM
- P
entiu
m 1
33 W
S, L
inux
1.3
.97
• U
-Net
impl
emen
ted
on i9
60
• T
x/F
ree
rings
map
ped
from
i960
RA
M
• B
uffe
rs, R
x rin
g in
pin
ned
mem
ory
segm
ents
... a
lway
s D
MA
-abl
e by
the
i960
• N
o O
/S, C
PU
inte
rven
tion
in T
x/R
x
Page
7H
PCA
-3, S
an A
nton
io, 5
Feb
ruar
y 19
97M
att W
elsh
, Cor
nell
Uni
vers
ity
AT
M a
nd F
ast E
ther
net N
Is f
or U
ser-
Lev
el C
omm
unic
atio
n
U-N
et F
ast
Eth
ern
et I
mp
lem
enta
tion
DE
Cch
ip 2
1140
FE
co
ntr
olle
r
- 10
0 M
bps
UT
P5
or fi
ber
- P
CI b
usm
aste
ring
inte
rfac
e
- B
ut, n
ot p
rogr
amm
able
- Lo
w c
ost:
$150
/boa
rd
- P
entiu
m 1
33 W
S, L
inux
1.3
.97
• S
ingl
e, s
hare
d T
x an
d R
x rin
gs, b
uffe
r po
ol
• A
ssum
es s
ingl
e O
/S a
gent
to m
ux th
e qu
eues
• U
-Net
impl
emen
ted
in k
erne
l tra
p an
d in
terr
upt r
outin
es
Page
8H
PCA
-3, S
an A
nton
io, 5
Feb
ruar
y 19
97M
att W
elsh
, Cor
nell
Uni
vers
ity
AT
M a
nd F
ast E
ther
net N
Is f
or U
ser-
Lev
el C
omm
unic
atio
n
Tra
nsm
it O
per
ati
on
U
-Net
/AT
M
1. U
ser
cons
truc
ts d
ata
in b
uffe
r re
gion
2. U
ser
push
es T
x de
scr
into
Tx
Rin
g
3. i9
60 p
olls
Tx
rings
, fet
ches
des
crip
tor
4. i9
60 in
itiat
es D
MA
to fi
ber
outp
ut
5. i9
60 s
ets
Tx
desc
r do
ne fl
ag
256K
SR
AM
25 M
Hz
i960
FO
RE
PC
A-2
00 A
TM
Inte
rfac
e
U-N
et E
nd
po
int
Use
r ad
dre
ss s
pac
e
U-N
et/F
ast
Eth
ern
et
1. U
ser
cons
truc
ts d
ata
2. U
ser
push
es T
x de
scr
3. U
ser
calls
trap
4. T
rap
push
es d
escr
to d
evic
e T
x R
ing
5. O
n T
x do
ne, t
rap
sets
Tx
desc
r do
ne fl
ag
Page
9H
PCA
-3, S
an A
nton
io, 5
Feb
ruar
y 19
97M
att W
elsh
, Cor
nell
Uni
vers
ity
AT
M a
nd F
ast E
ther
net N
Is f
or U
ser-
Lev
el C
omm
unic
atio
n
Rec
eive
Op
era
tion
U-N
et/A
TM
1. A
AL5
PD
U c
ells
arr
ive
at fi
ber
inpu
t
2. i9
60 fe
tche
s fr
ee b
uffe
r de
scr
3. i9
60 in
itiat
es D
MA
to fr
ee b
uffe
r
4. A
t End
-of-
PD
U, i
960
writ
es R
x de
scr
5. U
ser
polls
Rx
FIF
O, o
r up
call
256K
SR
AM
25 M
Hz
i960
FO
RE
PC
A-2
00 A
TM
Inte
rfac
e
U-N
et E
nd
po
int
Use
r ad
dre
ss s
pac
e
U-N
et/F
ast
Eth
ern
et
1. F
E p
acke
t arr
ives
, int
erru
pt r
aise
d
2. In
tr fe
tche
s fr
ee b
uffe
r de
scr
3. In
tr c
opie
s fr
om d
evic
e bu
ffer
to u
ser
buffe
r
4. In
tr w
rites
Rx
desc
r in
to R
x F
IFO
5. U
ser
polls
Rx
FIF
O, o
r up
call
Page
10
HPC
A-3
, San
Ant
onio
, 5 F
ebru
ary
1997
Mat
t Wel
sh, C
orne
ll U
nive
rsity
AT
M a
nd F
ast E
ther
net N
Is f
or U
ser-
Lev
el C
omm
unic
atio
n
U-N
et/F
E T
ran
smit
op
era
tion
Fas
t tr
ap t
o s
tart
tra
nsm
it, 4
.2 u
sec
any
size
pac
ket
- N
ull t
rap:
1 u
sec
- P
CI a
cces
s tim
e do
min
ates
- T
rap
sem
antic
s ar
e ’s
ervi
ce U
-Net
Tx
queu
e’
- T
rap
seen
as
’pro
tect
ed c
o-ro
utin
e’
- T
ake
(sm
all)
slic
e of
mai
n C
PU
tim
e to
mux
U-N
et
Page
11
HPC
A-3
, San
Ant
onio
, 5 F
ebru
ary
1997
Mat
t Wel
sh, C
orne
ll U
nive
rsity
AT
M a
nd F
ast E
ther
net N
Is f
or U
ser-
Lev
el C
omm
unic
atio
n
U-N
et/F
E R
ecei
ve o
per
ati
on
Inte
rru
pt
han
dle
r o
n R
x, c
op
y ti
me
do
min
ates
- M
sg a
rriv
es in
fixe
d bu
ffer
pool
in k
erne
l, co
py to
use
r
Mu
x/D
emu
x:
- U
-Net
’pro
toco
l ID
’ in
Eth
erne
t hea
der,
plu
s ’c
hann
el n
umbe
r’ an
d le
ngth
fiel
d
- N
eed
to in
tegr
ate
with
IP/p
acke
t filt
erin
g
Page
12
HPC
A-3
, San
Ant
onio
, 5 F
ebru
ary
1997
Mat
t Wel
sh, C
orne
ll U
nive
rsity
AT
M a
nd F
ast E
ther
net N
Is f
or U
ser-
Lev
el C
omm
unic
atio
n
Per
form
an
ce: B
an
dw
idth
0102030405060708090100
110
120
025
050
075
010
0012
5015
00
Bandwidth, Mbps
Mes
sage
siz
e, b
ytes
AT
MF
E, H
ubF
E, B
ay N
etw
orks
281
15
AT
M 120
Mbp
s T
AX
I use
d as
rec
eive
r
Fas
t E
ther
net
90 M
bps+
with
500
-byt
e m
essa
ges
... b
ut s
witc
h sh
aves
off
som
e b/
w?
Page
13
HPC
A-3
, San
Ant
onio
, 5 F
ebru
ary
1997
Mat
t Wel
sh, C
orne
ll U
nive
rsity
AT
M a
nd F
ast E
ther
net N
Is f
or U
ser-
Lev
el C
omm
unic
atio
n
Per
form
an
ce: L
ate
ncy
0
100
200
300
400
500
600
700
800
025
050
075
010
0012
5015
00
Round-trip time, usec
Mes
sage
siz
e, b
ytes
AT
MF
E, H
ubF
E, B
ay N
etw
orks
281
15F
E, C
able
tron
FN
100
FE
has
low
er la
ten
cy t
han
AT
M!
... fo
r sm
all m
essa
ges,
any
way
• F
E s
witc
hes
add
17 u
sec
one-
way
Page
14
HPC
A-3
, San
Ant
onio
, 5 F
ebru
ary
1997
Mat
t Wel
sh, C
orne
ll U
nive
rsity
AT
M a
nd F
ast E
ther
net N
Is f
or U
ser-
Lev
el C
omm
unic
atio
n
Sp
lit-C
Ben
chm
arks
: A
TM
vs.
FE
cpu
net
0 0.2
0.4
0.6
0.8
1 1.2
1.4
1.6
1.8
2
atm2 fe2
atm4 fe4
atm8 fe8
mm
128x
128
atm2 fe2
atm4 fe4
atm8 fe8
mm
16x1
6
atm2 fe2
atm4 fe4
atm8 fe8
sso
rtsm
512K
atm2 fe2
atm4 fe4
atm8 fe8
sso
rtlg
512K
atm2 fe2
atm4 fe4
atm8 fe8
rso
rtsm
512K
atm2 fe2
atm4 fe4
atm8 fe8
rso
rtlg
512K
mm
: Mat
rix M
ultip
ly
ssor
t: S
ampl
e so
rt
rsor
t: R
adix
sor
t
Spa
rcS
tatio
ns u
sed
in A
TM
clus
ter
• A
TM
fast
er fo
r la
rge
msg
s
• F
E fa
ster
for
smal
l msg
s
• S
PA
RC
fp fa
ster
than
Pen
tium
• P
entiu
m in
t ops
fast
er
than
SP
AR
CS
plit
-C
• N
ovel
par
alle
l lan
guag
e ba
sed
on C
• U
se o
f ’gl
obal
poi
nter
s’ to
acc
ess
othe
r pr
oc a
ddr
spac
e
Page
15
HPC
A-3
, San
Ant
onio
, 5 F
ebru
ary
1997
Mat
t Wel
sh, C
orne
ll U
nive
rsity
AT
M a
nd F
ast E
ther
net N
Is f
or U
ser-
Lev
el C
omm
unic
atio
n
Cu
rren
t w
ork
wor
k: M
emor
y M
an
age
men
tP
inn
ed b
uff
ers
and
qu
eues
- Lo
cked
into
phy
sica
l mem
ory
for
lifet
ime
of p
roce
ss
- R
equi
red
to a
llow
dire
ct D
MA
to/fr
om u
ser
spac
e
Pag
ing
En
dp
oin
ts
- O
n-de
man
d pa
ging
of U
-Net
buf
fers
- U
ses
softw
are
TLB
to c
ache
pag
e m
appi
ngs
- T
LB m
iss
caus
es k
erne
l int
erru
pt to
fetc
h pa
ge
- P
ages
dis
card
ed o
n T
LB c
apac
ity m
iss
Issu
es - W
ritea
ble
page
s ar
e ea
sy to
get
- W
hat a
bout
sw
appe
d-ou
t rea
d pa
ge?
(Can
’t sw
ap in
inte
rrup
t...)
- La
zy r
ead-
page
ret
rieva
l: T
ell t
he N
I to
try
agai
n
Impl
emen
tatio
ns fo
r P
CA
-200
& L
inux
, DC
2114
0, Z
eitn
et &
Win
dow
s N
T
Page
16
HPC
A-3
, San
Ant
onio
, 5 F
ebru
ary
1997
Mat
t Wel
sh, C
orne
ll U
nive
rsity
AT
M a
nd F
ast E
ther
net N
Is f
or U
ser-
Lev
el C
omm
unic
atio
n
Su
mm
ary
U-N
et e
xten
ded
to
no
n-p
rog
ram
mab
le N
ICs
- Im
plem
enta
tion
usin
g D
C21
140
FE
inte
rfac
e
- H
ardw
are
requ
ires
kern
el tr
ap a
nd c
opy
on r
ecei
ve
U-N
et e
xten
ded
to
Fas
t E
ther
net
- R
ound
-trip
late
ncy
star
ts a
t 57
usec
, 40
byte
pin
g-po
ng
- Lo
wer
late
ncy
than
OC
-3 A
TM
(12
0 us
ec, 4
0 by
te p
ing-
pong
)
- B
andw
idth
rea
ches
> 9
0 M
bps
with
500
-byt
e m
essa
ges
Co
ncl
usi
on
s
- U
-Net
mod
el e
xten
ds to
oth
er n
etw
orks
and
NI a
rchi
tect
ures
- S
plit-
C b
ench
mar
ks d
emon
stra
te c
ompa
rabl
e ap
p pe
rfor
man
ce
- F
ast E
ther
net i
s an
exc
elle
nt p
rice-
perf
orm
ance
poi
nt fo
r w
orks
tatio
n cl
uste
rs
Page
17
HPC
A-3
, San
Ant
onio
, 5 F
ebru
ary
1997
Mat
t Wel
sh, C
orne
ll U
nive
rsity
