Upload
buidien
View
235
Download
4
Embed Size (px)
Citation preview
IOS XR Platform Hardware Architectures
LJ Wobker, Principal Engineer
BRKSPG-2404
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Cisco Spark
Questions? Use Cisco Spark to communicate with the speaker after the session
1. Find this session in the Cisco Live Mobile App
2. Click “Join the Discussion”
3. Install Spark or go directly to the space
4. Enter messages/questions in the space
How
cs.co/ciscolivebot#BRKSPG-2404
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Teaser Slide…
• We’re going to talk about how routers are actually built. We’ll talk a lot about what things
go into the design choices, and why it’s really hard (actually, it’s impossible) to build things
that solve all the problems at the same time. What we’re really going to talk about are
engineering realities, and the tradeoffs that they impose on those of us who build, and
those who ultimately use, the systems. We’ll talk about the major dimensions that drive
hardware design, and the relative costs to make “one thing better” in a given system. We
will cover the high level architecture of several IOS XR platforms, but mostly in the context
of how and why they are different and what design choices went in to each one.
• Introduction –
• building a forwarding path
• Platform Design & Building Blocks
• IOS XR Platforms
• Hardware
• Virtualized
AgendaModern demands such as cloud computing, mobility,
and media delivery are driving greater bandwidth and
service requirements into provider networks. To
address this, Cisco offers a broad portfolio of systems
for provider networks. These systems all run the IOS
XR operating system, but have meaningful differences
in how they are designed from a system/hardware
standpoint. This presentation discusses system
architectures for the NCS-6000, ASR-9000, NCS-5500,
and other IOS XR platforms with a specific focus on the
high-level design decisions. A significant part of the talk
is around silicon selection (custom vs. merchant, for
example) and the mechanical and logical design of
linecards for these larger systems.
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Should I be here?Today’s topics and not-topics.
• Hardware architectures
• System-level design
• Data planes
• Packet forwarding
• High-speed, complex, (expensive?) systems
• Routing protocols / design
• Network level designs
• Control plane / OS infrastructure
• Selling a specific product
Yes!
No! (or at least not much)
BRKSPG-2404 6
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
You can’t always get what you wantBut if you try sometimes well you just might find you get what you need
• Nothing is free
• Some things are closer to free
• The further from the middle,
the more things “cost”
BRKSPG-2404 7
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
The IOS XR Router Family
NCS 5000 NCS 5500 NCS 6000 ASR 9000
CRS
BRKSPG-2404 8
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Optics
Power
Silicon
Cooling
Process PP
S
SE
RD
ES
NE
BS
Busbar
Chassis
Airflo
w
Backplane
Sig
nal In
teg
rity Pip
elin
e
Run to completion
LR4
QSFP28
CPAK
N:NFilters
StatsFeatures
Programmability
Die
siz
eInterposer
Cable Management
Fast Convergence
He
at S
inks
Aco
ust
ics
Materials
PCB
Junction temp
Liq
uid
co
olin
g
Co
nn
ecto
rs
Ro
uti
ng
Slo
t p
itch
MemoryFIB size
Buffering
Cap
acity
BandwidthO
ps/se
c
Se
rial
HB
M
HMC
GDDR5
DD
R3
TCAM
SRAM
On
-ch
ip
LLD
RA
M
ACL scale
Fabric
MPO
IPo
DW
DM
RouterHardware
BuildingBlocks
BRKSPG-2404 9
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
What’s needed to build a forwarding path?
1. Optical to electrical
2. Transport a signal from optics to NPU
3. Ingress forwarding operations
4. Transport a signal from ingress NPU to egress NPU (fabric)
5. Egress forwarding operations
6. NPU to optics
7. Electrical to optical
1/7
4
2/63/5
BRKSPG-2404 10
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Logical view of forwarding path components
Fabric
Interface
ASIC
Optics
Optics
FIA
FIA
Optics
Optics
FIAOptics
nPower X1
nPower X1
nPower X1
nPower X1 FIAOptics
Tables
Packet
Buffers
TCAM TM
nPower X1
Fabric
Control
EthernetCPUDRAM
Ethernet
Switch
NPU
LASERS / RECEIVERS
FAST MEMORY
FAST MEMORY
TCAM MEMORY
SERDES
SERDES
SILICON
SILICON
BRKSPG-2404 11
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Challenge: Scale routers faster than componentsNote: exponential scale
1x
64x
16x
4x
256x
1024x
4096x
Moore’s Law
?
?
?
?
BRKSPG-2404 12
1998 2000 2012 20162002 2004 201020082006 2014 2018
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Technology Trends – POS/Ethernet StandardsFastest Interface Bandwidth
1x
64x
16x
4x
256x
1024x
4096x
OC-12
GE
100G
OC-48
40G
10G
400G
Moore’s LawInterface Speeds
BRKSPG-2404 13
1998 2000 2012 20162002 2004 201020082006 2014 2018
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Technology Trends – POS/Ethernet StandardsFastest Interface & Router Bandwidth
1x
64x
16x
4x
256x
1024x
4096x
OC-12
GE
100G
40G
10G
400G
Moore’s LawInterface Speeds
58T
8G
8T
640G
2.4T
16T
Buffered Router Bandwidth
OC-4828G
24T
BRKSPG-2404 14
1998 2000 2012 20162002 2004 201020082006 2014 2018
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Technology Trends - OpticsBandwidth/Volume
1x
64x
16x
4x
256x
1024x
4096x
GBIC 100G QSFP28100G CFP10G SFP+10G XFP100G CPAK40G QSFP+
Driven by physics, not Moore’s Law, costs shifting
SerDes dependency (NPU interface)
Cooling challenge – 30C lower max temp vs. ASICs
BRKSPG-2404 15
1998 2000 2012 20162002 2004 201020082006 2014 2018
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Silicon & Modular Router PPS – Per NPU/LC*Mpps for Standalone and Modular/Buffered
1x
64x
16x
4x
256x
1024x
4096x
12575
164
700x6
280x5
CPU
3200
1200800
*represents a combination of memory, Silicon & SerDes
CRS-1
SoC – no buffer
Modular – buffered
NCS 6000
NCS 5500
12000
CRS-3
14x4
45x4
ASR 9000
ASR 9000
150x4
ASR 9000
BRKSPG-2404 16
1998 2000 2012 20162002 2004 201020082006 2014 2018
4800
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Technology Trends – MemoryCommodity & Custom Bandwidth
1998 2000 2012 20162002 2004 201020082006 2014
1x
64x
16x
4x
256x
1024x
4096x
105
1.60.8
2528
100
GDDR5
DDR4
SDRDDR
DDR2
DDR3
HBM
325HMC
55
160
NCS 6000
NCS 6000
ASR 9000
2nd & 3rd gen
Scaling FIB above ~256K IPv4
usec vs. msec buffering
Operations / second critical for FIB
BRKSPG-2404 17
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Technology Trends – SerDesHigh-performance Electrical Link (Speed in GHz)
1998 2000 2012 20162002 2004 201020082006 2014
1x
64x
16x
4x
256x
1024x
4096x
11.5
2.5
1.251.25
2515
5
Optics to NPU
NPU to fabric
NPU to TCAM
NPU to serial memory
BRKSPG-2404 18
2018
56
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Technology Trends Compared
1998 2000 2012 20162002 2004 201020082006 2014
1x
64x
16x
4x
256x
1024x
4096x
InterfacesPPS
OpticsMemory
SerDes
BRKSPG-2404 19
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Modular Buffered Router BandwidthSystems growing much faster than components – exponential scale
1998 2000 2012 20162002 2004 201020082006 2014
1x
64x
16x
4x
256x
1024x
4096x
InterfacesPPS
Memory
SerDes
*similar trends for conductive materials, connectors, fans, power supplies, CPUs, …
58T
8G
8T
640G
2.4T
16T
28G
24T
Optics
BRKSPG-2404 20
Platform Design
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Why not one platform?Requirements have a butterfly effect
• An individual requirement may dramatically impact architecture
• Every component “ahead of the curve” adds complexity
• Key Drivers• Time / Cost
• Buffering
• FIB scale
• System Scale
• Features/Flexibility
BRKSPG-2404 22
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Time –Terabit LCs in 2013 vs. 2016
• NCS 6000 1T – 200G NPU w/ full FIB, deep buffers, programmable• Significant investment in nPower X1 silicon development
• Custom memory for FIB and buffering – extremely high bandwidth and ops/sec
• $271M acquisition of Lightwire for small, low-power 100G optics
• ASR 9000 1.2T – similar speed, flexibility, and power – 3 years later• Commercial NPU, customized for Cisco, commodity memories
• Cisco fabric, backward compatible with earlier generations
• QSFP28 optics, full range of chassis sizes
• NCS 5500 3.6/2.4T – lighter silicon architecture• Commodity memory, partial bandwidth buffering, VoQ
• TCAM and non-TCAM options (density change)
• Reduced counters, QoS, queueing complexity
• 3.6T with very high scale only in 2018...BRKSPG-2404 23
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Buffering
• On-chip limited to microseconds
• But… Off-chip requires • ASIC I/O pins – fewer interfaces
• Board space and power
• Commodity bandwidth is limited• Many devices may be needed
• Better suited for buffering than FIB (1 read/write per packet optimized for 128B read/write)
• Stalled – High-end graphics and networking moving to custom
• Custom memories• Fewer devices are needed to reach bandwidth, fewer pins
• Development and per-unit costs – Extremely expensive
NPU
Po
rts
Po
rts
Ports
Ports
FIB
Buffers
NPU
Po
rts
Po
rts
Ports
FIB
BRKSPG-2404 24
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
FIB Scale
• On-chip FIB limited to ~256K LPM entries• Usually an on-chip TCAM or lookup tree
• Can be scaled up to ~1M with well-known prefixes
• Combination of flat tables and LPM
• Larger FIB tables require external memory:
• ASIC I/O pins redirected from interfaces to memory• 4-8 reads for every packet – high operations per second• Board space (density) and power
NPU
Po
rts
FIB
Buffers
Po
rts
BRKSPG-2404 25
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
System ScaleFour common approaches
• Fixed – single NPU / Forwarding ASIC
• All resources dedicated to network interfaces
• May or may not have external memories
• Fixed – multiple NPUs
• Connected via mesh or fabric chips
• Modular – Expand with line cards
• Resources for fabric and usually external memories
• Multi-chassis
• Adds fiber connections to fabric cards (more power & board space)
• Increases software complexity
NPU
Fab
ric
Po
rts
FIB
Buffers
BRKSPG-2404 26
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Features and Flexibility
• Packet Processing Engine (PPE)• C programmable
• Run to completion
• Anything is possible, no pure optimization
• Wide range of pipeline programmability• NCS 5000 < NCS 5500 < ASR 9000 < NCS6K
• Packet rate is strongly correlated to cost, power, and flexibility
• Counting stuff is much harder than most people realize
Netw
ork
In
terf
ace Ingress Egress
Port
TermParser
Link
Layer
VLAN
Trans
Tunnel
Service
Term
Fwd PMFFEC
ResFEC
Res
NCS 5500 PIPELINE
CRS PPES
BRKSPG-2404 27
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Easy summary...
... and ...
A “cut corner” is the same thing as
“a tradeoff
you didn’t know you were making”
BRKSPG-2404 28
• “Trying to build X things to solve X^N problems”• is the same thing as
• A bunch of “engineering optimizations”
IOS XR Platforms
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
The IOS XR Router Family
NCS 5000 NCS 5500 NCS 6000 ASR 9000
CRS
BRKSPG-2404 30
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
• Highest router bandwidth capacity
• 128 Tbps first generation multi-chassis (2013)
• Only Tbps card on the market in 2013-2015
• With buffers and full FIB
• Dramatic improvement in power efficiency
• $6000/100G/month power savings over CRS-3 64x100
• Merchant fabric
• Highly programmable forwarding, massive stats
• Custom memories for FIB and buffering
• Custom CPAK optics
NCS 6000 – First Dense 100G
BRKSPG-2404 31
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
NCS 6000 Line Card Architecture
10X 100GE
• Slice architecture• Optics, NPU & FIA per slice
• Run to Completion NPU w/ PPEs
• 1 Generation (so far)• 1T line cards w/ 200G NPUs (40 nm)
• 400G NPUs in lab
• TCAM for ACL/QoS scale
• Deep buffers (50+ msec)
Fabric
Interface
ASIC
Optics
Optics
FIA
FIA
Optics
Optics
FIAOptics
nPower X1
nPower X1
nPower X1
nPower X1 FIAOptics
Tables
Packet
Buffers
T
C
A
M
T
M
nPower X1
CPUDRAM
PPEs
BRKSPG-2404 32
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
ASR 9000 Series
• Full range of chassis and interfaces• 40G, 200G, and 1T generations
• Highly programmable forwarding• Partnered for a customized NPU – cisco software
• Highest scale, feature, and QoS capabilities
• Cisco fabric
• Commodity memories for FIB and buffering
• Mostly commodity optics
BRKSPG-2404 33
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
ASR 9000 System Design
• Backplane or midplane
• Flexible switch fabric options• RP and fabric may be integrated into RSP
• Variable # of fabrics for increased capacity and redundancy
• First stage of fabric on line card
• Multiple airflow designs• Front-to-back – 9922, 9912, 9010
• Side-to-back – 9006
• Side-to-side – 9001, 9004
• Modular options for lower-speed and legacy interfaces
BRKSPG-2404 34
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
ASR 9000 Line Card Architecture
8X 100GE
CPUDRAM
FIANP-5cCPAK
CPAK
FIB Packet BuffersTCAM
FIANP-5cCPAK
CPAK
FIANP-5cCPAK
CPAK
FIANP-5cCPAK
CPAK
Fa
bric
• Slice architecture• Optics, NPU & FIA per slice
• Flexible pipeline NPU
• 1st stage of fabric on line card
• 3 Generations (so far)• 40G line cards w/ 15G NPUs (90 nm)
• 200-360G line cards w/ 60G NPUs (55 nm)
• 800G-1.2T line cards with 240G NPUs (28 nm)
• (at least) one more generation of NPU/LC
• TCAM for ACL/QoS scale
• Huge FIB (5M+) & buffers (200 msec)
BRKSPG-2404 35
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
ASR 9000 Modular Line Card Architecture
CPUDRAM
Fabric
FIANP-5c
FIB Packet BuffersTCAM
FIANP-5c
FIB Packet BuffersTCAM
Bay
0
Bay
1
2/4/8X 10GE 20X 10G
1/2X 40G20X 1G 1/2X100G
PH
Y
10G SFP+
10G SFP+
10G SFP+
10G SFP+
10G SFP+
10G SFP+
10G SFP+
10G SFP+
PH
Y
10G SFP+
10G SFP+
10G SFP+
10G SFP+
10G SFP+
10G SFP+
10G SFP+
10G SFP+
10G SFP+
10G SFP+
PH
Y
10G SFP+
10G SFP+
PH
Y
10G SFP+
10G SFP+
10G SFP+
10G SFP+
PH
Y
40G
QSFP+
40G
QSFP+
PH
Y
100G
CPAK
100G
CPAK
PH
Y
40G
QSFP+
PH
Y
100G
CPAK
PH
Y
1G SFP
1G SFP
1G SFP
1G SFP
1G SFP
1G SFP
1G SFP
1G SFP
1G SFP
1G SFP
MODULAR 3RD GENERATION LC
BRKSPG-2404 36
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
• 2x NP5c NPUs, “mini fabric”
• Full feature set from ASR9k
• direct inheritance of all functionality / config
• Dense hierarchical QoS
• Mixed set of interface speeds / optics
• MACSEC integrated into PHY
• full bandwidth deep packet buffers
• 2xQSFP28, 24xSFP+, 16xQSFP
ASR 9901 – ~450Gbps, 2RU full featured system
BRKSPG-2404 37
CPUDRAM
Fabric
FIA
Packet Buffers
NP
-5c
FIB
TCAM Packet Buffers
FIA
Packet Buffers
NP
-5c
FIB
TCAM Packet Buffers
QSFP(+,28)
40/100GE
SFP(+)
10GE / 1GE
x12
SFP
1GE
x8
QSFP(+,28)
40/100GE
SFP(+)
10GE / 1GE
x12
SFP
1GE
x8
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
• Dense 100GE with deep buffers
• 8 & 16-slot modular, 1 & 2 RU fixed
• Up to 57.6 Tbps in ½ rack in 2016
• 2.4T - 3.6T line cards
• Options for route and ACL scale via TCAM
• High scale 3.6T in 2018
• Highly integrated Silicon
• Single ASIC for forwarding & fabric interface
• Dramatic power reduction to 0.24 W/Gbps
• QSFP28 optics
• GDDR5 commodity buffers
NCS 5500 – 3rd generation 100G
BRKSPG-2404 38
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
• New chassis design for XR
• Orthogonal direct connect
• Horizontal line cards and vertical fabric
• Direct connection between line cards and fabric cards
• No midplane
• Distributed air intake between cards
• Fans cover fabric cards
• Provides cool air equally to all optics
• Optics require ~30C cooler operation than silicon
• Avoid preheating air from module to module
NCS 5500 System Design
REAR VIEW
AIR INLET
FAN
REMOVED
BRKSPG-2404 39
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
NCS 5500 Line Card Architecture
36X 100GE
• Slice architecture• Optics & Pipelined Forwarding ASIC per slice – integrated FIA
• Optional TCAM for FIB/ACL scale
• 1st Generation• 3.6T line cards FA at 600G
• 2.9T line cards FA at 720G + TCAM
• Deep VoQ buffers (50+ msec)
CPUDRAM
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
QSFP28
Forwarding
ASIC
Optics x 6 FA
Optics x 6 FA
Optics x 6 FA
Optics x 6 FA
Optics x 6 FA
Buffers
BRKSPG-2404 40
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
NCS 5501/5502 Architecture
NCS 5501 800G
• NCS 5501• 1RU – Single 800 Gbps Forwarding ASIC
• 4x 100G + 40x 10G
• NCS 5502• 2RU – 8 600Gbps Forwarding ASICs
• Integrated switch fabric
• 48x 100G
• Optional TCAM for scale
• Deep VoQ buffers (50+ msec)
SF
P+
SF
P+
SF
P+
SF
P+
SF
P+
SF
P+
Forwarding
ASIC
QS
FP
28
QS
FP
28
QS
FP
28
Bu
ffe
rs
40x10G 4x100G
TC
AM
CP
UD
RA
M
QS
FP
28
QS
FP
28
QS
FP
28
QS
FP
28
QS
FP
28
QS
FP
28
QS
FP
28
Forwarding
ASIC
QS
FP
x 6
FA
QS
FP
x 6
FA
QS
FP
x 6
FAB
uff
ers
TC
AM
48x100G
Fabric
CP
UD
RA
M
QS
FP
x 6
FA
QS
FP
x 6
FA
QS
FP
x 6
FA
QS
FP
x 6
FA
SwitchSwitch
NCS 5502 4.8T
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
• 1 RU Fixed System: 24x QSFP28 ports
• Base version only (no room for TCAMs or MACSEC...)
• 2x 900 Gbps Forwarding ASICs(1200:900 nominal oversubscription)
• No Fabric ASIC, Forwarding ASICs
are directly connected “back-to-back”
NCS55A1: 24 x 100GE
BRKSPG-2404 42
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
NCS 5000 PlatformsExtending IOS XR for satellite, ToR, and beyond
NCS 5011
NCS 5002
NCS 5001
• Maximizing Silicon capabilities• No external memories – small FIB, usec buffers
• All bandwidth to ports, no fabric
• Low cost and power
• Full IOS XR Routing
Forwarding ASIC32x100
Buffers
FIB
BRKSPG-2404 43
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
CRS-X
• Scaling CRS into 100G
• 4x 100G
• 40x 100G
• 2x 100G + 5x 40G
• 3x 100G + 1x 100G IPoDWDM
• Up to 51.2T via multi-chassis
• Fully compatible with CRS-1 & CRS-3
BRKSPG-2404 44
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Virtualized IOS XR (“XRv 9000”)
• IOS XR on x86 hardware with Linux VM & containers
• Hosted & small PE with L2 & L3 VPNs
• Ideal route reflector• 64 bit
• 10+ M routes
• Router creation in seconds – rapid deployment
• Runs existing data plane forwarding code
• (recompiled) to an emulated NPU – maintains consistent behavior / features
BRKSPG-2404 45
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Virtualized IOS XR – Performance & QoS
• Targeted for 5-50 Gbps forwarding
• Power per bit looks awful• except when it doesn’t really matter!
• 20+ Gbps with features per socket
• 3-layer Hierarchical QoS
• Scale horizontally
BRKSPG-2404 46
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
• The requirement space is pretty broad
• Optimizing for everything optimizes for nothing
• But we have to optimize for some things or no one buys it!
• And we can’t build an arbitrarily large suite of platforms
• So some of us are lucky enough to get paid to spend lots of
time sitting around arguing about: which memory technologies,
which silicon NPU architectures, which optical parameters,
which mechanical designs, and a bunch of other things to trade
off against each other...
Summary: Why not one platform?
BRKSPG-2404 47
I’m grateful for the opportunity to be here and to talk about something (at least I think) is interesting. It’s
important to me personally that attendees find these presentations helpful and/or valuable. Certainly not
everyone will enjoy every presentation -- but you guys spend a lot of money to come here and Cisco and
the speakers do a lot of work to put on the event and the presentations. If the material is excellent, we’ve
done our jobs. If there’s something we can do better, please let us know... I know you’re not supposed to on
the internet, but I really do read the comments. ;-)
Thanks for your time today.
Lawrence J Wobker, Cisco
February 2018
One serious thing before we go
Q & A
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Cisco Spark
Questions? Use Cisco Spark to communicate with the speaker after the session
1. Find this session in the Cisco Live Mobile App
2. Click “Join the Discussion”
3. Install Spark or go directly to the space
4. Enter messages/questions in the space
How
cs.co/ciscolivebot#BRKSPG-2404
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
• Please complete your Online Session Evaluations after each session
• Complete 4 Session Evaluations & the Overall Conference Evaluation (available from Thursday) to receive your Cisco Live T-shirt
• All surveys can be completed via the Cisco Live Mobile App or the Communication Stations
Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at www.ciscolive.com/global/on-demand-library/.
Complete Your Online Session Evaluation
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Tech Circle
• Meet the Engineer 1:1 meetings
• Related sessions
BRKSPG-2404 52
Thank you