Toronto – August 19, 2011 1 S T A N F O R D U N I V E R S I T Y
NetFPGA InformationalTutorial
Presented by: Adam Covington(Stanford University)
Andrew W. Moore(University of Cambridge)
Toronto, CanadaAugust 19, 2011
http://NetFPGA.org
Toronto – August 19, 2011 2 S T A N F O R D U N I V E R S I T Y
Tutorial Outline
• Motivation– Introduction– The NetFPGA Platform
• Hardware Overview– NetFPGA 1G– NetFPGA 10G
• The Stanford Base Reference Router– Motivation: Basic IP review– Example: Reference Router running on the NetFPGA
• Community Contributions– Altera-DE4 NetFPGA Reference Router (UMassAmherst)– NetThreads (University of Toronto)
• Concluding Remarks
Toronto – August 19, 2011 4 S T A N F O R D U N I V E R S I T Y
NetFPGA = Networked FPGA
A line-rate, flexible, open networking platform for teaching and research
Toronto – August 19, 2011 5 S T A N F O R D U N I V E R S I T Y
NetFPGA 1G Board
NetFPGA consists of…
Four elements:
• NetFPGA board
• Tools + reference designs
• Contributed projects
• CommunityNetFPGA 10G Board
Toronto – August 19, 2011 6 S T A N F O R D U N I V E R S I T Y
NetFPGA 1G NetFPGA 10G
4 x 1Gbps Ethernet Ports 4 x 10Gbps SFP+
4.5 MB ZBT SRAM64 MB DDR2 SDRAM
27 MB QDRII-SRAM288 MB RLDRAM-II
PCI PCI Express x8
Virtex II-Pro 50 Virtex 5 TX240T
NetFPGA Board Comparison
Toronto – August 19, 2011 7 S T A N F O R D U N I V E R S I T Y
FPGAFPGA
MemoryMemory
1GE1GE
1GE1GE
1GE1GE
1GE1GE
NetFPGA board
PCI
CPUCPU MemoryMemory
NetFPGA Board
PC with NetFPGA
NetworkingSoftwarerunning on a standard PC
A hardware acceleratorbuilt with Field Programmable Gate Arraydriving Gigabit network links
Toronto – August 19, 2011 8 S T A N F O R D U N I V E R S I T Y
FPGAFPGA
MemoryMemory
1GE1GE
1GE1GE
1GE1GE
1GE1GE
Running the Router Kit
User-space development, 4x1GE line-rate forwarding
PCI
CPUCPU MemoryMemory
OSPFOSPF BGPBGP
My ProtocolMy Protocoluser
kernelRouting
Table
Usage #1
IPv4RouterIPv4
Router
1GE1GE
1GE1GE
1GE1GE
1GE1GE
FwdingTable
FwdingTable
PacketBuffer
PacketBuffer
“Mirror”
Toronto – August 19, 2011 9 S T A N F O R D U N I V E R S I T Y
FPGAFPGA
MemoryMemory
1GE1GE
1GE1GE
1GE1GE
1GE1GE
Enhancing Modular Reference Designs
PCI
CPUCPU MemoryMemory
Usage #2
NetFPGA DriverNetFPGA Driver
Java GUIFront Panel(Extensible)
Java GUIFront Panel(Extensible)
PW-OSPFPW-OSPF
In QMgmtIn Q
Mgmt
IPLookup
IPLookup
L2Parse
L2Parse
L3Parse
L3Parse
Out QMgmtOut QMgmt
1GE1GE
1GE1GE
1GE1GE
1GE1GE
Verilog modules interconnected by FIFO interfaces
MyBlockMy
Block
VerilogEDA Tools
(Xilinx, Mentor, etc.)
VerilogEDA Tools
(Xilinx, Mentor, etc.)
1. Design2. Simulate3. Synthesize4. Download
1. Design2. Simulate3. Synthesize4. Download
Toronto – August 19, 2011 10 S T A N F O R D U N I V E R S I T Y
FPGAFPGA
MemoryMemory
1GE1GE
1GE1GE
1GE1GE
1GE1GE
Creating new systems
PCI
CPUCPU MemoryMemory
Usage #3
NetFPGA DriverNetFPGA Driver
1GE1GE
1GE1GE
1GE1GE
1GE1GE
My Design
(1GE MAC is soft/replaceable)
My Design
(1GE MAC is soft/replaceable)
VerilogEDA Tools
(Xilinx, Mentor, etc.)
VerilogEDA Tools
(Xilinx, Mentor, etc.)
1. Design2. Simulate3. Synthesize4. Download
1. Design2. Simulate3. Synthesize4. Download
Toronto – August 19, 2011 11 S T A N F O R D U N I V E R S I T Y
Tools + Reference Designs 1G
Tools:• Compile designs• Verify designs• Interact with hardware
Reference designs:• Router (HW)• Switch (HW)• Network Interface Card (HW)• Router Kit (SW)• SCONE (SW)
Toronto – August 19, 2011 12 S T A N F O R D U N I V E R S I T Y
Contributed Projects
More projects:http://netfpga.org/foswiki/NetFPGA/OneGig/ProjectTable
Project Contributor
OpenFlow switch Stanford University
Packet generator Stanford University
NetFlow Probe Brno University
NetThreads University of Toronto
zFilter (Sp)router Ericsson
Traffic Monitor University of Catania
DFA UMass Lowell
Toronto – August 19, 2011 13 S T A N F O R D U N I V E R S I T Y
Community
Wiki• Documentation
– User’s Guide– Developer’s Guide
• Encourage users to contribute
Forums• Support by users for users• Active community - 10s-100s of posts/week
Toronto – August 19, 2011 14 S T A N F O R D U N I V E R S I T Y
International Community
Over 1,000 users, using 1,900 cards at150 universities in 32 countries
Toronto – August 19, 2011 15 S T A N F O R D U N I V E R S I T Y
NetFPGA’s Defining Characteristics
• Line-Rate– Processes back-to-back packets
• Without dropping packets • At full rate of Gigabit Ethernet Links
– Operating on packet headers • For switching, routing, and firewall rules
– And packet payloads• For content processing and intrusion prevention
• Open-source Hardware – Similar to open-source software
• Full source code available • BSD-Style License
– But harder, because • Hardware modules must meeting timing• Verilog & VHDL Components have more complex interfaces • Hardware designers need high confidence in specification of modules
Toronto – August 19, 2011 16 S T A N F O R D U N I V E R S I T Y
Test-Driven Design
• Regression tests– Have repeatable results – Define the supported features– Provide clear expectation on functionality
• Example: Internet Router– Drops packets with bad IP checksum– Performs Longest Prefix Matching on destination address– Forwards IPv4 packets of length 64-1500 bytes– Generates ICMP message for packets with TTL <= 1– Defines how packets with IP options or non IPv4
… and dozens more … Every feature is defined by a regression test
Toronto – August 19, 2011 17 S T A N F O R D U N I V E R S I T Y
Who, How, Why
Who uses the NetFPGA?– Teachers– Students– Researchers
How do they use the NetFPGA?– To run the Router Kit– To build modular reference designs
• IPv4 router• 4-port NIC• Ethernet switch, …
Why do they use the NetFPGA?– To measure performance of Internet systems– To prototype new networking systems
Toronto – August 19, 2011 20 S T A N F O R D U N I V E R S I T Y
Xilinx Virtex II Pro 50
• 53,000 Logic Cells• Block RAMs• Embedded PowerPC
Toronto – August 19, 2011 21 S T A N F O R D U N I V E R S I T Y
Network and Memory
• Gigabit Ethernet– 4 RJ45 Ports– Broadcom PHY
• Memories– 4.5MB Static RAM– 64MB DDR2 Dynamic
RAM
Toronto – August 19, 2011 22 S T A N F O R D U N I V E R S I T Y
Other IO
•PCI– Memory Mapped
Registers– DMA Packet Transferring
•SATA– Board to Board
communication
Toronto – August 19, 2011 23 S T A N F O R D U N I V E R S I T Y
NetFPGA-10G
• A major upgrade• State-of-the-art technology
Toronto – August 19, 2011 24 S T A N F O R D U N I V E R S I T Y
NetFPGA 1G NetFPGA 10G
4 x 1Gbps Ethernet Ports 4 x 10Gbps SFP+
4.5 MB ZBT SRAM64 MB DDR2 SDRAM
27 MB QDRII-SRAM288 MB RLDRAM-II
PCI PCI Express x8
Virtex II-Pro 50 Virtex 5 TX240T
Comparison
Toronto – August 19, 2011 25 S T A N F O R D U N I V E R S I T Y
10 Gigabit Ethernet
• 4 SFP+ Cages• AEL2005 PHY• 10G Support
– Direct Attach Copper– 10GBASE-R Optical
Fiber
• 1G Support– 1000BASE-T Copper– 1000BASE-X Optical
Fiber
Toronto – August 19, 2011 26 S T A N F O R D U N I V E R S I T Y
Others
• QDRII-SRAM– 27MB– Storing routing tables,
counters and statistics
• RLDRAM-II– 288MB– Packet Buffering
• PCI Express x8– PC Interface
• Expansion Slot
Toronto – August 19, 2011 27 S T A N F O R D U N I V E R S I T Y
Xilinx Virtex 5 TX240T
• Optimized for ultra high-bandwidth applications
• 48 GTX Transceivers• 4 hard Tri-mode
Ethernet MACs• 1 hard PCI Express
Endpoint
Toronto – August 19, 2011 28 S T A N F O R D U N I V E R S I T Y
Beyond Hardware
• NetFPGA-10G Board• Xilinx EDK based IDE• Reference designs with
ARM AXI4• Software (embedded and
PC)• Public Repository
(GitHub)• Public Wiki (PBWorks)
Reference DesignsReference Designs AXI4 IPsAXI4 IPs
Xilinx EDKXilinx EDK
MicroBlaze SWMicroBlaze SW PC SWPC SW
PBWorks, GitHub, User CommunityPBWorks, GitHub, User Community
Toronto – August 19, 2011 29 S T A N F O R D U N I V E R S I T Y
NetFPGA-1G Cube Systems
• PCs assembled from parts– Stanford University– Cambridge University
• Pre-built systems available – Accent Technology Inc.
• Details are in the Guidehttp://netfpga.org/static/guide.html
Toronto – August 19, 2011 30 S T A N F O R D U N I V E R S I T Y
Rackmount NetFPGA-1G Servers
NetFPGA inserts in PCI or PCI-X slot
2U Server (Dell 2950)
Thanks: Brian Cashman for providing machine
1U Server (Accent Technology Inc.)
Toronto – August 19, 2011 31 S T A N F O R D U N I V E R S I T Y
Stanford NetFPGA-1G Cluster
Statistics• Rack of 40
• 1U PCs with NetFPGAs
• Managed • Power• Console• LANs
• Provides 4*40=160 Gbps of full line-rate processing bandwidth
Toronto – August 19, 2011 33 S T A N F O R D U N I V E R S I T Y
Internet Protocol (IP)
Data
DataIP
Hdr
Eth Hdr
DataIP
Hdr
Data to betransmitted:
IP packets:
EthernetFrames:
DataIP
HdrData
IP Hdr
Eth Hdr
DataIP
HdrEth Hdr
DataIP
Hdr
…
…
Toronto – August 19, 2011 34 S T A N F O R D U N I V E R S I T Y
Internet Protocol (IP)
Data
DataIP
Hdr…
16 3241
Options (if any)
Destination Address
Source Address
Header ChecksumProtocolTTL
Fragment OffsetFlagsFragment ID
Total Packet LengthT.ServiceHLenVer
20 b
ytes
Toronto – August 19, 2011 35 S T A N F O R D U N I V E R S I T Y
Basic operation of an IP router
R3
A
B
C
R1
R2
R4 D
E
FR5
R5F
R3E
R3D
Next HopDestination
D
Toronto – August 19, 2011 36 S T A N F O R D U N I V E R S I T Y
Basic operation of an IP router
A
B
C
R1
R2
R3
R4 D
E
FR5
Toronto – August 19, 2011 37 S T A N F O R D U N I V E R S I T Y
Forwarding tables
Entry Destination Port
12⋮ 232
0.0.0.00.0.0.1
⋮255.255.255.255
12⋮12
~ 4 billion entries
Naïve approach:One entry per address
Improved approach:Group entries to reduce table sizeEntry Destination Port
12⋮50
0.0.0.0 – 127.255.255.255128.0.0.1 – 128.255.255.255
⋮248.0.0.0 – 255.255.255.255
12⋮12
IP address 32 bits wide → ~ 4 billion unique address
Toronto – August 19, 2011 38 S T A N F O R D U N I V E R S I T Y
IP addresses as a line
0 232-1
Entry Destination Port
12345
StanfordBerkeley
North AmericaAsia
Everywhere (default)
12345
All IP addresses
North AmericaAsia
BerkeleyStanford
Your computer My computer
Toronto – August 19, 2011 39 S T A N F O R D U N I V E R S I T Y
Longest Prefix Match (LPM)
Entry Destination Port
12345
StanfordBerkeley
North AmericaAsia
Everywhere (default)
12345
Universities
Continents
Planet
DataTo:
Stanford
Matching entries:•Stanford•North America•Everywhere
Most specific
Toronto – August 19, 2011 40 S T A N F O R D U N I V E R S I T Y
Longest Prefix Match (LPM)
Entry Destination Port
12345
StanfordBerkeley
North AmericaAsia
Everywhere (default)
12345
Universities
Continents
Planet
DataTo:
Canada
Matching entries:•North America•Everywhere
Most specific
Toronto – August 19, 2011 41 S T A N F O R D U N I V E R S I T Y
Implementing Longest Prefix Match
Entry Destination Port
12345
StanfordBerkeley
North AmericaAsia
Everywhere (default)
12345
Most specific
Least specific
Searching
FOUND
Toronto – August 19, 2011 42 S T A N F O R D U N I V E R S I T Y
Basic components of an IP router
Control Plane
Data Planeper-packet processing
SwitchingForwarding
Table
Routing Table
Routing Protocols
Management& CLI
Softw
areH
ardware
Queuing
Toronto – August 19, 2011 43 S T A N F O R D U N I V E R S I T Y
IP router components in NetFPGA
SCONE
Routing Table
Routing Protocols
Management& CLI
Output PortLookup
ForwardingTable
InputArbiter
OutputQueues
Switching Queuing
Linux
Routing Table
Routing Protocols
Management& CLI
Router Kit
OR
Softw
areH
ardware
Toronto – August 19, 2011 45 S T A N F O R D U N I V E R S I T Y
Operational IPv4 router
Control Plane
Data Planeper-packet processing
Softw
areH
ardware
Routing Table
Routing Protocols
Management& CLI
SCONE
SwitchingForwarding
TableQueuing
Reference router
Java GUI
Toronto – August 19, 2011 47 S T A N F O R D U N I V E R S I T Y
Streaming video
PC & NetFPGA(NetFPGA in PC)
NetFPGA runningreference router
Toronto – August 19, 2011 48 S T A N F O R D U N I V E R S I T Y
Streaming video
Video streaming over shortest path
Videoclient
Videoserver
Toronto – August 19, 2011 49 S T A N F O R D U N I V E R S I T Y
Streaming video
Videoclient
Videoserver
Toronto – August 19, 2011 50 S T A N F O R D U N I V E R S I T Y
Observing the routing tables
Columns:•Subnet address•Subnet mask•Next hop IP•Output ports
Toronto – August 19, 2011 52 S T A N F O R D U N I V E R S I T Y
Review
NetFPGA as IPv4 router:•Reference hardware + SCONE software•Routing protocol discovers topology
Demo:•Ring topology•Traffic flows over shortest path•Broken link: automatically route around failure
Toronto – August 19, 2011 54 S T A N F O R D U N I V E R S I T Y
FPGAFPGA
MemoryMemory
1GE1GE
1GE1GE
1GE1GE
1GE1GE
Running the Router Kit
User-space development, 4x1GE line-rate forwarding
PCI
CPUCPU MemoryMemory
OSPFOSPF BGPBGP
My ProtocolMy Protocoluser
kernelRouting
Table
Usage #1
IPv4RouterIPv4
Router
1GE1GE
1GE1GE
1GE1GE
1GE1GE
FwdingTable
FwdingTable
PacketBuffer
PacketBuffer
“Mirror”
Toronto – August 19, 2011 55 S T A N F O R D U N I V E R S I T Y
Altera-DE4 NetFPGA Reference Router
• Migration of NetFPGA infrastructure to DE4 Stratix IV – 4X logic vs. Virtex 2
• PCI Express Gen2 – 5.0Gbps/lane data • External DDR2 RAM – 8-Gbyte capacity.• Status: Functional – basic router
performance matches current NetFPGA• Lots of logic for additional functions• Russ Tessier ([email protected])
1 GigE
1 GigE
1 GigE
1 GigE
Stratix IV
SSRAM
DDRRAM
JTAG
SOPC SYSTEM
TSE MAC
TSE MAC
TSE MAC
JTAG BRIDGE
CU
ST
OM
IN
TE
RF
AC
E
PCIe
HO
ST
CO
MP
UT
ER
DE4 BOARD
TSE MAC
MACRxQ
MACRxQ
MACRxQ
MACRxQ
CPURxQ
CPURxQ
CPURxQ
CPURxQ
CPUTxQ
CPUTxQ
CPUTxQ
CPUTxQ
MACTxQ
MACTxQ
MACTxQ
MACTxQ
Input Arbiter
Output portLookup
Output Queue
Free repository available from UMass in September 2011
UMassAmherst
Toronto – August 19, 2011 56 S T A N F O R D U N I V E R S I T Y
FPGAFPGA
MemoryMemory
1GE1GE
1GE1GE
1GE1GE
1GE1GE
Enhancing Modular Reference Designs
PCI
CPUCPU MemoryMemory
Usage #2
NetFPGA DriverNetFPGA Driver
Java GUIFront Panel(Extensible)
Java GUIFront Panel(Extensible)
PW-OSPFPW-OSPF
In QMgmtIn Q
Mgmt
IPLookup
IPLookup
L2Parse
L2Parse
L3Parse
L3Parse
Out QMgmtOut QMgmt
1GE1GE
1GE1GE
1GE1GE
1GE1GE
Verilog modules interconnected by FIFO interfaces
MyBlockMy
Block
VerilogEDA Tools
(Xilinx, Mentor, etc.)
VerilogEDA Tools
(Xilinx, Mentor, etc.)
1. Design2. Simulate3. Synthesize4. Download
1. Design2. Simulate3. Synthesize4. Download
Toronto – August 19, 2011 57 S T A N F O R D U N I V E R S I T Y
FPGAFPGA
MemoryMemory
1GE1GE
1GE1GE
1GE1GE
1GE1GE
Creating new systems
PCI
CPUCPU MemoryMemory
Usage #3
NetFPGA DriverNetFPGA Driver
1GE1GE
1GE1GE
1GE1GE
1GE1GE
My Design
(1GE MAC is soft/replaceable)
My Design
(1GE MAC is soft/replaceable)
VerilogEDA Tools
(Xilinx, Mentor, etc.)
VerilogEDA Tools
(Xilinx, Mentor, etc.)
1. Design2. Simulate3. Synthesize4. Download
1. Design2. Simulate3. Synthesize4. Download
Toronto – August 19, 2011 60 S T A N F O R D U N I V E R S I T Y
To get started with your project
1. Get familiar with hardware description languagehttp://www-netfpga.cl.cam.ac.uk/
Support provided by Redgate Software
Toronto – August 19, 2011 61 S T A N F O R D U N I V E R S I T Y
To get started with your project
2. Prepare for your project
b) Complete a hands-on tutorial
a) Learn NetFPGA by yourself
Toronto – August 19, 2011 62 S T A N F O R D U N I V E R S I T Y
Learn by Yourself
Users Guide
NetFPGA website (www.netfpga.org)
Toronto – August 19, 2011 63 S T A N F O R D U N I V E R S I T Y
Learn by Yourself
Developers Guide
NetFPGA website (www.netfpga.org)
Forums
Toronto – August 19, 2011 64 S T A N F O R D U N I V E R S I T Y
Attend (or host) a hands-on tutorial
Stanford
Cambridge
Toronto – August 19, 2011 65 S T A N F O R D U N I V E R S I T Y
Attend a hands-on tutorial
Events
NetFPGA website (www.netfpga.org)
Toronto – August 19, 2011 66 S T A N F O R D U N I V E R S I T Y
Upcoming Tutorials
EU/CHANGE/OFELIA Half-day NetFPGA IntroductionBerlin, GermanyThursday November 10, 2011 09:00-12:00http://changeofelia.info.ucl.ac.be/
EU/PURSUIT Full-day NetFPGA IntroductionCambridge, UKThursday September 1, 2011 http://www.fp7-pursuit.eu/PursuitWeb/?page_id=463
Toronto – August 19, 2011 68 S T A N F O R D U N I V E R S I T Y
Conclusions
• NetFPGA Provides– Open-source, hardware-accelerated Packet Processing– Modular interfaces arranged in reference pipeline – Extensible platform for packet processing
• NetFPGA Reference Code Provides– Large library of core packet processing functions– Scripts and GUIs for simulation and system operation– Set of Projects for download from repository
• The NetFPGA Base Code– Well defined functionality defined by regression tests– Function of the projects documented in the Wiki Guide
Toronto – August 19, 2011 69 S T A N F O R D U N I V E R S I T Y
Nick McKeown, Glen Gibb, Jad Naous, David Erickson, G. Adam Covington, John W. Lockwood, Jianying Luo, Brandon Heller, Paul
Hartke, Neda Beheshti, Sara Bolouki, James Zeng, Jonathan Ellithorpe, Sachidanandan Sambandan, Eric Lo
Acknowledgments
NetFPGA Team at Stanford University (Past and Present):
NetFPGA Team at University of Cambridge (Past and Present):
Andrew Moore, David Miller, Muhammad Shahbaz, Martin Zadnik
All Community members (including but not limited to):
Paul Rodman, Kumar Sanghvi, Wojciech A. Koszek, Yahsar Ganjali, Martin Labrecque, Jeff Shafer,
Eric Keller , Tatsuya Yabe, Bilal Anwer,Yashar Ganjali, Martin Labrecque
Kees Vissers, Michaela Blott, Shep Siegel
Toronto – August 19, 2011 70 S T A N F O R D U N I V E R S I T Y
Special thanks to our Partners:
Other NetFPGA Tutorial Presented At:
SIGMETRICS
Ram Subramanian, Patrick Lysaght, Veena Kumar, Paul Hartke, Anna Acevedo
Xilinx University Program (XUP)
See: http://NetFPGA.org/tutorials/
Toronto – August 19, 2011 71 S T A N F O R D U N I V E R S I T Y
Thanks to our Sponsors:
• Support for the NetFPGA project has been provided by the following companies and institutions
Disclaimer: Any opinions, findings, conclusions, or recommendations expressed in these materials do not necessarily reflect the views of the National Science Foundation or of any other sponsors supporting this project.