Gullik Webjörn

Gigabit EthernetGigabit Ethernet

1 of 17

Terena Summer 1999

Session Overview

What is Gigabit Ethernet ?– Technology– Media

Gigabit Ethernet Applications– Computer rooms– Server farms– Cluster

Problems and Solutions Gigabit Ethernet– DMD, differential mode delay– Collision domains

2 of 17

What is Gigabit Ethernet?

Ethernet with a speed of 1 Gbit/sek.– IEEE 802.3z– Both CSMA/CD and Full Duplex– Both fiber and copper ( Fiber Channel, 8b/10b )– Follows the same development rate as 100 Mbit

Ethernet but converges much faster. Suggested ‘improvements’ and adaptions

– Jumbo Frames ( 9K bytes )– VLAN– Protocol stack parameter modifications.

3 of 17

1000baseFiber

Multimode fiber, both 50/125 och 62/125. – Installed fiber in buildings and campuses– 850 nM = 1000base SX , 220 - 500 m (cost)– 1300nM = 1000baseLX , 550 m (distance)

Singlemode fiber 9/125– Metropolitan networks and large campus net’s– 5000 m with 1000baseLX optics– 10000 m with “hot optics”– 80000 m with “proprietary optics”

4 of 17

1000baseCopper

STP / Coax - 1000baseCX - 802.3z– 25 m over 2 150 ohm pairs / 2 75 ohm Coaxes– Relatively low cost and simple– Preferably in the Computer Room– 8b/10b coding and simple transmission at 1250

baud Cat 5 UTP - 1000baseT - 802.3ab

– 100 m and uses all 4 pairs– Complicated technology, DSP, PAM-5, FEC– Most standardized copper media– 250 Mbit / 125 Mbaud * 4 par

5 of 17

Problems and Solutions, GBE

Pure scaling of Ethernet to 1 Gbit – Slottime becomes 512 nS– Max network diameter becomes 28 - 43 m– Hard to achieve efficiency at half duplex

Increased slot-time– 4 uSek slot-time creates 200m collision domain– Packet bursting increases efficiency

Full Duplex– Limited only by transmission characteristics– Capacity = 2 * half duplex– Wire-speed now becomes realistic

6 of 17

Problems and Solutions, GBE

DMD, differential mode delay – In multi-mode fiber discrete modes can dominate– The problem increases with shorter wavelength– The problem increases with better coherence– Worst at 850 nM and 62/125 fiber– Reduces distance due to jitter and ISI

But, the problem might be overrated... – 802.3z June 98 meeting– 1000baseSX gets reduced distance (220/62/160)– 1000baseLX gets 550 m over all MMF– 1000baseLX increased from 3 to 5 km over SMF

7 of 17

Full Duplex and Switching

More common than CSMA/CD 1998 already– The arbitration problem goes away.– Suitable for silicon based switches and routers– Greater freedom to choose topology– At least twice the performance of half duplex– Possibility to switch ‘cut-through’– Enables clusters with low latency and high

bandwidth.– Full duplex is available today– NO more HUBS!

8 of 17

Synergy effects

Disks and Networks– Both based on “fiber channel”– Server area networks -- 250 Mbyte/sec FDX– Storage networks -- U-SCSI and GBE– Common components results in a nice

price/performance improvement (GBIC, t.ex.) Metropolitan Networks

– The technology with best price/performance– 10000’s of users at 100 kbit/sec– 100’s concurrent MPEG-2 video streams– Makes multicast video possible (BC IP/TV)

9 of 17

Adapters and Server performance

Gigabit adapters need bus bandwidth – 64 bit (and 66 Mhz ) PCI– Proprietary busses of abt. 200 Mbytes/sek– 1518 bytes is too short a packetlength– A full GBE @ 1518 bytes = 81000 packets/sec/HDX

Performance examples– High end Intel server std stack -- 270 Mbit/sec

Tested in house with 4 clients.– Alpha Server 4100 400-500 Mbit/sec– Alpha Server using Driver API - wire speed

Your mileage will vary !!!

10 of 17

Cluster and Gigabit Ethernet

Redundant PHY -- media fault tolerance

Need or wish...

Available over FDDI but not over Fast Ethernet

Available in switches, when do we get a NIC ? Jumbo frames -- storage transport efficiency

Ethernet and Fast Ethernet 1500 bytes

FDDI - 4478 bytes

GBE - 9000 bytes Cut through -- Lock management efficiency

Efficient lock mgmt with large # of nodes.

11 of 17

Quality of Service

As for other technologies: in the network boxes– Not a specific characteristic of GBE– High bandwidth reduces the need– IEEE 802.1p and 802.1Q - frame tagging– Networks have barely caught up with CPU’s

Common with level 2,3,4 switching in GBE-switches.– Not really QOS– Allows priority for time critical services– Or, you can route such traffic a different path

12 of 17

Present and Future

Hunt groups– 2-many aggregated links– Redundancy, multi links - single logical port

10 Gbit Ethernet – 2.5 Gbit single pipe– 4 * 2.5 Gbit / sec WDM– 10 Gbit single pipe

Aggressive cost curve - 1999 projections– Shared Gigabit -- $450 - $700 per port (NOT !)– Switched Gigabit -- $1000 - $1600 per port (!!)– All switch or nic functions on single CMOS ASIC

13 of 17

Present and Future

1550 nM GBIC’s– 40-100 km on single mode fiber– Will fit almost any GBE box with modular phy

GBIC’s are getting cheap...– 850 nm at $ 300 list– 1300 nm at $ 900 list– Fixed optics is going away…..

VLAN is going WAN– 802.1 p and Q over packet over SONET– SONET/SDH is no more “the only way”– Black colors are coming…with optical muxes

14 of 17

Compaq SW54xx Gigabit Products

• Policy-Based QoS • Layer-3 IP Routing

(RIP, RIP2, OSPF)• 256 VLANs per switch• 802.1p and Q• 128,000 MAC addresses• Port bonding (4 ports)

SW5422 -- 16 10/100 Base-TX and 6 Gigabit SX PortsSW5411 -- 8 10/100 Base-TX and 3 Gigabit GBIC Ports with 1 redundant phySW5406 -- 6 Gigabit SX and 2 GBIC PortsSW5450 -- 48 10/100 Base-TX and 2 Gigabit GBIC Ports with 2 redundant physSW5425 -- 24 10/100 Base-TX and 1 Gigabit GBIC Port with 1 redundant phy

Features:

Backbone Layer 2/3/4 switching 8 slot chassis (16 slot coming soon)

– Over 30 Gb/s switching throughput– Gigabit Ethernet and Fast Ethernet

connections via modules– Future WAN interface modules

Deploy as a backbone interconnection between switches and as a connection to high-performance servers.

Industrial-strength, GIGAswitch-class performance and reliability for high-speed backbones.

GIGAswitch/RouterGIGAswitch/Router

GIGAswitch/Router

Independently Verified– 16 Gbps, 15 Million pps

56 10/100 ports14 Gigabit portsFuture Support for

ATM, FDDI, SONET, WAN

Independently Verified– 32Gbps, 30 Million pps

120 10/100 ports30 Gigabit portsFuture Support for

ATM, FDDI, SONET, WAN

Two GIGAswitch/Router ChassisTwo GIGAswitch/Router Chassis

8-Slot Chassis 16-Slot Chassis

Compaq Computer Corporation© 1999