Cisco Systems CAE.
Fibre Channel Basics
Cisco Systems CAE.
SCSI I/O Channel
Small Computer System Interface - SCSI
SCSI is a standard that defines an interface between an initiator (usually a computer) and a target (usually a storage device such as a hard disk)
Logical Unit Number (LUN): A 64-bit field within SCSI that identifies the Logically addressable Unit within a target SCSI device
Host (Initiator) Disk (Target)
READ DATA DATA DATA
WRITE DATA DATA DATA
STATUS
STATUS
SCSI I/O Channel STATUS STATUS
Disk (Target) Host (Initiator)
SCSI READ OPERATION
SCSI WRITE OPERATION READY
Cisco Systems CAE.
The SCSI I/O Channel SCSI I/O Channel provides half-
duplex pipe for SCSI command, data, and status
SCSI I/O channel can be internal or external to host
Multiple SCSI I/O channels can exist within host
A network approach can scale the I/O channel in many areas (length, devices, throughput)
NIC Adapter SCSI Adapter
NIC Driver
TCP/IP Stack
Adapter Driver
SCSI Generic
Block Device
File System
Applications Half-Duplex I/O Channel
SCSI
SCSI Initiator
SCSI Target
Cisco Systems CAE.
Fibre Channel Fabric
SCSI
Target
Fibre Channel HBA
Host System
Initiator
Fibre Channel I/O Networking Fibre Channel provides high speed transport
for SCSI payload via Host Buss Adapter (HBA)
Fibre Channel overcomes many shortcomings of Parallel I/O including: Addressing for up to 16 million nodes Loop (shared) and fabric (switched) transport Host speeds of 100 to 400 MBps (1–4 Gbps) Segments of up to 10km (without extenders) Support for multiple protocols
Combines best attributes of a channel and a network together
Cisco Systems CAE.
Consolidated I/O Networking Traditional Ethernet with Fibre
Channel over Ethernet Consolidated Network Adapter
– (CAN) Native Ethernet NIC Native FC HBA
Relies on Lossless Ethernet PAUSE per IEEE 802.1p
Able to build Ethernet-based SANs using FCoE arrays
FCoE Host
(Initiator)
FCoE (Target)
Storage (Target)
Ethernet Network
FC Fabric FC HBA
Attached Host (Initiator)
FCoE
FCoE
CNA
SCSI
Cisco Systems CAE.
Fibre Channel Communications Point-to-point oriented Facilitated through device login
N_Port-to-N_Port connection Logical node connection point
Flow controlled Buffer-to-buffer credits and end-to-end basis
Acknowledged For certain classes of traffic, none for others
Multiple connections allowed per device
N_Port Receiver
N_Port
Node Transmitter
Receiver
Link
Transmitter
Node
Cisco Systems CAE.
Fibre Channel Port Types ‘N’ port: Node ports used to connect devices to
switched fabric or point to point configurations ‘F’ port: Fabric ports residing on switches
connecting ‘N’ port devices ‘L’ port: Loop ports are used in arbitrated loop
configurations to build networks without FC switches; these ports often also have ‘N’ port capabilities and are called ‘NL’ ports
‘E’ port: Expansion ports are essentially trunk ports used to connect two Fibre Channel switches
‘GL’ port: A generic port capable of operating as either an ‘E’ or ‘F’ port; it’s also capable of acting in an ‘L’ port capacity; Auto Discovery
N N
N F
NL FL
L L
E E
Cisco Systems CAE.
FL_Port
G_Port
G_Port
G_Port
E_Port
F_Port
F_Port
F_Port
E_Port
Node N_Port
Node N_Port
Node N_Port
Node NL_Port
Node NL_Port
Node NL_Port
Fibre Channel Port Types Fibre Channel Switch
Fabric Switch
Input Port
Output Port
Fabric X
Cisco Systems CAE.
Inter-Switch Link (ISL)
The interconnection between switches is called the ISL E_Port to E_Port (‘Expansion port)
Supports all classes of service Class 1, 2, 3, and a special Class F (switch-to-switch)
FC-PH permits consecutive frames of a sequence to be routed over different ISL links for maximum throughput
Cisco’s implementation is to dedicate an FC_ID pair and/or a given exchange to an ISL bundle member to guarantee in-order delivery for exchange/sequence frames
Cisco Extended ISL (EISL, TE port)
EISL
Cisco Systems CAE.
Worldwide Names
Each switch element is assigned a WWN at time of manufacture
Each switch port is assigned a WWN at the time of manufacture
During FLOGI the switch identifies the WWN in the service parameters of the accept frame and assigns a Fibre Channel ID (FCID)
These address assignments can then correlate each fabric port with the switch element
Cisco Systems CAE.
8 Bits
00 00 Arbitrated Loop Physical Address (AL_PA)
Private Loop Device Address Model
Switch Topology Model
Switch Domain Area Device
8 Bits 8 Bits
Public Loop Device Address Model
Switch Domain Area
Arbitrated Loop Physical Address (AL_PA)
Fabric Addressing FC_ID Address Model
FC_ID address models help speed up routing Switches assign FC_ID addresses to N_Ports Some addresses are reserved for fabric services Private loop devices only understand 8-bit address (0x0000xx) FL_Port can provide proxy service for public address translation Maximum switch domains = 239 (based on standard)
Cisco Systems CAE.
Fabric Routing: FSPF
FSPF routes traffic based on destination domain ID For FSPF a domain ID identifies a single switch
This limits the max number of switches that can support in the Fabric to 239 when FSPF is supported
FSPF performs hop-by-hop routing FSPF uses total cost as the metric to determine most
efficient path FSPF supports hierarchical path selection
Provides the scalable routing tables in large topologies
Static routes can be applied
Cisco Systems CAE.
16 Km
1 Gbps FC
4 Gbps FC ~ ½ km per Frame
~ 2 km per Frame
Buffer to Buffer Credit Flow Control BB_Credits and Distance
BB_Credits are used to ensure enough FC frames in flight A full (2112 byte) FC frame is approx 2 km long at 1 Gbps, 1 km long at 2 Gbps and ½ km long at 4 Gbps As distance increases, the number of available BB_Credits need to increase as well Insufficient BB_Credits will throttle performance—no data will be transmitted until R_RDY is returned
2 Gbps FC ~ 1 km per Frame
Cisco Systems CAE.
Fibre Channel Fabric Zoning
Zoning operation Zone members “see” only other members of the zone Zones can be configured dynamically based on WWN Devices can be members of more than one zone FC-AL zoning allows the creation of private loops on a single hub Switched fabric zoning can take place at the port or device level Based on physical switch port Based on device WWN Based on LUN ID
Benefits Secured device access Allows operating system co-existence
SAN
Disk1
Host2 Disk4
Host1
Disk2 Disk3
ZoneA
ZoneB
ZoneC
Cisco Systems CAE.
Virtual SANs (VSANs)
Analogous to VLANs in Ethernet Virtual fabrics created from larger cost-
effective redundant physical fabric Reduces wasted ports of island approach Fabric events are isolated per VSAN—
maintains isolation for HA (i.e. RSCNs) Hardware-based isolation—traffic is explicitly
tagged across interswitch links with VSAN membership info
Statistics can be gathered per VSAN
Physical SAN Islands Are Virtualized onto Common SAN Infrastructure
Cisco MDS 9000 Family with VSAN Service A Virtual SAN (VSAN) Provides a
Method to Allocate Ports within a Physical Fabric to Create Virtual Fabrics
Cisco Systems CAE.