ISCSI Design Tr0106

  • View
    8

  • Download
    0

Embed Size (px)

DESCRIPTION

Design, Implementation, and PerformanceAnalysis of Session Layer Protocolsfor SCSI over TCP/IP

Text of ISCSI Design Tr0106

  • Design, Implementation, and Performance

    Analysis of Session Layer Protocols

    for SCSI over TCP/IP

    Anshul Chadda

    Robert D. Russell

    TR 01-06

    August 2001

  • ii

    Table of Contents

    List of Tables ...................................................................................................................... v List of Figures .................................................................................................................... vi ABSTRACT..................................................................................................................... viii CHAPTER PAGE 1. Introduction..................................................................................................................... 1

    1.1 Motivation and Goal for this Thesis ......................................................................... 1 1.2 Resources Used......................................................................................................... 1 1.3 Organization of the Thesis ........................................................................................ 2

    2. SCSI, NAS, and SAN ..................................................................................................... 3

    2.1 SCSI (Small Computer System Interface) ................................................................ 3 2.1.1 Phase sequences in the SCSI Protocol ............................................................... 4 2.1.2 Tasks .................................................................................................................. 6 2.1.3 Command Descriptor Block (CDB)................................................................... 6 2.1.4 Task Management Functions ............................................................................. 7

    2.2 Network Attached Storage (NAS) ............................................................................ 8 2.3 Storage Area Network (SAN)................................................................................... 9 2.4 SAN Approaches .................................................................................................... 10

    2.4.1 SCSI on top of SAN on top of Ethernet........................................................... 10 2.4.2 SCSI on top of SAN on top of IP..................................................................... 11 2.4.3 SCSI on top of SAN on top of UDP ................................................................ 11 2.4.4 SCSI on top of SAN on top of TCP................................................................. 12

    2.5 SAN Approach for this thesis ................................................................................. 12 3. Session Layer Protocols................................................................................................ 15

    3.1 SCSI Encapsulation Protocol (SEP) ....................................................................... 15 3.2 Internet SCSI (iSCSI) ............................................................................................. 17

    3.2.1 Steps involved in iSCSI Protocol to support I/O operation ............................. 17 3.2.2 iSCSI PDU format ........................................................................................... 20 3.2.3 Sequence Numbering ....................................................................................... 20 3.2.4 Error Recovery................................................................................................. 23 3.2.5 Important Operational Parameters ................................................................... 23

    4. Fast Kernel Tracing....................................................................................................... 25

    4.1 FKT Setup......................................................................................................... 25 4.2 Application Programmer Interface ......................................................................... 26 4.3 Kernel Programmer Interface ................................................................................. 27 4.4 Recording and Printing ........................................................................................... 27

  • iii

    5. SCSI SubSystem in Linux ............................................................................................ 29

    5.1 SCSI Layers in Linux.............................................................................................. 29 5.1.1 Upper level....................................................................................................... 29 5.1.2 Middle level ..................................................................................................... 30 5.1.3 Low-level drivers ............................................................................................. 30

    5.2 Important SCSI Data Structures.............................................................................. 30 5.2.1 Scsi_Host_Template struct .............................................................................. 30 5.2.2 Scsi_Host struct .......................................................................................... 34 5.2.3 Scsi_Cmnd struct ............................................................................................. 35

    5.3 Discovery Process for SCSI Targets....................................................................... 37 6. SEP Initiator Design And Implementation ................................................................... 38

    6.1 Overview of SEP Low-Level Driver (LLD) Design............................................... 38 6.2 sep_config ............................................................................................................... 40 6.3 Data structures involved in SEP LLD..................................................................... 40 6.4 The SCSI_Host_Template Implementation............................................................ 42

    7. iSCSI Initiator Design And Implementation................................................................. 46

    7.1 Overview of iSCSI Low-Level Driver (LLD) Design............................................ 46 7.2 Data structures involved in iSCSI LLD .................................................................. 48

    7.2.1 session struct .................................................................................................... 49 7.2.2 connection struct .............................................................................................. 51 7.2.3 command struct ................................................................................................ 51

    7.3 The SCSI_Host_Template Implementation............................................................ 53 7.4 Low-level iSCSI Driver Design.............................................................................. 57

    7.4.1 Processing a WRITE SCSI Command............................................................. 57 7.4.2 Processing a READ SCSI Command .............................................................. 61 7.4.3 Processing an ABORT Command ................................................................... 62

    8. Performance Analysis ................................................................................................... 64

    8.1 Test Set-Up ............................................................................................................. 64 8.1.1 CPUs ................................................................................................................ 64 8.1.2 Ethernet Technologies ..................................................................................... 64 8.1.3 Fiber_Channel Technologies ........................................................................... 65 8.1.4 Version of Linux Operating System ................................................................ 65

    8.2 Performance Metrics ............................................................................................... 65 8.3 Performance Variables............................................................................................ 65 8.4 Accuracy of Data .................................................................................................... 67

    8.4.1 Side Effects of Adding Probes......................................................................... 67 8.4.2 Confidence Level of Data ................................................................................ 67

    8.5 Ethernet Payload ..................................................................................................... 68 8.6 Performance Results for SEP.................................................................................. 68

    8.6.1 Effect of Target Domain on Bandwidth for SEP............................................. 69

  • iv

    8.6.2 Effect of Target Block Size on Bandwidth for SEP ........................................ 72 8.6.3 Effect of Initiator Scatter Gather List Size on Bandwidth for SEP ................. 74 8.6.4 Effect of Ethernet Link Speed on Bandwidth for SEP .................................... 75 8.6.5 Effect of Ethernet Packet Size on Bandwidth for SEP .................................... 76 8.6.6 Effect of Coalescing Interrupt Time Interval on Bandwidth for SEP.............. 77 8.6.7 Effect of Old (Alteon) and New (3-Com) Acenic Cards on Performance....... 78

    8.7 Performance Results for iSCSI ............................................................................... 78 8.7.1 Effect of LLD Queuing Length on Bandwidth for iSCSI................................ 79 8.7.2 Effect of Max PDU Size on Bandw