Accessing the NIC

A look at the mechanisms that software can use to interact with our 82573L network interface

nic

Typical NIC hardware

TX FIFO

RX FIFO

transceiver LANcableB

US

main memory

packet

buffer

CPU

No way to communicate

Linux operating system kernel(no knowledge of the NIC)

Network interface controller(no knowledge of the OS)

HARDWARE

SOFTWARE

These components lack a way to interact

Role for a ‘device driver’

Linux operating system kernel(no knowledge of the NIC)

Network interface controller(no knowledge of the OS)

HARDWARE

SOFTWARE

device driver module (knows about both the OS and the NIC)

Three x86 address-spaces

memoryspace(4GB)

i/o space(64KB)

PCIconfiguration

space(16MB)

accessed using a large variety of processor instructions (mov, add, or, shr, push, etc.) and virtual-to-physical address-translation

accessed only by using the processor’s special ‘in’ and ‘out’ instructions (without any translation of port-addresses)

i/o-ports 0x0CF8-0x0CFF dedicated to accessing PCI Configuration Space

reserved

Interface to PCI Configuration Space

CONFADD( 0x0CF8)

CONFDAT( 0x0CFC)

31 23 16 15 11 10 8 7 2 0

EN

bus(8-bits)

device(5-bits)

doubleword (6-bits)

function(3-bits) 00

PCI Configuration Space Address Port (32-bits)

PCI Configuration Space Data Port (32-bits)

31 0

Enable Configuration Space Mapping (1=yes, 0=no)

82573L

• Two mechanisms for accessing the NIC:– I/O space (allows booting over the network)– Memory-mapped I/O (available after booting)

• Both mechanisms require probing for PCI Configuration Space information (for I/O port-number or memory-mapped address)

• Probing requires knowing the device’s IDs (the VENDOR_ID and the DEVICE_ID)

Our NIC’s ID numbers

• The VENDOR_ID for Intel Corporation:– 0x8086 (famous chip-number for IBM-PCs)

• The DEVICE_ID for Intel’s 82573L NIC:– 0x109A (found in Intel’s documentation p.141)

NIC’s access mechanisms

• The two ways for accessing our network controller’s ‘control’ and ‘status’ registers are explained in Chapter 13 of the Intel Open Source Programmer’s Reference

• This Chapter also includes a detailed list of the names and functional descriptions for the various network interface registers

• All are accessed as 32-bit ‘doubewords’

PCI Configuration Space

PCI Configuration Space Body(48 doublewords – variable format)

64doublewords

PCI Configuration Space Header(16 doublewords – fixed format)

A non-volatile parameter-storage area for each PCI device-function

PCI Configuration Header

StatusRegister

CommandRegister

DeviceID

VendorID

BISTCacheLineSize

Class CodeClass/SubClass/ProgIF

RevisionID

Base Address 0

SubsystemDevice ID

SubsystemVendor ID

CardBus CIS Pointer

reservedcapabilities

pointer Expansion ROM Base Address

MinimumGrant

InterruptPin

reserved

LatencyTimer

HeaderType

Base Address 1

Base Address 2Base Address 3

Base Address 4Base Address 5

InterruptLine

MaximumLatency

31 0 31 0

16 doublewords

Dwords

1 - 0

3 - 2

5 - 4

7 - 6

9 - 8

11 - 10

13 - 12

15 - 14

reserved

Interface to PCI Configuration Space

CONFADD( 0x0CF8)

CONFDAT( 0x0CFC)

31 23 16 15 11 10 8 7 2 0

EN

bus(8-bits)

device(5-bits)

doubleword (6-bits)

function(3-bits) 00

PCI Configuration Space Address Port (32-bits)

PCI Configuration Space Data Port (32-bits)

31 0

Enable Configuration Space Mapping (1=yes, 0=no)

Reading PCI Configuration Data

• Step one: Output the desired longword’s address (bus, device, function, and dword) with bit 31 set to 1 (to enable access) to the Configuration-Space Address-Port

• Step two: Read the designated data from the Configuration-Space Data-Port:# read the PCI Header-Type field (byte 2 of dword 3) for bus=0, device=0, function=0

movl $0x8000000C, %eax # setup address in EAXmovw $0x0CF8, %dx # setup port-number in DX outl %eax, %dx # output address to port

mov $0x0CFC, %dx # setup port-number in DXinl %dx, %eax # input configuration longwordshr $16, %eax # shift word 2 into AL registermovb %al, header_type # store Header Type in variable

Demo Program

• We created a short Linux utility that searches for and reports all of your system’s PCI devices

• It’s named “pciprobe.cpp” on our CS686 website• It uses some C++ macros that expand to Intel

input/output instructions -- which normally are ‘privileged’ instructions that a Linux application-program is not allowed to execute (segfault!)

• Our system administrator (Alex Fedosov) has created a utility (named “iopl3”) that will allow your command-shell to acquire I/O privileges

Example: network interface

• We identify the network interface controller in our classroom PC’s by class-code 0x02

• The subclass-code 0x00 is for ‘ethernet’• We can identify the NIC from its VENDOR and

DEVICE identification-numbers:• VENDOR_ID = 0x8086 (for Intel Corporation)• DEVICE_ID = 0x109A (for 82573L controller)

• You can use the ‘grep’ command to search for these numbers in this header-file:

</usr/src/linux/include/linux/pci_ids.h>

The NIC’s PCI ‘resources’

StatusRegister

CommandRegister

DeviceID0x109A

VendorID0x8086

BISTCacheLineSize

Class CodeClass/SubClass/ProgIF

RevisionID

Base Address 0

SubsystemDevice ID

SubsystemVendor ID

CardBus CIS Pointer

reservedcapabilities

pointer Expansion ROM Base Address

MinimumGrant

InterruptPin

reserved

LatencyTimer

HeaderType

Base Address 1

Base Address 2Base Address 3

Base Address 4Base Address 5

InterruptLine

MaximumLatency

31 0 31 0

16 doublewords

Dwords

1 - 0

3 - 2

5 - 4

7 - 6

9 - 8

11 - 10

13 - 12

15 - 14

Linux PCI helper-functions

#include <linux/pci.h>

struct pci_dev *devp; unsigned int mmio_base; unsigned int mmio_size; void *io;

devp = pci_get_device( VENDOR_ID, DEVICE_ID, NULL );if ( devp == NULL ) return –ENODEV;

mmio_base = pci_resource_start( devp, 0 );mmio_size = pci_resource_len( devp, 0 );io = ioremap_nocache( mmio_base, iomm_size );if ( io == NULL ) return –ENOSPC;

Mechanisms compared

kernel memory-space

NIC i/o-memory

CPU’s ‘virtual’ address-space

io

user memory-space

Each NIC register has its own address in memory (allows one-step access)

addr data

Access to all of the NIC’s registers is muliplexed through a pair of I/O-ports(requires multiple instructions)

CPU’s ‘I/O’ address-space

C language hides complexity

• For the multiplexed i/o-space access…

// Your module uses just one C statement to ‘input’ a NIC register-value:int device_status = inl( ioport + 0x0008 );

// but the C compiler translates this statement into SIX cpu-instructions:

mov ioport, %dxmov $8, %eaxout %eax, %dx

add $4, %dxin %dx, %eaxmov %eax, device_status

Seeing through the C

• For the i/o-memory ‘mapped’ access…

// Your module uses just one C statement to ‘fetch’ a NIC register-value:int device_status = ioread32( io + 0x0008 );

// but the C compiler translates this statement into three cpu-instructions:

mov io, %esimov 8(%esi), %eaxmov %eax, device_status

Course’s continuing theme is…

“Using the computer to study the computer”

TimeStamp Counter

• The x86 processor has a 64-bit register named the TimeStamp Counter (TSC)

• It continuously increments each cpu cycle (so with a 2-GHz processor, it increments two-billion times every second)

• It can be read at any time using a special processor instruction (named RDTSC); its value appears in the EDX:EAX registers

Using CLI and STI

• For doing accurate timing measurements, your module can temporarily disable your CPU’s response to ‘interrupt’ requests

Basic steps for performing an uninterrupted ‘elapsed time’ measurement: step 1: Turn off interrupts (using ‘cli’) step 2: Read the TimeStamp Counter step 3: Perform a NIC register access step 4: Read the TimeStamp Counter step 5: Turn on interrupts (using ‘sti’) step 6: Subtract start-time from finish-time

In-class exercise

• Look at our ‘timing.c’ demo module – for an ‘inline’ assembly language example using the ‘rdtsc’, ‘cli’ and ‘sti’ instructions

• Add some extra code of your own, and do an additional timing measurement, so you can compare the execution-times for the NIC’s two register-access mechanisms

• How much faster is memory-mapped I/O?