More 82573L details

Getting ready to write and test a character-mode device-driver for our

anchor-LAN’s ethernet controllers

A ‘nic.c’ character driver?

open

read

write

my_fops

ioctl my_ioctl()

my_open()

my_read()

my_write()

my_release()

my_isr()

module_init() module_exit()

release

Statistics registers

• The 82573L has several dozen statistical counters which automatically operate to keep track of significant events affecting the ethernet controller’s performance

• Most are 32-bit ‘read-only’ registers, and they are automatically cleared when read

• Your module’s initialization routine could read them all (to start counting from zero)

Initializing the nic’s counters

• The statistical counters all have address- offsets in the range 0x04000 – 0x04FFF

• You can use a very simple program-loop to ‘clear’ each of these read-only registers

// Here ‘io’ is the virtual base-address of the nic’s i/o-memory region{

int r;

// clear all of the Pro/1000 controller’s statistical countersfor (r = 0x4000; r < 0x4FFF; r += 4) ioread32( io + r );

}

A few ‘counter’ examples

0x4000 CRCERRS CRC Errors Count0x400C RXERRC Receive Error Count0x4014 SCC Single Collision Count0x4018 ECOL Excessive Collision Count0x4074 GPRC Good Packets Received0x4078 BPRC Broadcast Packets Received0x407C MPRC Multicast Packets Received0x40D0 TPR Total Packets Received0x40D4 TPT Total Packets Transmitted0x40F0 MPTC Multicast Packets Transmitted0x40F4 BPTC Broadcast Packets Transmitted

the packet’s data ‘payload’ goes here(usually varies from 56 to 1500 bytes)

Ethernet packet layout

• Total size normally can vary from 64 bytes up to 1536 bytes (unless ‘jumbo’ packets and/or ‘undersized’ packets are enabled)

• The NIC expects a 14-byte packet ‘header’ and it appends a 4-byte CRC check-sum

destination MAC address (6-bytes)

source MAC address(6-bytes)

Type/length(2-bytes)

Cyclic RedundancyChecksum (4-bytes)

0 6 12 14

Filter registers

• All the modern ethernet controllers have a built-in ‘filtering’ capability which allows the NIC to automatically discard any packets having a destination-address different from the controller’s own unique MAC address

• But the 82573L offers a more elaborate filtering mechanism (and can also ‘reject’ packets based on the ‘source’ addresses)

How ‘receive’ works

descriptor0descriptor1descriptor2descriptor3

0000

Buffer0

Buffer1

Buffer2

Buffer3

List of Buffer-Descriptors

We setup memory-buffers where we want received packets to be placed by the NIC

We also create a list of buffer-descriptors and inform the NIC of its location and size

Then, when ready, we tell the NIC to ‘Go!’ (i.e., start receiving), but to let us know when these receptions have occurred

Random Access Memory

Receive Control (0x0100)

R=0

0 0FLXBUFSE

CRCBSEX R

=0PMCF DPF R

=0CFI

CFIEN

VFE BSIZE

BAM

R=0

MO DTYP RDMTS

ILOS

SLU

LPE UPE 0 0 R=0

15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0

SBPEN

31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16

LBM MPE

EN = Receive Enable DTYP = Descriptor Type DPF = Discard Pause Frames SBP = Store Bad Packets MO = Multicast Offset PMCF = Pass MAC Control FramesUPE = Unicast Promiscuous Enable BAM = Broadcast Accept Mode BSEX = Buffer Size ExtensionMPE = Multicast Promiscuous Enable BSIZE = Receive Buffer Size SECRC = Strip Ethernet CRCLPE = Long Packet reception Enable VFE = VLAN Filter Enable FLXBUF = Flexible Buffer sizeLBM = Loopback Mode CFIEN = Canonical Form Indicator EnableRDMTS = Rx-Descriptor Minimum Threshold Size CFI = Cannonical Form Indicator bit-value

Registers’ Names

Memory-information registers RDBA(L/H) = Receive-Descriptor Base-Address Low/High (64-bits) RDLEN = Receive-Descriptor array Length RDH = Receive-Descriptor Head RDT = Receive-Descriptor Tail

Receive-engine control registers RXDCTL = Receive-Descriptor Control Register RCTL = Receive Control Register

Notification timing registers RDTR = Receive-interrupt packet Delay Timer

RADV = Receive-interrupt Absolute Delay Value

Rx-Desc Ring-Buffer

Circular buffer (128-bytes minimum)

RDBA base-address

RDLEN (in bytes)

RDH (head)

RDT (tail)

= owned by hardware (nic)

= owned by software (cpu)

0x00

0x10

0x20

0x30

0x40

0x50

0x60

0x70

0x80

Rx-Descriptor Control (0x2828)

R=0

R=0

R=0

R=0

R=0

R=0

R=0

GRAN

R=0

R=0

1---------

0

ADVD3

WUC

SDP1DATA---------

0

SDP0DATA---------D/UDstatus

0WTHRESH(Writeback Threshold)

R=0

R=0

0 FRCDPLX

FRCSPD 0HTHRESH

(Host Threshold)R=0

R=0

ASDE

0LRST

0 0

15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0

0 0

31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16

PTHRESH(Prefetch Threshold)

GRAN (Granularity): 1=descriptor-size, 0=cacheline-size

Prefetch Threshold – A prefetch operation is considered when the number of valid, but unprocessed, receive descriptors that the ethernet controller has in its on-chip buffer drops below this threshold.

Host Threshold - A prefetch occurs if at least this many valid descriptors are available in host memory

Writeback Threshold - This field controls the writing back to host memory of already processed receive descriptors in the ethernet controller’s on-chip buffer which are ready to be written back to host memory

Legacy Rx-Descriptor Layout

VLAN tag

0x0

0x4

0x8

0xC

Packet Checksum

Buffer-Address high (bits 63..32)

Buffer-Address low (bits 31..0)

31 0

Packet Length (in bytes)

StatusErrors

Buffer-Address = the packet-buffer’s 64-bit address in physical memory Packet Length = number of bytes in the data-packet that has was received Packet Checksum = the16-bit one’s-complement of the entire logical packet Status = shows if descriptor has been used and if it’s last in a logical packet Errors = valid only when DD and EOP are set in the descriptor’s Status field

Suggested C syntax

typedef struct {unsigned long long base_addr;unsigned short pkt_length;unsigned short checksum;unsigned char desc_stat;unsigned char desc_errs;unsigned short vlan_tag;} rx_descriptor;

RxDesc Status-field

PIF IPCS TCPCS VP IXSM EOP DD

7 6 5 4 3 2 1 0

DD = Descriptor Done (1=yes, 0=no) shows if nic is finished with descriptor EOP = End Of Packet (1=yes, 0=no) shows if this packet is logically last IXSM = Ignore Checksum Indications (1=yes, 0=no) VP = VLAN Packet match (1=yes, 0=no) USPCS = UDP Checksum calculated in packet (1=yes, 0=no) TCPCS = TCP Checksum calculated in packet (1=yes, 0=no) IPCS = IPv4 Checksum calculated on packet (1=yes, 0=no) PIF = Passed In-exact Filter (1=yes, 0=no) shows if software must check

UDPCS

RxDesc Error-field

RXE IPE TCPE reserved=0 SE CE

7 6 5 4 3 2 1 0

RXE = Received-data Error (1=yes, 0=no) IPE = IPv4-checksum error TCPE = TCP/UDP checksum error (1=yes, 0=no) SEQ = Sequence error (1=yes, 0=no) SE = Symbol Error (1=yes, 0=no) CE = CRC Error or alignment error (1=yes, 0=no)

SEQreserved=0

Network Administration

• Some higher-level networking protocols require the Operating System to setup a translation between the ‘hostname’ for a workstation and the hardware-address of its Network Interface Controller

• One mechanism for doing this is creation of a specially-named textfile (‘/etc/ethers’) that provides database for translations

In-class exercise #1

• We put a file named ‘ethers’ on our course website that offers a template for defining the translation database that software can consult on our ‘anchor’ cluster’s LAN

• One of the eight workstations’ entries has been filled in already:

• Can you complete this database by adding the MAC addresses for the other 7 machines?

00:30:48:8A:30:03 anchor00.cs.usfca.edu

Our ‘seereset.c’ demo

• We created this LKM to demonstrate the sequence of ‘state-changes’ that three of our network controller’s registers undergo in response to initiating a ‘reset’ operation

• The programming technique used here is one which we think could be useful in lots of other hardware programming situations where a vendor’s manual may not answer all our questions about how devices work

In-class exercise #2

• Try redirecting the output from this ‘cat’ command to a file, like this:

$ cat /proc/seereset > seereset.out

• Then edit this textfile, adding a comment to each line which indicates the bit(s) that experienced a ‘change-of-state’ from the line that came before it (thereby providing yourself with a running commentary as to how the NIC proceeds through a ‘reset’)