A look at memory issues
Data-transfers must occur between system memory and the
network interface controller
Typical Chipset Layout
MCHMemory Controller Hub
(Northbridge)
ICHI/O Controller Hub
(Southbridge)
CPUCentral Processing Unit
DRAMDynamic RandomAccess Memory
NICNetwork Interface
Controller
HDCHard Disk Controller
ACAudio Controller
GraphicsController
Timer Keyboard Mouse Clock
MultimediaController
FirmwareHub
USBcontroller
Typical Chipset Layout
MCHMemory Controller Hub
(Northbridge)
ICHI/O Controller Hub
(Southbridge)
CPUCentral Processing Unit
DRAMDynamic RandomAccess Memory
NICNetwork Interface
Controller
HDCHard Disk Controller
ACAudio Controller
GraphicsController
Timer Keyboard Mouse Clock
MultimediaController
FirmwareHub
USBcontroller
DMA
PCI Bus Master DMA82573L i/o-memory
RX and TX FIFOs(32-KB total)
Host’s Dynamic Random Access Memory
Descriptor Queue
packet-buffer
packet-buffer
packet-buffer
packet-buffer
packet-buffer
packet-buffer
packet-buffer
DMA
on-chip RX descriptors
on-chip TX descriptors
Memory-mapped I/O
• We mentioned that Intel’s x86 architecture originally was designed with two separate address-spaces, one for memory and the other for I/O ports, unlike the designs for CPUs by many of Intel’s competitors in which I/O access was “memory-mapped”
• But now the newer Intel processors also can support memory-mapped I/O as well
Address-bus widths
• The Intel Core-2 Quad processors in our classroom and Lab machines potentially could address 236 physical memory cells (i.e., 64GB), although only 4GB of RAM actually are installed at the present time
• Some PCI-compliant hardware devices were designed for a 32-bit address-bus, thus they must be “mapped” below 4G
Physical-address assignments
The CPU’s physical address-space
Devices’ registers must be mapped to addresses in the bottom 4G
Dynamic Random Access Memory
Virtual addresses
• Software running on the x86 processor is unable to use actual memory addresses, but instead uses ‘virtual’ addresses that map to physical addresses by means of mapping-tables which Linux dynamically defines for each different process it runs
• This complicates the steps software must take to arrange for the DMA to take place
Our ‘dram.c’ module
• To help us confirm that our hardware-level network software is working as we intend, or to diagnose our ‘bugs’ if it isn’t, we can use an LKM we’ve written that implements a character-mode device-driver for system memory, allowing us to view the contents of physical memory as if it were a file; for example, by using our ‘fileview.cpp’ tool
How to view system memory
• Download ‘dram.c’ from course website
• Compile it using our ‘mmake’ utility
• Install ‘dram.ko’ by using ‘/sbin/insmod
• Insure the ‘/dev/dram’ device-node exists
• Download ‘fileview.cpp’ from website and compile it with ‘g++’ (or with ‘make’)
• Execute ‘fileview /dev/dram’ and use the arrow-keys to navigate (or hit <ENTER>)
“canonical” addresses0000000001000100001100100001010011000111
01000010010101001011011000110101110011111000010001100101001110100101011011010111
1100011001110101101111100111011111011111
“non-canonical”(invalid)
virtual addresses
“canonical” addresses
“canonical” addresses
0xFFFFFFFFFFFFFFFF … 0xFFFF800000000000
0x00007FFFFFFFFFFF … 0x0000000000000000
Analogy using 5-bit values
64-bit “vrtual”address space
4-Levels of mapping
PageMap
Level-4Table
CR3
PageDirectoryPointerTable
PageDirectory
PageTable
PageFrame(4KB)
offset
64-bit ‘canonical’ virtual address
sign-extension PML4 PDPT PDIR PTBL 63 48 47 39 38 30 29 21 20 12 11 0
Each mapping-table contains up to 512 quadword-size entries
4-level address-translation
• The CPU examines any virtual address it encounters, subdividing it into five fields
offset into page-frame
index into page-table
63 48 47 39 38 30 29 21 20 12 11 0
16-bits 9-bits 12-bits
index into page-
directory
index into page-
directorypointer table
index intolevel 4
page-maptable
9-bits9-bits9-bits
sign-extension
Any 48-bit virtual-address is sign-extended to a 64-bit “canonical” address
Only “canonical” 64-bit virtual-addresses are legal in 64-bit mode
Reserved(must be 0)
Format of 64-bit table-entries
63 62 52 51 40 39 32
31 12 11 9 8 7 6 5 4 3 2 1 0
Page-frame physical base-address[31..12]
Page-framephysical
base-address[39..32]
EXB
PWUPWT
PCD
A
avl
avl
Meaning of these bits varies with the table
Legend: P = Present (1=yes, 0=no) PWT = Page Cache Disable (1=yes, 0=no) W = Writable (1=yes, 0=no) PWT = Page Write-Through (1=yes, 0=no) U = User-page (1=yes, 0=no) avl = available for user-defined purposes A = Accessed (1=yes, 0=no) EXB = Execution-disabled Bit (if EFER.NXE=1)
Our ‘mem64.c’ module
• We wrote an LKM to create a pseudo-file that will let us see how the virtual memory is being utilized by an application program
• Download this file, compile it with ‘mmake’ and install ‘mem64.ko’ in the Linux kernel
• Then view the virtual-memory mapping that is being used by the ‘cat’ program:
$ cat /proc/mem64
The NIC’s PCI ‘resources’
StatusRegister
CommandRegister
DeviceID0x109A
VendorID0x8086
BISTCacheLineSize
Class CodeClass/SubClass/ProgIF
RevisionID
Base Address 0
SubsystemDevice ID
SubsystemVendor ID
CardBus CIS Pointer
reservedcapabilities
pointer Expansion ROM Base Address
MinimumGrant
InterruptPin
reserved
LatencyTimer
HeaderType
Base Address 1
Base Address 2Base Address 3
Base Address 4Base Address 5
InterruptLine
MaximumLatency
31 0 31 0
16 doublewords
Dwords
1 - 0
3 - 2
5 - 4
7 - 6
9 - 8
11 - 10
13 - 12
15 - 14
Mechanisms compared
kernel memory-space
NIC i/o-memory
CPU’s ‘virtual’ address-space
io
user memory-space
Each NIC register has its own address in memory (allows one-step access)
addr data
Access to all of the NIC’s registers is muliplexed through a pair of I/O-ports(requires multiple instructions)
CPU’s ‘I/O’ address-space
‘nicstatus.c’
• Here’s an LKM that creates a pseudo-file (called ‘/proc/nicstatus’) which will allow a user to view the current value in our Intel 82573L Network Interface Controller’s ‘DEVICE_STATUS’ register
• It uses the I/O-port interface to the NIC’s registers, rather than a ‘memory-mapped’ interface to those device-registers
0
Device Status (0x0008)
? 0 0 0 0 0 0 0 0 0 0 0GIO
MasterEN
0 0 0
0 0 0 0 PHYRA ASDV
ILOS
SLU
0 TXOFF 0 0
FD
15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0
FunctionID
LU
31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16
SPEED
FD = Full-DuplexLU = Link UpTXOFF = Transmission PausedSPEED (00=10Mbps,01=100Mbps, 10=1000Mbps, 11=reserved)ASDV = Auto-negotiation Speed Detection ValuePHYRA = PHY Reset Asserted
82573L
some undocumented functionality?
‘82573.c’
• This is a more elaborate example of an LKM which not only creates a pseudo-file (i.e., ‘/proc/82573’) that we can view using the Linux ‘cat’ command and that lets us see our NIC’s PCI Configuration Space, but also implements some device-driver functions that let us view the NIC’s device registers by using our ‘fileview.cpp’ tool
Linux PCI helper-functions
#include <linux/pci.h>
struct pci_dev *devp; unsigned int mmio_base; unsigned int mmio_size; void *io;
devp = pci_get_device( VENDOR_ID, DEVICE_ID, NULL );if ( devp == NULL ) return –ENODEV;
mmio_base = pci_resource_start( devp, 0 );mmio_size = pci_resource_len( devp, 0 );io = ioremap_nocache( mmio_base, iomm_size );if ( io == NULL ) return –ENOSPC;
In-class exercise
• Two of the NIC’s 32-bit device registers are used to hold its 48-bit Ethernet MAC address – which will be a different value for each of the hosts in our classroom
• These two registers are located at offsets 0x5400 and 0x5404 in device-memory
• The six bytes occur in network byte-order
• Write code to show the MAC address!