12/4/2002 Lab Board A description of the board you will do measurements on

12/4/2002

Lab Board

A description of the board you will do measurements on

12/4/2002Introduction

2

Lab Board Description Code name is ‘King’s Canyon’

Referred to hereafter as ‘KC’You will do measurements on this board in the lab

Background for KCLow-end server boardInexpensive board for customers who want servers

ServerComputer which serves data to othersSupports more memory, more processing power, and more I/O (i.e., network bandwidth, hard drives) than desktop computersMore reliable (crashes less and recovers better) than desktop computers


3

Server in a network

``

`

Server

Desktop PCPC

PC

Shared Printer

Network Switch


4

Inside the Server

Plug in memory boardsGigabytes of memory

2 CPUs

MCH

Hard drive storage

Network (Ethernet)to network switch

Server Circuit Board

12/4/2002

Circuit Board Block Diagram

Front Side Bus (FSB)

Front Side Bus (FSB)

Hub Interface

Hub Interface

SS

SS

SS

SS

SSSS

SS – Source Synchronous

CC- Common Clock

CC

CC

CC

CC

CC

4 PCI-X busses4 PCI-X busses

Low speed 33MHz PCI bus

Low speed 33MHz PCI bus

CPU 0

CPU 1

MCH

Memory

Memory

Two (2) DDR (Double Data Rate) Memory Busses

Two (2) DDR (Double Data Rate) Memory Busses

ICHVideo

P64H2 HI to PCI bridge

P64H2 HI to PCI bridge

Details to Follow ….

12/4/2002

Components on KC

‘the chips’


7

CPU CPUs –Central Processing Units

The brains of the computer that ‘run’ your programRuns the Operating System (e.g. windows) and all the softwareSpeed of CPU has large effect on speed of system

Currently measured in GHz (e.g., 3.0 GHz)Consumer pc’s typically have 1 cpu inside the box – called ‘UP’

KC has 2 cpus – this is called ‘DP’, i.e. dual processorSome boards have 4+ cpus, they are called ‘MP’, i.e. multi-processor

KC uses Intel Xeon CPU’sCurrently at 3GHzSimilar to Pentium 4

Bigger cache (superfast memory inside CPU) Can operate with other CPUs on same bus

The Silicon die (2) is under this metal heat spreader (1). The die is significantly smaller than the heat spreader. A ‘Fansink’ mounts to the heat spreader to keep die cool.


8

MCH

MCH – Memory Controller HubHelps CPU communicate with rest of system

The ‘gatekeeper’ to the outside world from perspective of CPUTalks to memory, CPU, and I/O

I/O means busses and chips and boards which ultimately connect to things like hard drives, networks, keyboards, monitor, etc..

Maintains memorytells it when to refresh, etc.. (dynamic memory has to be refreshed or it forgets the data it holds)Detects errors in what is stored in memory (ECC) and periodically corrects these errors

KC uses Intel E7501 MCH

http://www.intel.com/design/chipsets/e7501/index.htm?iid=ipp_srvr_proc_xeon+e7501&

Actual Die: Passive (no fan) Heatsink mounts on top.

Package decoupling capacitors. Try to hold voltage rails (vcc) steady. Closer to the die the better.


9

ICH

ICH – Input/Output Controller HubI/O device that deals with critical but slow speed devices

Video controller (basic 2-d video on servers, not meant for 3-d graphics)

Talks to Keyboard, mouse via ‘Super IO’ chip BIOS (contains software which computer runs when it’s

first powered on. BIOS gets computer up and running and then passes control to the OS)

IDE Hard drive (slow hard drive that OS boots off of. The big capacity hard drives do not connect to ICH)

Helps with initial bootup of systemTalks to mch over a slow version of the Hub Interface busKC uses Intel 82801CA ICH

Still uses wire-bond technology. Doesn’t require heatsink.


10

Memory Modules Memory – High speed storage place for code and data used

by CPUMuch faster than hard drive storage, but not as massive in quantity

2-16 Gigabytes vs. 100’s of GigabytesSlower than CPU cache (small amount of memory internal to CPU)

Memory die’s are inside wire-bond packages called ‘DRAMs’Dynamic Random Access Memory

Many DRAMs solder to a memory board called the ‘DIMM’Dual In-line Memory ModuleAnywhere from 9 to 36 DRAMs on one memory board2 types of memory boards: Unbuffered and Registered. Unbuffered is used in desktop. KC (and most servers) only uses Registered DIMMs.

Memory board plugs into KC board via a connector.KC can support up to 8 DIMMs

Many different DIMM manufacturersMicron, Samsung, Elpida,Infineon, …

DRAM

2 DIMMs

http://www.micron.com/products/modules/ddrsdram/index.html


11

PCI-bridge P64H2 – PCIX 64-bit to Hub Interface 2 bridge

Gateway to High Speed I/O devices such as hard drive arrays adapters or ethernet adaptersTalks to MCH on one side

Via high speed variey of ‘Hub Interface’ BusTalks to 2 separate pci-busses on other side

Several connectors connect to each pci busUser can slide a card into each of these connectorsTypically Hard Disk Controller Cards

Each Provides several SCSI or SATA connection to hard drive Can end up with quite a few hard drives and large storage

capacityOr Network Adapater Cards

Provide several 100Mb/s ethernet connections, which go to a switch or router

Servers tend to read data from the hard drive into memory, process the data with the CPUs, and then send it out to the requesting computer via the network (e.g., web server)

‘P64H2’ is the code name for the Intel 82870P2

P64H2 die, may require passive heatsink


12

Identifying parts on the board CPUs

there are two of them and they have really huge fansinks.NOTE: If the fansink doesn’t run, and the system is on, the heatsink will get VERY hot (burn skin) and system will eventually shut itself off.

Memory memory consists of 8 wide connectors all in a row.

At least 2 DIMMs must be plugged in (‘populated’) for system to boot up. There are rules regarding what kind of dimms to use and which order to plug them into the connectors.

MCHIt’s the chip between the cpus and the memoryProbably has a passive heatsink (a small heatsink w/ no fan)

ICH & P64H2’sHard to find, but it looks like the chip pictured in the above slideThere is one ICH and 2 P64H2’s (second one not always populated, i.e. sometimes it’s missing)

PCI SlotsLarger connectors on opposite side of board from DIMM connectors.One slot is from ICH and is slow speed legacy PCI bus (33MHz, 32-bit)The rest are high speed (66-133MHz, 64-bit – i.e. 64 data bits wide)

12/4/2002

Interconnects on KC

‘the wires that connect the chips together’


14

Front Side Bus (FSB) Function: CPUs talk to each other and to the MCH. This is

how the CPUs get data and code from memory, and communicate with rest of the system.

Technology: Source synchronous533MBit/second/wire (266MHz signals)

This means each data wire on this bus can send 533 million bits of data in one second

64 data bits wideSo the whole bus can transfer 64*533 million bits of data per second

Topology: Multi-drop bus. CPU1 has to listen to when CPU0 acceses memory because CPU1 may have that memory data in it’s cache

‘Cache Coherency’ Each cpu has a small superfast memory inside it called the ‘cpu cache’. The cache holds recently accessed memory data. If one cpu holds the data for a memory address that another cpu needs, it has to let it know not to get that data from main memory.

Termination: Parallel termination at both ends of bus (MCH and CPU0)


15

FSB Topology

VTT = 1.5V

50 W

INSIDE DIE

Package Trace Board Trace Board Trace

VTT = 1.5V

50 W

INSIDE DIE

Package Trace

INSIDE DIENote: no 50 ohm

termination

Package Trace

MCHCPU #0

‘Middle Agent’

CPU #1‘End Agent’


16

FSB Signal Groups Data Signal Group

HD[63:0]# – 64 bits of dataDINV[3:0]# - dynamic inversionHDSTBP[3:0]# - P strobesHDSTBN[3:0]# - N strobes

Example, HDSTBP[0]# & HDSTBN[0]# are used to strobe in HD[0-16]# & DINV[0]#

E.g., two strobe signals are used to clock in a set of data bits, more on this later

Address Signal GroupSwitches at ½ the rate of the FSB Data SignalsHAB[35:3]# - Address Bits. E.g., what memory address is being requestedHADSTB[1:0]# - Strobes for address signal group

HADSTB[0]# strobes A[16:3]#, HADSTB[1]# strobes the rest Common Clock Signals

There a whole lot of common clock signalsHCLKINP, HCLKINN – Common clock for all signals in the ‘Host Clock Domain’, i.e. all common clock signals, and outer loop timings for strobes (also called BCLK)


17FSB Timings – 4X Source Synchronous

Timings not published for 533 MHz FSB, 400 MHz timing diagrams shown


18FSB Timings – 2X Source Synchronous

Timings not published for 533 MHz FSB, 400 MHz timing diagrams shown


19DDR (Double Data Rate) Memory Bus Function: MCH reads and writes data from memory over

this high speed bus. Signaling Technology: Source Synchronous

266Mbit/sec/wire (133Mhz signals)A DDR bus is 64 data bits wideKC’s MCH uses 2 DDR busses in parallel, so it likes having a 128-bit wide bus

Bandwidth is 128*266 Million bits per second

Topology:Multi-Drop Bus. Each of the two separate instances of this bus on this system go to four (4) DIMM slots.

Each DIMM can have up to 2GBytes of memory, for a max total of 16GB.

Topology and TimingsThe subject of the next class


20Memory speed and size within a system

CPUL1

L2

L3

CPUL1

L2

L3

MCH MAIN MEMORYMAIN MEMORYMAIN MEMORYMAIN MEMORY

PCI BRIDGE(P64H2)

Adapter Card(s)

Fas

ter

Ret

rieva

l to

CP

U

Greater D

ensity

When we study DDR signaling and timings, it will be clear the overhead and latency that main memory exhibits, and thus the need for local L1-L3 CPU cache.

Hard Drives


21

HI (Hub Interface) Bus Function: Intel proprietary bus to provide flexible high

speed data transfer from MCH to various downstream components

Signaling Technology: Source SynchronousHI1: From MCH to ICH: 266Mbit/sec/wire (133Mhz signals)

8 data bits wide, bandwidth = 8*266 Million bits per second

HI2: From MCH to P64H2’s: 533Mbit/sec/wire (266Mhz signals)

16 data bits wide, bandwidth = 16*533 Million bits per second

Topology: Point to Point. Termination: There are parallel terminations at both ends of the bus


22

Hub Interface Topology

80 W

INSIDE DIE

Package Trace Board Trace

INSIDE DIE

Package Trace

50 W

80 W

50 W

VCC=1.2VCC=1.2


23

Hub Interface Signal Groups

Data BitsHI[15:0] – Data SignalsPSTRB_[1:0], PSTRB_[1:0]# – Strobe Signals

PSTRB_[0] & PSTRB_[0]# strobe in HI[0:7]PSTRB_[1] & PSTRB_[1]# strobe in HI[15:8]

OtherHI[18:16] – Command Signals

Common Clock signals

GCLKIN66MHz common clock for HI[18:16] and outer loops of the PSTRB’s

Timings not published for Hub Interface


24

PCI and PCI-X Function: Industry standard bus for plug-in peripherals

In servers, quite often plug-in hard drive adapters or network adapters are connected to the pci bus

Technology: Common ClockPCI-X: From P64H2 – 133Mbit/sec/wire, clocks are 133MHz signals, data is 66MHz

64 data bits wide, bandwidth = 64*133 Million bits per secondPCI: From ICH – 33Mbit/sec/sire, clocks are 33MHz, data is 16MHz

32 data bits wide, bandwidth = 32*33 Million bits per secondThis speed and width are common in current desktop pc pci slots

Topology: Multi-Drop Bus. The bus consists of devices soldered down on the board and slots for plug-in devices. They all dangle off the PCI bus.

Termination: Series terminated inside the transmitting agent. This signaling technolgy depends on a doubling of the waveform at the end of the bus (due to open-circuit) in order to function.


25

PCIX-133 2-slot Topology

Package Trace

Board Trace

INSIDE DIE

Board Trace

INSIDE DIE INSIDE DIE

Plug-in Board Trace

Plug-in Board Trace

Package Trace

Package Tracex

xx


26

PCI-X Signal Groups

Data and Address use same signals:AD[63:0]Common Clocked

Control SignalsDEVSEL, TRDY, … a bunch moreCommon Clocked

Asynchronous SignalsREQ, GNT – Arbitration signalsInterrupt Signals

ClocksPCLK[6:0] – goes to every device that hangs off the pci bus including the bridge itself


27PCI-X Data to Common Clock Timings

ns

ns


28

PCI-X Common Clock


29

KC Clock Distribution

CK 408 Chip

P64H2 #1

P64H2 #2

CPU #2

MCH

14.31818MHz

66MHz

66MHz

CPU #1133MHz

133MHz133MHz

Differential Clock

Differential ClockDifferential Clock

66MHz

PCI to Gbit Ethernet

PCI BUS B

PCI BUS A

133 MHz

100 MHz

Dual Channel

SCSI

PCI BUS A

PCI BUS A

PCI “Riser Card”

Miscellaneous Other ClocksLPC (low pin count) Legacy BusVideo ClockICH PCI SlotOther Clocks used by ICH...

DDR Bus A Slots

DDR Bus B Slots


30

Magic Decoder Ring Names for Signals given so far are from the device sheets KC Board uses it’s own names

Board’s have to name signals clearly and uniquely across the whole boardDevices only have to name then uniquely across the device

Decoder Ring:FSB: HD[63:0]# FSB_HD63_N

Rule is append FSB_ in front and replace ‘#’ with ‘_N’ in the device signal name

HI: HI[15:0] P64H2_1_HI[15:0]Rule is append P64H2_1_ before the device signal nameNote that the ‘1’ in P64H2_1_ may be a ‘1’ or a ‘2’ because KC has two P64H2 devices

PCI-X: AD[63:0] P64H2_1_PB_AD[63:0]Rule is append P64H2_1_PB to beginning of device signal nameNote that the ‘1’ in P64H2_1_PD.. Can be a ‘2’ or ‘3’ … depending on which pci bus the signal belongs too

DDR: Given Later

Documents

12/4/2002 Lab Board A description of the board you will do measurements on