Fabric System Architecture Design: two distinct board designs; replicate and connect –modularity...

Fabric System Architecture

• Design: two distinct board designs; replicate and connect– modularity allows

us to build any configuration of size 2n

• Board 1: Quad Raw Board

• Board 2: I/O & Memory Board

The Saman Flip

• How do we use the same board designs for every position in the fabric?– Quad board

• symmetric in both dimensions

– IO Board• symmetric about x-axis• compensate for board flip in firmware

Quad Board

•4 RAW chips per board

•16 152-pin MICTOR connectors total (4 per side)

•Power distributed over separate cables from other signals

•Connectors are stacked to save space

Quad Board LayoutAnant, I created the graphic in the previous slide to overlay exactly on this one. Their placement on the slide is also the same. I recommend fading from the previous slide to this one, so the audience can make the conceptual leap from the block diagram to the real layout.

11”

11”

I/O & Memory Board

•4 FPGAs

•2 64-bit PCI slots

•2 Expansion Ports (same as on Raw Handheld board)

•4 SDRAM banks

•symmetric design (thanks Saman!)

11”

Here’s the IO Board schematic in case you want to use it…

AD[63:0]PCI CONNECTORS

1

AMP

INFORMATION SCIENCES INTSTITUTE

5

4

3

2

1

5

4

3

2

1

A B

A B

SCHEMATIC NAME:

FAX (703) 812-3712

TEL (703) 243-9423

ARLINGTON, VA 22203

SHEET SIZE E

Copyright, UNIVERSITY OF SOUTHERN CALIFORNIA

PROJECT NAME:

3811 NORTH FAIRFAX DRIVE

SUITE #200

SHEET OF 30

MPD[15:0]

MEM_A[20:0]

MEM_D[15:0]

MPA[6:0]

CONFIGURATION CONTROLLER

M1DQ[63:0] M1CB[15:0]M1DQMB[7:0]M1S_N[3:0] M1A[13:0] M0DQ[63:0] M0CB[15:0]M0DQMB[7:0]M0S_N[3:0] M0A[13:0]

SDRAM PCI 1

M1DQ[63:0] M1CB[15:0]M1DQMB[7:0]M1S_N[3:0] M1A[13:0] M0DQ[63:0] M0CB[15:0]M0DQMB[7:0]M0S_N[3:0] M0A[13:0]

SDRAM EXP 0

M1DQ[63:0] M1CB[15:0]M1DQMB[7:0]M1S_N[3:0] M1A[13:0] M0DQ[63:0] M0CB[15:0]M0DQMB[7:0]M0S_N[3:0] M0A[13:0]

SDRAM PCI 0

AD[63:0]PCI CONNECTORS

IO0_[189:0]

IO1_[189:0]

EXPANSION CONNECTORS

IO0_[189:0]

IO1_[189:0]

EXPANSION CONNECTORS

UTILITY

CLOCKS

POWER

MPA[6:0]

MPD[15:0]

MEM_A[20:0]

MEM_D[15:0]

CONFIGURATION MEMORY

M1DQ[63:0] M1CB[15:0]M1DQMB[7:0]M1S_N[3:0] M1A[13:0] M0DQ[63:0] M0CB[15:0]M0DQMB[7:0]M0S_N[3:0] M0A[13:0]

SDRAM EXP 1

M1DQ[63:0] M1CB[15:0]M1DQMB[7:0]M1S_N[3:0] M1A[13:0] M0DQ[63:0] M0CB[15:0]M0DQMB[7:0]M0S_N[3:0] M0A[13:0]

PIE_[36:0]

POE_[36:0]

POD_[36:0]

PID_[36:0]

AD[63:0] FPGA PCI 1

M1DQ[63:0] M1CB[15:0]M1DQMB[7:0]M1S_N[3:0] M1A[13:0] M0DQ[63:0] M0CB[15:0]M0DQMB[7:0]M0S_N[3:0] M0A[13:0]

PIE_[36:0]

POE_[36:0]

POD_[36:0]

PID_[36:0]

IO0_[189:0]

IO1_[189:0]

FPGA EXP 1

M1DQ[63:0] M1CB[15:0]M1DQMB[7:0]M1S_N[3:0] M1A[13:0] M0DQ[63:0] M0CB[15:0]M0DQMB[7:0]M0S_N[3:0] M0A[13:0]

PIE_[36:0]

POE_[36:0]

POD_[36:0]

PID_[36:0]

IO0_[189:0]

IO1_[189:0]

FPGA EXP 0

M1DQ[63:0] M1CB[15:0]M1DQMB[7:0]M1S_N[3:0] M1A[13:0] M0DQ[63:0] M0CB[15:0]M0DQMB[7:0]M0S_N[3:0] M0A[13:0]

PIE_[36:0]

POE_[36:0]

POD_[36:0]

PID_[36:0]

AD[63:0] FPGA PCI 0

PO6_[36:0]

PI7_[36:0]

PI5_[36:0]

PO5_[36:0]

PO4_[36:0]

PI3_[36:0]

PO3_[36:0]

PO2_[36:0]

PO1_[36:0]

PI2_[36:0]

PI1_[36:0]

PI0_[36:0]

PO0_[36:0]

PI4_[36:0]

PO7_[36:0]

PI6_[36:0]

Connectors

QUAD RAW IO BOARD

REV#=1.0RAW

QUAD_RAW_IO

7-10-2003_12:55

MPA[6:0]

MPD[15:0]

MEM_A[20:0]

EXP1_IO0_[189:0]

PCI1_AD[63:0]

PCI0_M0DQ[63:0]

PCI0_M0A[13:0]

PCI0_M0S_N[3:0]

PCI0_M1DQB[7:0]

PCI0_M1S_N[3:0]

EXP0_M1S_N[3:0]

EXP0_M1A[13:0]

EXP0_M1DQB[7:0]

EXP0_M1DQ[63:0]

EXP0_M1CB[15:0]

EXP0_M0S_N[3:0]

EXP0_M0A[13:0]

EXP0_M0DQB[7:0]

EXP1_M1S_N[3:0]

EXP1_M1DQB[7:0]

PCI1_M1S_N[3:0]

PCI1_M0DQ[63:0]

PCI1_M0CB[15:0]

PO7_[36:0]

PI7_[36:0]

PO6_[36:0]

PI6_[36:0]

PO5_[36:0]

PI5_[36:0]

PO4_[36:0]

PI4_[36:0]

PO3_[36:0]

PI3_[36:0]

PO2_[36:0]

PI2_[36:0]

PO1_[36:0]

PI1_[36:0]

PO0_[36:0]

PI0_[36:0]

PCI1_M0DQB[7:0]

PCI1_M0A[13:0]

PCI1_M0S_N[3:0]

PCI1_M1DQ[63:0]

PCI1_M1CB[15:0]

PCI1_M1DQB[7:0]

PCI1_M1A[13:0]

EXP1_M0DQ[63:0]

EXP1_M0CB[15:0]

EXP1_M0DQB[7:0]

EXP1_M0A[13:0]

EXP1_M0S_N[3:0]

EXP1_M1DQ[63:0]

EXP1_M1CB[15:0]

EXP1_M1A[13:0]

EXP0_M0DQ[63:0]

EXP0_M0CB[15:0]

PCI0_M0CB[15:0]

PCI0_M0DQB[7:0]

PCI0_M1DQ[63:0]

PCI0_M1CB[15:0]

Clock Distributionfrom external input

DLL

• Synchronized clocks for all Raw chips in fabric • Delay-Locked Loop uses feedback to tune delay line for clock

synchronization• Dip switches keep clock dist. general no custom firmware

Power Distribution

• 48V distributed to all boards, then down-converted

• DC-DC converters on each board– 1.8V Raw core– 1.5V Raw I/O– 3V other logic– 1.5V is also further down converted to 0.75V

supply for HSTL termination

• System-wide power supply can be up to 3kW

Power Distribution

• distributed separately from signals

• external power supply feeds top and bottom rows of I/O Boards

power supply

Clock Distribution

• signal generated and distributed from a center board over MICTOR connectors

• uses PLLs to deskew the clock at each connection

• every quad board sends and receives a copy of the clock to its neighbors and we can select which of the input clocks to use using dip switches

clock generator

Reset Distribution

• signal generated by one of the I/O boards and distributed over MICTOR connectors

reset originates

here