Serial Network SDRAM

ENEE 759H

Spring 2003

Introduction

SDRAM system drawbacksNo parallelism for memory accessesMultitude of pins for address/command/data

Overall Goals Increase parallelism, reduce latencyReduce pin countAttempt to increase bandwidth

Motivation

Poulton’s idea Bi-directional serial

links. Theoretically high

bandwidth! Less pins required for

same functionality! Looks perfect!

*Graphic from Poulton’s Signaling Tutorial

Evolution I

Initial design Split topology. Effectively halve

latency. Complicated protocol

and connection details.

Memory Controller

Address, etc.

Evolution II

Initial design Individual DRAM chips

directly connected. High overall

bandwidth. Inflexible, lower

capacity for system.

We need a better design!

Memory Controller

8 SDRAMChips

The Next Step

Want simple system interconnects Keep basic SDRAM chip structure intact Utilize the strengths of both parallel and

serial connections Create a system that facilitates parallelism

System Overview

Take a “step back”… Consider memory

module interface. Consider inter-chip

interface on module.

MemoryController

MemoryModules

Serial Lines@ fast clock

System Overview

1 logical channel, 4 physical channels 3.2 GHz point-to-point connections Each channel called “module” 5 pins/module on memory controller Intra-module connections: parallel External connections: high speed serial

Module Topology

MemoryController 256 Mbit x8

SDRAM Parts

CLK1DIN

CLK2DOUTCMD

Memory Module

8 bit, Data-in buses8 bit, Data-out buses18 bit Addr/Cmd buses

Serial Lines@ 3.2 GHz

DIN/DOUT Buffers

Translator Circuits

System Details I

8

8

8

8

8

8

8

8

DIN

DOUT

COMMAND 18

DOUT Translator

DIN Translator

From

DO

UT

Buffer

To D

IN B

ufferT

o SD

RA

M C

hips

System Details IIDOUT Buffer DIN Buffer

8

8

8

8

4, 8-bit registers

8

8

8

8

From SDRAM Chip 0

From SDRAM Chip 1

From SDRAM Chip 2

From SDRAM Chip 3

To SDRAM Chip 0

To SDRAM Chip 1

To SDRAM Chip 2

To SDRAM Chip 3

To

DO

UT T

rans

lato

r

Fro

m D

IN T

rans

lato

r

System Details – Protocol I

The Command SetCMD USE OP ADDR?

NOP No operation. 000 N

ACT Activate a row; uses bank and row address. 001 Y

READ Selects bank/column, initiates read burst. 010 Y

WRITE Select bank and column, initiate write burst. 011 Y

PREC Precharge; deactivate row in bank. 100 *

AUTOR Auto-refresh; enter refresh mode. 101 N

XXX Reserved 110

XXX Reserved 111

System Details – Protocol II

Packets 18 bit command/address 32 bit data packets

COMMAND Activate this row and bank…

0 0 1 0 1 1 1 1 1 0 0 1 0 0 1 1 1 1

COMMAND Start a READ burst at this column…

0 1 0 0 1 1 0 1 1 0 0 1 0 0 1 1 0 0

*Operating at 3.2GHz, command packets take 5.62ns; data packetstake 10ns (the same as SDRAM operating at 100 MHz).

Cubing I

“Chip stacking” Developed by Irvine-

Sensors Corp. Currently can stack

two 256 Mbit chips. Smaller footprint/area! Much shorter

connection wires!

*Graphics from Irvine-Sensors Data Sheet

Cubing II – Serial Network

Point-to-point star topology.

Dedicated circuits -high speed serial lines.

Departure from “traditional” bus concept.

MemoryController

4-stack Cubes

Address/Command lineDOUT lineDIN lineClock line @ 3.2 GHz

System Access Protocol

Consecutive access to same module Similar timing as

SDRAM. Bandwidth matched

between parallel and serial.

DIN/DOUT buffers - no additional timing constraints.

*Graphic from Dr. Jacob and Dave Wang

System Access Protocol

Independent, simultaneous access to separate modules. No inter-module timing

issues.

*Graphic from Dr. Jacob and Dave Wang

Conventional SDRAM:

Serial Network Advantages I

Path length matching No more heroic

routing! Star topology is

symmetric.

No clock mismatch issues… Everyone is on time!

*Graphic from Dr. Jacob and Dave Wang

Serial Network Advantages IIa

No need for bus termination. Point-to-point

communication, terminated in module.

*Graphic from Dr. Jacob and Dave Wang

Serial Network Advantages IIb

Serial/P2P vs. RAMBUS multi-drop. Faster signaling! No ringing! Clean timing. Serial wins…

RAMBUSted!

*Graphic from Dr. Jacob and Dave Wang

System Simulation

SimpleScalarSingle CPU, Single ThreadSNSDRAM(32 Meg x 8)1 rank in every memory moduleChannel width : 32 bitsOne extra cycle of Transaction Queue Delay

to model the parallel to serial conversion

Simulation Run I - Parallel Bus

Channel Rank Per Channel Sim_Cycles

1 1 884521

1 2 881421

1 4 880361

1 8 880361

Simulation Run I - Serial Network

Channel Rank Per Channel Sim_Cycles

1 1 885291

2 1 805721

4 1 766711

8 1 766711

Simulation I Cycles Chart

700

750

800

850

900

Total Cycles in

Thousands

1 2 4 8

Number of Channels (Serial Link)Number of Ranks (Parallel Bus)

test-printf

Serial Link

Parallel Bus

Simulation Run II – Parallel Bus

Channel Rank Per Channel Sim_Cycles

1 1 13206613

2 1 13169500

4 1 13144737

8 1 13144737

Simulation Run II – Serial Network

Channel Rank Per Channel Sim_Cycles

1 1 13264603

2 1 12633349

4 1 12510912

8 1 12510912

Simulation II Cycles Chart

1200012200124001260012800130001320013400

Total Cycles in

Thousands

1 2 4 8

Number of Channels (Serial Link)Number of Ranks (Parallel Bus)

test-printf

Serial Link

Parallel Bus

Memory Mapping

Basic SDRAM

High Performance SDRAM

Row ID Rank Bank Hi Col ID Channel ID Lo Col ID Col Size

Rank Row ID Bank Hi Col ID Channel ID Lo Col ID Col Size

Analysis

Cache line = 64 byte channel width Read after Read Multi-CPU Single CPU Multi-Thread

Summary I

Recall… SDRAM has complex

interface, simple chips. RDRAM has a simple

interface, but very complex chips.

SNSDRAM… Blends these seemingly

split philosophies!

Summary II

AdvantagesSmaller pin count on memory controller. Independent memory modules facilitate

parallelism.Simulated performance improvement over

similar SDRAM configurations.Smaller system footprint with cubing

technology.Theoretically scalable.