Memory Hierarchy

Computer Organization and Assembly Languages Yung-Yu Chuang 2007/01/08with slides by CMU15-213

Announcement• Please submit homeworks if you haven’t by 1/10.• Final project demo time on 1/24 or 1/31. The mechanism for signup will be mailed to the mailing list soon.• Mail your report to TA before your demo time. The length of report depends on your project type. It can be html, pdf, doc, ppt…• TA evaluation today

Reference• Chapter 6 from “Computer System: A

Programmer’s Perspective”

Computer system model• We assume memory is a linear array which

holds both instruction and data, and CPU can access memory in a constant time.

SRAM vs DRAM

Tran. Access Needs per bit time refresh? Cost Applications

SRAM 4 or 6 1X No 100X cache memories

DRAM 1 10X Yes 1X Main memories,frame buffers

The CPU-Memory gap The gap widens between DRAM, disk, and CPU

speeds.

110

1001,000

10,000100,000

1,000,00010,000,000

100,000,000

1980 1985 1990 1995 2000

year

ns

Disk seek timeDRAM access timeSRAM access timeCPU cycle time

register cache memory diskAccess time

(cycles)

1 1-10 50-100 20,000,000

Memory hierarchies• Some fundamental and enduring

properties of hardware and software:– Fast storage technologies cost more per byte,

have less capacity, and require more power (heat!).

– The gap between CPU and main memory speed is widening.

– Well-written programs tend to exhibit good locality.

• They suggest an approach for organizing memory and storage systems known as a memory hierarchy.

Memory system in practice

Larger, slower, and cheaper (per byte) storage devices

registers

on-chip L1cache (SRAM)

main memory(DRAM)

local secondary storage(local disks)

remote secondary storage(tapes, distributed file systems, Web servers)

off-chip L2cache (SRAM)

L0:

L1:

L2:

L3:

L4:

L5:

Smaller, faster, and more expensive (per byte) storage devices

Why does it work?• Most programs tend to access the storage

at any particular level more frequently than the storage at the lower level.

• Locality: tend to access the same set of data items over and over again or tend to access sets of nearby data items.

Why learning it?• A programmer needs to understand this

because the memory hierarchy has a big impact on performance.

• You can optimize your program so that its data is more frequently stored in the higher level of the hierarchy.

• For example, the difference of running time for matrix multiplication could up to a factor of 6 even if the same amount of arithmetic instructions are performed.

Locality• Principle of Locality: programs tend to reuse

data and instructions near those they have used recently, or that were recently referenced themselves.– Temporal locality: recently referenced items are

likely to be referenced in the near future.– Spatial locality: items with nearby addresses

tend to be referenced close together in time.• In general, programs with good locality run faster then programs with poor locality• Locality is the reason why cache and virtual memory are designed in architecture and operating system. Another example is web browser caches recently visited webpages.

Locality example

• Data– Reference array elements in succession (stride-

1 reference pattern):– Reference sum each iteration:

• Instructions– Reference instructions in sequence:– Cycle through loop repeatedly:

sum = 0;for (i = 0; i < n; i++)

sum += a[i];return sum;

Spatial locality

Spatial locality

Temporal locality

Temporal locality

Locality example• Being able to look at code and get a

qualitative sense of its locality is important. Does this function have good locality?

int sum_array_rows(int a[M][N]){ int i, j, sum = 0;

for (i = 0; i < M; i++) for (j = 0; j < N; j++) sum += a[i][j]; return sum;} stride-1 reference pattern

Locality example• Does this function have good locality?

int sum_array_cols(int a[M][N]){ int i, j, sum = 0;

for (j = 0; j < N; j++) for (i = 0; i < M; i++) sum += a[i][j]; return sum;} stride-N reference pattern

Memory hierarchies

Larger, slower, and cheaper (per byte) storage devices

registers

on-chip L1cache (SRAM)

main memory(DRAM)

local secondary storage(local disks)

remote secondary storage(tapes, distributed file systems, Web servers)

off-chip L2cache (SRAM)

L0:

L1:

L2:

L3:

L4:

L5:

Smaller, faster, and more expensive (per byte) storage devices

Cache memories• Cache memories are small, fast SRAM-

based memories managed automatically in hardware.

• CPU looks first for data in L1, then in L2, then in main memory.

• Typical system structure:

mainmemory

I/Obridgebus interfaceL2 data

ALU

register fileCPU chip

SRAM Port system busmemory busL1

cache

Caching in a memory hierarchy

0 1 2 3

4 5 6 7

8 9 10 11

12 13 14 15

Larger, slower, cheaper Storage device at level k+1 is partitioned into blocks.

Data is copied between levels in block-sized transfer units

8 9 14 3Smaller, faster, more Expensive device at level k caches a subset of the blocks from level k+1

level k

level k+1

4

4

4 10

10

10

Request14

Request12

General caching concepts• Program needs object d, which is

stored in some block b.• Cache hit

– Program finds b in the cache at level k. E.g., block 14.

• Cache miss– b is not at level k, so level k cache

must fetch it from level k+1. E.g., block 12.

– If level k cache is full, then some current block must be replaced (evicted). Which one is the “victim”?

• Placement policy: where can the new block go? E.g., b mod 4

• Replacement policy: which block should be evicted? E.g., LRU

9 3

0 1 2 3

4 5 6 7

8 9 10 11

12 13 14 15

level k

level k+1

1414

12

14

4*

4*12

12

0 1 2 3

Request12

4*4*12

Type of cache misses• Cold (compulsory) miss: occurs because the

cache is empty.• Capacity miss: occurs when the active cache

blocks (working set) is larger than the cache.• Conflict miss

– Most caches limit blocks at level k+1 to a small subset of the block positions at level k, e.g. block i at level k+1 must be placed in block (i mod 4) at level k.

– Conflict misses occur when the level k cache is large enough, but multiple data objects all map to the same level k block, e.g. Referencing blocks 0, 8, 0, 8, 0, 8, ... would miss every time.

General organization of a cache

• • • B–110

• • • B–110

valid

valid

tag

tagset 0:

B = 2b bytesper cache block

E lines per set

S = 2s sets

t tag bitsper line

Cache size: C = B x E x S data bytes

• • •

• • • B–110

• • • B–110

valid

valid

tag

tagset 1: • • •

• • • B–110

• • • B–110

valid

valid

tag

tagset S-1: • • •

• • •

Cache is an arrayof sets.

Each set containsone or more lines.

Each line holds ablock of data.

1 valid bit per line

Addressing caches

t bits s bits b bits

<tag> <set index><block offset>

0m-1

Address A:

• • •B–110

• • •B–110

v

v

tag

tagset 0: • • •

• • •B–110

• • •B–110

v

v

tag

tagset 1: • • •

• • •B–110

• • •B–110

v

v

tag

tagset S-1: • • •

• • • The word at address A is in the cache ifthe tag bits in one of the <valid> lines in set <set index> match <tag>.

The word contents begin at offset <block offset> bytes from the beginning of the block.

Addressing caches

t bits s bits b bits

<tag> <set index><block offset>

0m-1

Address A:

• • •B–110

• • •B–110

v

v

tag

tagset 0: • • •

• • •B–110

• • •B–110

v

v

tag

tagset 1: • • •

• • •B–110

• • •B–110

v

v

tag

tagset S-1: • • •

• • • 1. Locate the set based on <set index>

2. Locate the line in the set based on <tag>

3. Check that the line is valid4. Locate the data in the line based on

<block offset>

Direct-mapped cache• Simplest kind of cache, easy to build

(only 1 tag compare required per access)• Characterized by exactly one line per set.

valid

valid

valid

tag

tag

tag• • •

set 0:

set 1:

set S-1:

E=1 lines per setcache block

cache block

cache block

Cache size: C = B x S data bytes

Accessing direct-mapped caches• Set selection

– Use the set index bits to determine the set of interest.

t bits s bits0 0 0 0 1

0m-1

b bits

tag set index block offset

selected setvalid

valid

valid

tag

tag

tag• • •

set 0:

set 1:

set S-1:

cache block

cache block

cache block

Accessing direct-mapped caches• Line matching and word selection

– Line matching: Find a valid line in the selected set with a matching tag

– Word selection: Then extract the word

t bits s bits100i0110

0m-1

b bits

tag set index block offset

selected set (i): 1 0110 w3w0 w1 w2

30 1 2 74 5 6

=1?(1) The valid bit must be set

= ?(2) The tag bits in the

cache line must match the tag bits in the address

If (1) and (2), then cache hit

Accessing direct-mapped caches• Line matching and word selection

– Line matching: Find a valid line in the selected set with a matching tag

– Word selection: Then extract the word

t bits s bits100i0110

0m-1

b bits

tag set index block offset

selected set (i): 1 0110 w3w0 w1 w2

30 1 2 74 5 6

(3) If cache hit,block offset selects starting byte.

Direct-mapped cache simulationM=16 byte addresses, B=2 bytes/block, S=4 sets, E=1 entry/set

Address trace (reads):0 [00002], 1 [00012], 7 [01112], 8 [10002], 0 [00002]

xt=1 s=2 b=1

xx x

0 ? ?v tag data

miss

1 0 M[0-1]

hitmiss

1 0 M[6-7]

miss

1 1 M[8-9]

miss

1 0 M[0-1]

What’s wrong with direct-mapped?float dotprod(float x[8], y[8]) { float sum=0.0; for (int i=0; i<8; i++) sum+= x[i]*y[i]; return sum;}

element address set element address setx[0] 0 0 y[0] 32 0x[1] 4 0 y[1] 36 0x[2] 8 0 y[2] 40 0x[3] 12 0 y[3] 44 0x[4] 16 1 y[4] 48 1x[5] 20 1 y[5] 52 1x[6] 24 1 y[6] 56 1x[7] 28 1 y[7] 60 1

two setsblock size=16 bytes

Solution? padding

element address set element Address setx[0] 0 0 y[0] 48 1x[1] 4 0 y[1] 52 1x[2] 8 0 y[2] 56 1x[3] 12 0 y[3] 60 1x[4] 16 1 y[4] 64 0x[5] 20 1 y[5] 68 0x[6] 24 1 y[6] 72 0x[7] 28 1 y[7] 76 0

float dotprod(float x[12], y[8]) { float sum=0.0; for (int i=0; i<8; i++) sum+= x[i]*y[i]; return sum;}

Set associative caches• Characterized by more than one line per

set

E=2lines per set

valid tagset 0:

set 1:

set S-1:

• • •

cache blockvalid tag cache block

valid tag cache blockvalid tag cache block

valid tag cache blockvalid tag cache block

E-way associative cache

Accessing set associative caches• Set selection

– identical to direct-mapped cachevalidvalid

tagtagset 0:

validvalid

tagtagset 1:

validvalid

tagtagset S-1:

• • •

cache blockcache block

cache blockcache block

cache blockcache block

t bits s bits0 0 0 0 1

0m-1

b bits

tag set index block offset

selected set

Accessing set associative caches• Line matching and word selection

– must compare the tag in each valid line in the selected set.

1 0110 w3w0 w1 w2

1 1001selected set (i):

30 1 2 74 5 6

t bits s bits100i0110

0m-1

b bits

tag set index block offset

=1?(1) The valid bit must be set

= ?

(2) The tag bits in one of the cache lines must match the tag bits in the address

If (1) and (2), then cache hit

Accessing set associative caches• Line matching and word selection

– Word selection is the same as in a direct mapped cache

1 0110 w3w0 w1 w2

1 1001selected set (i):

30 1 2 74 5 6

t bits s bits100i0110

0m-1

b bits

tag set index block offset

(3) If cache hit,block offset selects starting byte.

2-Way associative cache simulation

M=16 byte addresses, B=2 bytes/block, S=2 sets, E=2 entry/set

Address trace (reads):0 [00002], 1 [00012], 7 [01112], 8 [10002], 0 [00002]

xxt=2s=1 b=1

x x

0 ? ?v tag data000

miss

1 00 M[0-1]

hitmiss

1 01 M[6-7]

miss

1 10 M[8-9]

hit

Why use middle bits as index?

• High-order bit indexing–adjacent memory

lines would map to same cache entry

–poor use of spatial locality

4-line Cache High-OrderBit Indexing

Middle-OrderBit Indexing

00011011

0000000100100011010001010110011110001001101010111100110111101111

0000000100100011010001010110011110001001101010111100110111101111

What about writes?• Multiple copies of data exist:

– L1– L2– Main Memory– Disk

• What to do when we write?– Write-through– Write-back (need a dirty bit)

• What to do on a replacement?– Depends on whether it is write through or write

back

Multi-level caches• Options: separate data and instruction

caches, or a unified cache

size:speed:$/Mbyte:line size:

200 B3 ns

8 B

8-64 KB3 ns

32 B

128 MB DRAM60 ns$1.50/MB8 KB

30 GB8 ms$0.05/MB

larger, slower, cheaper

MemoryRegs

UnifiedL2

Cache

Processor

1-4MB SRAM6 ns$100/MB32 B

disk

L1 d-cache

L1 i-cache

Intel Pentium III cache hierarchy

Processor Chip

L1 Data1 cycle latency

16 KB4-way assoc

Write-through32B lines

L1 Instruction16 KB, 4-way

32B lines

Regs. L2 Unified128KB--2 MB4-way assocWrite-back

Write allocate32B lines

MainMemory

Up to 4GB

The memory mountain• Read throughput: number of bytes read

from memory per second (MB/s)• Memory mountain

– Measured read throughput as a function of spatial and temporal locality.

– Compact way to characterize memory system performance.

void test(int elems, int stride) { int i, result = 0; volatile int sink; for (i = 0; i < elems; i += stride)

result += data[i]; /* So compiler doesn't optimize away the loop */ sink = result;}

The memory mountain/* Run test(elems, stride) and return read throughput (MB/s) */double run(int size, int stride, double Mhz){ double cycles; int elems = size / sizeof(int);

test(elems, stride); /* warm up the cache */ /* call test(elems,stride) */ cycles = fcyc2(test, elems, stride, 0); /* convert cycles to MB/s */ return (size / stride) / (cycles / Mhz);}

The memory mountain

s1

s3

s5

s7

s9

s11

s13

s15

8m

2m 512k 12

8k 32k 8k

2k

0

200

400

600

800

1000

1200

L1

L2

mem

xeSlopes ofSpatialLocality

Pentium III550 MHz

16 KB on-chip L1 d-cache16 KB on-chip L1 i-cache

512 KB off-chip unifiedL2 cache

Ridges ofTemporalLocality

Working set size(bytes)Stride (words)

Thro

ughp

ut (M

B/se

c)

Ridges of temporal locality• Slice through the memory mountain

(stride=1)– illuminates read throughputs of different

caches and memory

0

200

400

600

800

1000

1200

8m 4m 2m

1024

k

512k

256k

128k 64

k

32k

16k 8k 4k 2k 1k

working set size (bytes)

read

thro

ugpu

t (M

B/s

)

L1 cacheregion

L2 cacheregion

main memoryregion

L2 is unified

callingoverheadis notamortized

A slope of spatial locality• Slice through memory mountain

(size=256KB)– shows cache block size.

0

100

200

300

400

500

600

700

800

s1 s2 s3 s4 s5 s6 s7 s8 s9 s10 s11 s12 s13 s14 s15 s16

stride (words)

read

thro

ughp

ut (M

B/s

)

one access per cache line

Matrix multiplication example• Major cache effects to consider

–Total cache size•Exploit temporal locality and keep the working set small

(e.g., use blocking)–Block size

•Exploit spatial locality

• Description:–Multiply N x N matrices–O(N3) total operations–Accesses

•N reads per source element•N values summed per destination

– but may be able to hold in register

/* ijk */for (i=0; i<n; i++) { for (j=0; j<n; j++) { sum = 0.0; for (k=0; k<n; k++) sum += a[i][k] * b[k][j]; c[i][j] = sum; }}

Variable sumheld in register

Miss rate analysis for matrix multiply• Assume:

– Line size = 32B (big enough for four 64-bit words)

– Matrix dimension (N) is very large• Approximate 1/N as 0.0

– Cache is not even big enough to hold multiple rows

• Analysis method:– Look at access pattern of inner loop

CA

k

i

B

k

j

i

j

Matrix multiplication (ijk)

/* ijk */for (i=0; i<n; i++) { for (j=0; j<n; j++) { sum = 0.0; for (k=0; k<n; k++) sum += a[i][k] * b[k][j]; c[i][j] = sum; }}

A B C(i,*)

(*,j)(i,j)

Inner loop:

Column-wise

Row-wise Fixed

Misses per Inner Loop Iteration:A B C

0.25 1.0 0.0

Matrix multiplication (jik)

/* jik */for (j=0; j<n; j++) { for (i=0; i<n; i++) { sum = 0.0; for (k=0; k<n; k++) sum += a[i][k] * b[k][j]; c[i][j] = sum }}

A B C(i,*)

(*,j)(i,j)

Inner loop:

Row-wise Column-wise

FixedMisses per Inner Loop Iteration:

A B C0.25 1.0 0.0

Matrix multiplication (kij)

/* kij */for (k=0; k<n; k++) { for (i=0; i<n; i++) { r = a[i][k]; for (j=0; j<n; j++) c[i][j] += r * b[k][j]; }}

A B C(i,*)

(i,k) (k,*)

Inner loop:

Row-wise Row-wiseFixed

Misses per Inner Loop Iteration:A B C

0.0 0.25 0.25

Matrix multiplication (ikj)

/* ikj */for (i=0; i<n; i++) { for (k=0; k<n; k++) { r = a[i][k]; for (j=0; j<n; j++) c[i][j] += r * b[k][j]; }}

A B C(i,*)

(i,k) (k,*)

Inner loop:

Row-wise Row-wiseFixed

Misses per Inner Loop Iteration:A B C

0.0 0.25 0.25

Matrix multiplication (jki)

/* jki */for (j=0; j<n; j++) { for (k=0; k<n; k++) { r = b[k][j]; for (i=0; i<n; i++) c[i][j] += a[i][k] * r; }}

A B C

(*,j)(k,j)

Inner loop:(*,k)

Column -wise

Column-wise

Fixed

Misses per Inner Loop Iteration:A B C

1.0 0.0 1.0

Matrix multiplication (kji)

/* kji */for (k=0; k<n; k++) { for (j=0; j<n; j++) { r = b[k][j]; for (i=0; i<n; i++) c[i][j] += a[i][k] * r; }}

A B C

(*,j)(k,j)

Inner loop:(*,k)

FixedColumn-wise

Column-wise

Misses per Inner Loop Iteration:A B C

1.0 0.0 1.0

Summary of matrix multiplication

ijk (& jik): • 2 loads, 0 stores• misses/iter = 1.25

kij (& ikj): • 2 loads, 1 store• misses/iter = 0.5

jki (& kji): • 2 loads, 1 store• misses/iter = 2.0

for (i=0; i<n; i++) { for (j=0; j<n; j++) { sum = 0.0; for (k=0; k<n; k++) sum += a[i][k] * b[k][j]; c[i][j] = sum; }} for (k=0; k<n; k++) { for (i=0; i<n; i++) { r = a[i][k]; for (j=0; j<n; j++) c[i][j] += r * b[k][j]; }}for (j=0; j<n; j++) { for (k=0; k<n; k++) { r = b[k][j]; for (i=0; i<n; i++) c[i][j] += a[i][k] * r; }}

Pentium matrix multiply performance

• Miss rates are helpful but not perfect predictors.

• Code scheduling matters, too.

0

10

20

30

40

50

60

25 50 75 100 125 150 175 200 225 250 275 300 325 350 375 400

Array size (n)

Cyc

les/

itera

tion

kjijkikijikjjikijk

kji & jki

kij & ikj

jik & ijk

Improving temporal locality by blocking• Example: Blocked matrix multiplication

– Here, “block” does not mean “cache block”.– Instead, it mean a sub-block within the matrix.– Example: N = 8; sub-block size = 4

C11 = A11B11 + A12B21 C12 = A11B12 + A12B22

C21 = A21B11 + A22B21 C22 = A21B12 + A22B22

A11 A12

A21 A22

B11 B12

B21 B22X =

C11 C12

C21 C22

Key idea: Sub-blocks (i.e., Axy) can be treated just like scalars.

Blocked matrix multiply (bijk)for (jj=0; jj<n; jj+=bsize) { for (i=0; i<n; i++) for (j=jj; j < min(jj+bsize,n); j++) c[i][j] = 0.0; for (kk=0; kk<n; kk+=bsize) { for (i=0; i<n; i++) { for (j=jj; j < min(jj+bsize,n); j++) { sum = 0.0 for (k=kk; k < min(kk+bsize,n); k++) { sum += a[i][k] * b[k][j]; } c[i][j] += sum; } } }}

Blocked matrix multiply analysis– Innermost loop pair multiplies a 1 X bsize sliver

of A by a bsize X bsize block of B and accumulates into 1 X bsize sliver of C

– Loop over i steps through n row slivers of A & C, using same B

A B C

block reused n times in succession

row sliver accessedbsize times

Update successiveelements of sliver

i ikk

kk jjjj

for (i=0; i<n; i++) { for (j=jj; j < min(jj+bsize,n); j++) { sum = 0.0 for (k=kk; k < min(kk+bsize,n); k++) { sum += a[i][k] * b[k][j]; } c[i][j] += sum; }

InnermostLoop Pair

Blocked matrix multiply performance• Blocking (bijk and bikj) improves

performance by a factor of two over unblocked versions (ijk and jik)– relatively insensitive to array size.

0

10

20

30

40

50

60

Array size (n)

Cyc

les/

itera

tion

kjijkikijikjjikijkbijk (bsize = 25)bikj (bsize = 25)

Writing cache friendly code•Repeated references to variables are good(temporal locality)

•Stride-1 reference are good (spatial locality)•Examples: cold cache, 4-byte words, 4-word cache blocks

int sum_array_rows(int a[4][8]){ int i, j, sum = 0;

for (i = 0; i < M; i++) for (j = 0; j < N; j++) sum += a[i][j]; return sum;}

int sum_array_cols(int a[4][8]){ int i, j, sum = 0;

for (j = 0; j < N; j++) for (i = 0; i < M; i++) sum += a[i][j]; return sum;}

Miss rate = Miss rate = 1/4 = 25% 100%

Cache-conscious programming• make sure that memory is cache-aligned

• Use union and bitfields to reduce size and increase locality• Split data into hot and cold (list example)

Cache-conscious programming• Prefetching • Blocked 2D array

….

w

h

…….

….

h

w

K-d treestruct KdAccelNode { u_int flags; float split; // Interior u_int aboveChild; // Interior

u_int nPrims; // Leaf MailboxPrim *Primitives; // Leaf}

interior

nleaf

K-d tree8-byte (reduced from 20-byte, 20% gain)struct KdAccelNode { ... union { u_int flags; // Both float split; // Interior u_int nPrims; // Leaf }; union { u_int aboveChild; // Interior MailboxPrim **primitives; // Leaf };}

interior

nleaf

Tree representation

Flag: 0,1,2 (interior x, y, z) 3 (leaf)

S E M

flags

1 8 23

2

n

Concluding observations• Programmer can optimize for cache

performance– How data structures are organized– How data are accessed

• Nested loop structure• Blocking is a general technique

• All systems favor “cache friendly code”– Getting absolute optimum performance is very

platform specific• Cache sizes, line sizes, associativities, etc.

– Can get most of the advantage with generic code• Keep working set reasonably small (temporal locality)• Use small strides (spatial locality)