TUNA and HEAPO: Suite for Developing Software for NVM

Youjip WonHanyang University, Seoul, Korea

Flash Memory Summit 2015Santa Clara, CA 1

Outlines

TUNA: NVRAM Emulation Platform

HEAPO: Persistent Heap Layer for Non-Volatile Memory

2

TUNA: The Platform for Emulating NVM

3

Non-volatile Memory

4

NVRAM Flash/HDDBlock addressing

Slow

DRAMVolatile

FastByte-addressable

Non-volatileHigh Capacity

Flash Memory Summit 2015Santa Clara, CA

Characteristics of Memory Devices

5

Perf

orm

ance

/Ene

rgy

DRAM

FRAM

PCM

STT-MRAM

NAND-Flash

Read : 10-50nsWrite : 10-50nsEnergy/bit : 0.005pJEndurance : 1016

Density : 6~8F2Read : 10-30nsWrite : 13-95nsEnergy/bit : 0.1~2.5pJEndurance : 1015

Density : 14-64F2

Read : 25usWrite : 250usEnergy/bit : 0.000002pJEndurance : 103-103

Density : 4~8F2

Read : 20-70nsWrite : 50-500nsEnergy/bit : 2-25pJEndurance : 107-108

Density : 8~16F2

Read : 110ns Write : 150nsEnergy/bit : 1pJEndurance : 1010-1013

Density : 6~10F2

CapacityFlash Memory Summit 2015Santa Clara, CA

Software Layers for NVM

6

BPFS (Condit, MS, SOSP’09)

SCMFS (Wu, Texas A&M, SC’11)

PMFS (Dulloor, Intel, Eurosys’14)

PMBD (Chen, LSU, MSST’14)

Byte-addressible Filesystem

FRASH (Jung, HYU, ACM TOS’12)

NV-heap(Coburn, UCSD, ASPLOS’11)

Mnemosyne(Volos, UW, ASPLOS’11)

SoftPM (Guerra, FIU, Usenix ATC’12)

HEAPO (Hwang, HYU, ACM TOS’14)

Persistent Heap

WSP (Narayanan, MS, ASPLOS’12)

NVM Duet (Liu, NTU, ASPLOS’14)

Fast Block Device

Moneta (Caulfield, UCSD, MICRO’10)

Onyx (Akel, UCSD, HotStorage’11)

PCMSSD (Kim, IBM, FAST’14)

DC Express (Vucinic, UCSD, FAST’14)

nvramdisk (Jung, HYU)

DRAM

NVM

Process Address Space

Persistentheap Load/

Store

Physical Memory

PhysicalMemory

DRAM

NVM

Kernel Address Space

User buffer

Read/Write

CPU

NV

Me

DR

AM

Flash Memory Summit 2015Santa Clara, CA

Are the performance results credible?

Sources for inaccuracy

Real NVM device: small scale and for dedicated purpose

Emulating NVM with DRAM: DRAM is much faster.

Introducing software delay: code overhead and subsequent cache pollution

Cycle-accurate simulation: Lengthy simulation and cannot simulate realistic

scenario

NVRAM emulation platform (Intel PMEP): Only available in x86, cannot

simulate the non-volatility.7Flash Memory Summit 2015

Santa Clara, CA

Hardware platform for NVM emulation for mobile device

Variable Latency• PCM, STT-MRAM, and etc.

Non-volatility emulation

Mobile platform

8Flash Memory Summit 2015Santa Clara, CA

TUNA: Hardware platform for NVM emulation

Use DRAM to emulate NVRAM

For Non-volatility

• Separate power source for emulated NVRAM

For Variable latency

• API for adjusting the NVM latency

• Separate latencies for load and store

For Mobile platform

• ARM

• Android ported

9Flash Memory Summit 2015Santa Clara, CA

TUNA Organization

10

Xilinx Zynq (XC7Z045)

Processing System(PS, ARM Co-A9)

XilinxDRAM

controller

Programmable Logic(PL, Xilinx Kintex-7)

PS-DRAMComponent

Delay

PL-DRAMSo-DIMM

Memory interface

* PS is hardwired

AXI master

Core #1

Cor

e #2

Separatepower sourceFlash Memory Summit 2015

Santa Clara, CA

Emulating non-volatility with DRAM Separate power control

• Self-refresh management circuit• PS-only reboot (NVM-DRAM under self refresh or auto-refresh)

Flash Memory Summit 2015Santa Clara, CA 11

FPGA

DRAM controller

Self-refreshmaintain logic

2:1 Switch

DRAMSelf-refresh

P2

P1

Based on real measurement on steady state logic value!

(2) SEL

(3) Power-off

Support Variable Latency

Independent RD/WR latency tuning Delay AXI packets from PS

• Postpone handshake signals on AW/AR channel

Flash Memory Summit 2015Santa Clara, CA 12

DRAMcontroller

Delay

PL-DRAMSo-DIMM

PLDelay( {ARVALID, ARREADY}, read_lat);Delay( {AWVALID, AWREADY}, write_lat);

read_lat[n-1:0]write_lat[n-1:0]

TUNA specification

Xilinx Zynq SoC

• ARM Cortex-A9

• Xilinx Kintex-7

Memory

• 1GB PS DDR3-SDRAM

• For NVM: 8GB PL DDR3-

SDRAM

13Flash Memory Summit 2015Santa Clara, CA

ZYNQ7045

NVM

~8GBDDR3

PS-DRAM1GB DDR3

Mainpower

NVMpower

SD Card

HDMI

PCIe

Ethernet

JTAG

UART

0

500

1000

1500

2000

2500La

tenc

y [n

s]

Latency Setting [ns]

MeasuredIdeal

Range : 1GBStride size: 4KB Error = 1.04%

Latency Emulation: load

Flash Memory Summit 2015Santa Clara, CA

0

500

1000

1500

2000

2500La

tenc

y [n

s]

Latency Setting [ns]

MeasuredIdeal

Range : 1GBStride size: 4KB Error = 0.20%

Latency Emulation: store

Flash Memory Summit 2015Santa Clara, CA

HEAPO: Persistent Heap for Linux

Flash Memory Summit 2015Santa Clara, CA

16

Persistent Heap

17

ProcessAddress Space

exit()

Process Descriptor

ProcessAddress Space

fork()

Process Descriptor

DataData

ProcessAddress Space

Persistent

Area

exit()

Process Descriptor

fork()

Data

OrthogonalPersistent!

Legacy heap Persistent heap

Volatile

Flash Memory Summit 2015Santa Clara, CA

Overall Structure of HEAPO

18Process

Address Space

TEXTPersistent

Object

Physical Memory

DRAMRegion

STT-MRAMRegion

PersistentObject

Kernel Address Space

lib

lib

TEXT

TemporalObject

heapo_malloc(“Object Storage A”, 1836);

HEAPO Library

Kernel

heapo_malloc()

heapo_seg_alloc()

heapo_free()

heapo_seg_free()

heapo_free(“Object Storage A”, X);heapo_malloc(“Object Storage A”, 2824);heapo_free(“Object Storage A”, Y);

Persistent heap layer Dynamic allocation/deallocation Filesystem-independent namespace Support pointer usage

Flash Memory Summit 2015Santa Clara, CA

Application

Software Stack

Flash Memory Summit 2015Santa Clara, CA 19

Application

Object Alloc/Dealloc ModuleObject Storage Management Module

Name Table Cache

POS area Management

SegmentAllocation & MappingName Space

Management

HEAPO Library

System Calls

Library Interfaces

User

Kernel HEAPO Kernel Component

heapo_create() / heapo_delete()heapo_map() / heapo_unmap()heapo_malloc() / heapo_free()heapo_set_prime_object() / heapo_get_prime_object()

sys_heapo_create() / sys_heapo_delete()sys_heapo_map() / sys_heapo_unmap()sys_heapo_seg_alloc() / sys_heapo_seg_free()

Object Storage & Object Object storage: Persistent memory region with unique name Object: Unit of memory allocation/deallocation

20

Object Storage

B

Object Storage

A

0x5FFEF8000000

0x7FFEF8000000

POS area

Stack

DataText

Heap

mmap

32TB

Object ObjectObject Object Object

ObjectObjectObject Object Object

Flash Memory Summit 2015Santa Clara, CA

Usage Example

21Flash Memory Summit 2015Santa Clara, CA

Stack

DataText

Heap

Used

Process Address Space

NVRAM

heapo_create(“name”)

heapo_malloc(“name”, size)

heapo_malloc(“name”, size)

exit()

fork()

heapo_map(“name”)

heapo_unmap(“name”)

heapo_malloc(“name”, size)

heapo_malloc(“name”, size)

heapo_free(address)

heapo_free(address)

obj obj

obj obj

obj obj

obj obj

obj obj

heapo_malloc(“name”, size)

H-KVLib: HEAPO-based Key-value Library

Internal data structure: B-tree, list, hash Guarantee fail-safe atomicity (minimal logging) Support lock-based and STM-based concurrency control

Flash Memory Summit 2015Santa Clara, CA 22

lock-based STM-based

Thread Thread Thread

Lock

H-KVLiblist hash btree

Unlock

Critical Section

HEAPO Library

Thread Thread Thread

Transaction Start

H-KVLiblist hash btree

TransactionCommit

HEAPO Library

Use case: Web Application

23

Block Block Block

Block Block Block

Block Block

Parsing(memory copy)

Rendering

Storage

Web Browser

DRAM

Disk IO

Block

Block Block Block

IO Buffer

Heap

Obj

Obj Obj

Obj Obj

Obj

L R

L R

R

DOM Tree

Flash Memory Summit 2015Santa Clara, CA

Launching Web Applications with HEAPO

24

No Disk IO !No Parsing !

RenderingWeb Browser

Object Storage (NVRAM)

Obj

Obj Obj

Obj Obj

Obj

L R

L R

R

Flash Memory Summit 2015Santa Clara, CA

HEAPO-based Key-value Store Performance

Flash Memory Summit 2015Santa Clara, CA 25

I: Insert, L: Lookup, D: Delete, U: Upate

Fail Safe Atomicity Support in HEAPO

Logging for key-value operations in H-KVLIB

For persistence and ordering

• clflush and mfence in x86

• dccmvac in ARM

Flash Memory Summit 2015Santa Clara, CA 26

TUNA & HEAPO: HW/SW for NVRAM

TUNA platform (http://www.opennvram.org)

• OS support: linux & android

• Applications and Library : HEAPO and SQLite for NVM

HEAPO (https://github.com/ESOS-Lab/HEAPO)

• HW support: x86 & ARM

• Provide dedicated key-value library

Flash Memory Summit 2015Santa Clara, CA 27

Summary HEAPO over TUNA

• Comprehensive Platform for developing software for NVM based system

• All open sourced

TUNA platform (http://www.opennvram.org)

• OS support: linux & android

HEAPO (https://github.com/ESOS-Lab/HEAPO) for x86 and ARM

• Provide dedicated key-value library

Flash Memory Summit 2015Santa Clara, CA 28

Collaborators

Joint work with

• CRZ Co.

• Prof. Seungjoo Yoo at Seoul National University

References• Hwang et.al. “HEAPO: Heap-based Persistent Object Store”, ACM Trans. on Storage, 11(1), Dec.

2014

• Lee et.al. “FPGA-based prototyping systems for emerging memory technologies,” IEEE

International Symposium on Rapid System Prototyping (RSP), 2014

Flash Memory Summit 2015Santa Clara, CA 29