The Open Source ProtoFlex Simulator

Computer Architecture Lab at

RAMP Retreat, June 2009

The Open SourceProtoFlex Simulator

Eric S. Chung, Michael K. Papamichael,James C. Hoe, Babak Falsafi, Ken Mai

The ProtoFlex Simulator

• History

– Project started (circa 2007) to build scalable, full-system multiprocessor simulators using FPGAs

• Key Features

– Functional simulator for N-way UltraSPARC III server (~50-90 MIPS)

– Using hybrid simulation, runs real server apps + Solaris OS

– Employs multithreading to virtualize # CPUs per FPGA core

2Hybrid Simulation Virtualization

Open Sourcing ProtoFlex

• Why open source?

– Demonstration of FPGAs as viable architecture research vehicle

– Facilitate adoption of hybrid simulation & host multithreading

– Encourage building on top of our work

• What are we releasing?

– Bluespec source HDL, Verilog and pre-generated netlistsfor SPARCV9 CPU model + interfaces

– XUPV5 Reference Design for EDK 10.1

– Virtutech Simics plug-ins for hybrid simulation

– Top-level SW controller, user command-line interface

– Documentation through online wiki 3

Outline

• Motivation

• The ProtoFlex Simulator

– High level components

– UltraSPARC core model

• Using ProtoFlex

• XUPV5 Reference Design

• Distribution Details4


• User perceives familiar SW-like UltraSPARC III simulator

– Software user interface similar to Simics

– Applications load directly from Simics checkpoints

– Standard simulation features: state viewing, scripting, single-stepping, checkpointing, terminal, profiling/monitoring

5


• User perceives familiar SW-like UltraSPARC III simulator

– Software user interface similar to Simics

– Applications load directly from Simics checkpoints

– Standard simulation features: state viewing, scripting, single-stepping, checkpointing, terminal, profiling/monitoring

6

FPGA Linux PC

PFMON

SIMICS

(I/O)

FPGA

Core

Main Memory

PowerPC

(or uBlaze)

Ethernet User

Interfac

e

Our UltraSPARC III Core Model

• ISA Specifications

– 64-bit SPARCV9 ISA + US III extensions

– 8 register windows, 4 global register files

– 512-entry D-TLB, 128 I-TLB

• Implementation

– 14-stage, multi-threaded pipeline, switch context on each cycle

– On Virtex-5, XST~148MHz,Placed & routed @ 100MHz

– Parameterized non-blocking caches

– FP + rare MMU instructions areSW-emulated by nearby uBlaze

– 100% mirrors Virtutech Simics model 7

I-TLB Stage 1

I-TLB Stage 2

64-bit ALU Stage 1

Context Scheduler

I-Fetch Address Generate

I-Fetch Tag Check

US III Decoder

64-bit ALU Stage 2

D-TLB Stage 1

D-TLB Stage 2

D-TLB Stage 3

D-Cache Address Generate

D-Cache Tag Check

Multi-Cycle Instruction Unit

NonblockingI-cache(BRAM)

Writeback

Arbiter to DDR Memory

Integer RF(BRAM)

NonblockingD-cache(BRAM)

Core Design Statistics

• Runs 100MHz on V5

– Synthesizes up to 148MHz using standard tools (ISE XST)

• Logic usage

– 23.5 KLUTs (11.3% LX330T)

• BRAM usage

– 120 BRAMs for 16-context configuration (37% LX330T)

• Future optimizations

– Paging structures to SRAM or DRAM can reduce BRAM by significant amount

– Will release in future updates 8

Outline

• Motivation


• Using ProtoFlex

• XUPv5 Reference Design

• Distribution

9

Using ProtoFlex

• Add passive monitors

– Counters, histograms

– Roll your own

10

I-TLB Stage 1

I-TLB Stage 2

64-bit ALU Stage 1

Context Scheduler

I-Fetch Address Generate

I-Fetch Tag Check

US III Decoder

64-bit ALU Stage 2

D-TLB Stage 1

D-TLB Stage 2

D-TLB Stage 3

D-Cache Address Generate

D-Cache Tag Check

Multi-Cycle Instruction Unit

NonblockingI-cache(BRAM)

Writeback

Arbiter to DDR

Memory

Integer RF(BRAM)

NonblockingD-cache(BRAM)

• Trace-based simulation

– Collect dynamic traces

– Feed traces to functional-first timing model

• Sampled Program Monitoring

– Use micro-blaze (or PPC) to monitor core/memory state

– Unintrusive profiling w/o changes to target SW

CountersCountersCounters

Histogram

TrackerHistogram

TrackerHistogram

Trackers

Timing

Model

FPGA

Hard/Soft Core

(PowerPC or

MicroBlaze)

Applications of ProtoFlex

• Examples

– Functional-first CMP cache coherency model for first-order timing models and functional warming *TRETS’09+

– Real-time stack trace profiling

– CMP interconnect model (in progress)

– Realistic CPU traffic generators (in progress)

• … running real 16-CPU server workloads

– Oracle TPC-C, IBM DB/2 TPC-C, TPC-H, SPEC2K

11

Piranha CMP Cache

(First-Order Timing

Model)

Statistics + Warmed

Coherency &

Tag States

What does the RTL look like?

• We use Bluespec System Verilog (high-level, synthesizable HDL)

– 4-8 weeks learning curve for normal HDL users

– Once learned, easier to read/modify than conventional RTL

– Requires BSV compiler (free for academics)

– Paper in MEMOCODE’09 describes BSV coding/validation of core

• Sample code:

12

rule split_ALU_pipeline (True);…p1 = piperegs[DECODE];piperegs[ALU1] <= doALUStage1(p1, alu_ifc);p2 = piperegs[ALU1];piperegs[ALU2] <= doALUStage2(p2, alu_ifc);…

endrule

rule merged_ALU_pipeline (True);…p1 = piperegs[DECODE];p_tmp1 = doALUStage1(p1, alu_ifc);p_tmp2 = doALUStage2(p_tmp1, alu_ifc);piperegs[ALU] <= p_tmp2;…

endrule

2-stage ALU 1-stage ALU

Other Simulator Features

• Changing Core Parameters

– Number of CPU contexts

– Cache sizes

– Merge/split pipeline stages

– Enable/disable modules for profiling & debugging

– Clock frequency (tested @ 10 MHz – 100 MHz)

– Set optimal LUTRAM size (16 = V2P, 64 = V5)

– Choose LUTRAMs or BRAMs for any CPU state

• System Parameters

– UDP or TCP/IP (for PFMON-to-FPGA communication)

– XUPv5, BEE2

13

Outline

• Motivation


• Using ProtoFlex


• Distribution

14

Platform Release: XUPV5

• Why XUPv5?

– Inexpensive (~$750), easily accessible

– Standard tool flows (EDK, ISE)

– Reference design portable to other platforms – just drop in our ‘pcores’

• Supporting other platforms

– Future ports to BEE3 & Xilinx Accelerated Computing Platform (ACP)

– Plan to release with future updates

15

Required Equipment

16

FPGA BoardLinux PC

Ethernet

BlueSPARCPFMON+ Simics

Required Equipment

17

FPGA BoardLinux PC

Ethernet


18

XUPv5 Overview

• Virtex-5 LX110T

• DDR2 Memory

– up to 2GB

• 1ΜΒ SRAM

• 1Gbps Ethernet

• 3Gbps SATA

• Serial Port

SRAMController

Multi-Port Memory Controller

Reference Design Block Diagram

19

PLB

BlueSPARC MicroBlaze

Ethernet BRAM Serial PortBRAMBRAM

XUPv5

LX110T

DRAM SRAM

SRAMController


BlueSPARC

20

PLB

MicroBlaze


XUPv5

LX110T

DRAM SRAM

• EDK IP core

– connects to PLB & NPI

• Runs @ 100MHz

• 4 CPU contexts

• 64KB I&D L1 caches

BlueSPARC

SRAMController



21

PLB



XUPv5

LX110T

DRAM SRAM

SRAMController


BlueSPARC

22

PLB


Ethernet Serial Port

XUPv5

LX110T

DRAM SRAM

• 81% utilization

– Core 51% (76 out of 148)

– Rest 30% (45 out of 148)

BRAMBRAMBRAM

SRAMController



23

PLB



XUPv5

LX110T

DRAM SRAM

SRAMController


Ethernet

24

PLB


BRAM Serial PortBRAMBRAM

XUPv5

LX110T

DRAM SRAM

• 4 MB/sec bandwidth

• 350 usec RTT latency

• Socket Abstraction

– using LWIP RAW interface

Ethernet

SRAMController



25

PLB



XUPv5

LX110T

DRAM SRAM

SRAMController

DDR2 Memory Controller

26

PLB



XUPv5

LX110T

DRAM SRAM


• 1.5GB/s peak BW

• 115ns latency

• Multiple ports/interfaces

Required Equipment

27

FPGA BoardLinux PC

Ethernet


Required Equipment

28

FPGA BoardLinux PC

Ethernet


29

Linux PC

• Software requirements

– SuSE Linux 10.1

– CAD tools + licenses (Bluespec compiler, Xilinx ISE/EDK)

– Simics 3.0.22

– Hybrid simulation plug-in modules

– ProtoFlex MONitor tool (PFMON)

30

Linux PC

• Runs PFMON (ProtoFlex MONitor)

– Orchestrates communication between Simics & BlueSPARC

– Provides CLI interface to simulator (like Simics Console)

• Runs Simics

– Handles I/O, FPGA Core and memory initialization

BlueSPARCPFMON

Linux PC

Simics

31

From RTL to Running System

• Bluespec Verilog

– Bluespec compiler

– ~30 minutes

• Verilog Bitstream

– Xilinx EDK

– ~ 3 hours

• BitstreamWorking System

– Stream mem. image over ethernet

– ~ 5 minutes (for 512MB image)

Bluespec code

Verilog code

Bitstream

1

2

3

1

2

WorkingSystem

3

XUPv5 Caveats

• Limitations

– Due to BRAM limits, only 4 CPU contexts (compared to 16 on BEE2’s V2P70)

– Slow PC-to-FPGA latency via LwIP/Ethernet (1 key/sec in terminal)

– Limited DDR2 RAM capacity (up to 2GB)

• However…

– Our XUPv5 design still has much room for improvement

– Useful for familiarizing with ProtoFlex tools

– Can still run multithreaded workloads + perform monitoring

– Easily portable to more powerful platforms

32

Outline

• Motivation


• Using ProtoFlex


• Distribution

33

Distribution Details

• Release dates

– All source code will be available in 1 week (June)

– Send email to [email protected] for more info

• Licensing

– GNU GPL

• Support

– Documentation: www.ece.cmu.edu/~protoflex

– Support mailing list/forum will be available soon

34

mailto:[email protected]

http://www.ece.cmu.edu/~protoflex

Thank you!

• Acknowledgments

– Funding for this work provided by: NSF CCF-0811702, NSF CNS-0509356, C2S2 Marco Center, and Sun Microsystems

– Paul Hartke & Xilinx for FPGA donations & email support

• Topics & References

– Hybrid Simulation and FPGA Core MultithreadingA Complexity-Effective Architecture for Accelerating Full-System Multiprocessor Simulations Using FPGAs [FPGA’08]

– Functional-First, First-Order Timing Model for a CMP cacheProtoFlex: Towards Scalable, Full-System Multiprocessor Simulations Using FPGAs [TRETS’09]

– FPGA Core Design & Validation Using Flight Data RecorderImplementing a High-performance Multithreaded Microprocessor: A Case Study in High-level Design and Validation [MEMOCODE’09]

35

Documents

The Open Source ProtoFlex Simulator