QEMU Support for the RISC-V Instruction Set Architecture

QEMUSupportfortheRISC-VInstructionSetArchitecture

SagarKarandikarsagark@eecs.berkeley.edu

KVMForum2016

https://github.com/riscv/riscv-qemu

Outline

• WhyRISC-V?• BenefitsofanOpenISA• RISC-VISABasics• VirtualizationSupport• QEMURISC-VTargetSupport• WorkinProgress/TODOsforUpstreaming/FutureWork

ISAsdon’tmatter

• Mostoftheperformanceandenergyofrunningsoftwareonacomputerisdueto:• Algorithms• Applicationcode• Compilers• OS/Runtimes• ISA• Microarchitecture(coreandmemoryhierarchy)• Circuitdesign• Physicaldesign• Fabricationprocess

+Inasystem,therearealsodisplays,radios,DC/DCconverters,sensors,actuators…

WhyInstructionSetArchitectureMatters

• Whycan’tIntelsellmobilechips?• 99%+ofmobilephones/tabletsbasedonARMv7/v8ISA

• Whycan’tARMpartnerssellservers?• 99%+oflaptops/desktops/serversbasedonAMD64ISA(over95%+builtbyIntel)

• HowcanIBMstillsellmainframes?• IBM360,oldestsurvivingISA(50+years)

ISAisthemostimportantinterfaceinacomputersystemwheresoftwaremeetshardware

WhysomanyISAsonanSoC?

• Applicationsprocessor(usuallyARM)• Graphicsprocessors• Imageprocessors• RadioDSPs• AudioDSPs• Securityprocessors• Power-managementprocessor• ….• AppsprocessorISA(e.g.ARM)toolargeformostaccelerators• IPboughtfromdifferentplaces,eachproprietaryISA• Home-grownISAcores• OveradozenISAsonsomeSoCs– eachwithuniquesoftwarestack

NVIDIATegraSoC

DoweneedallthesedifferentISAs?

Musttheybeproprietary?

WhatiftherewereonefreeandopenISAeveryonecoulduseforeverything?

ISAsshouldbeFreeandOpen

• WhileISAsmaybeproprietaryforhistoricalorbusinessreasons,thereisnogoodtechnicalreasonforthelackoffree,openISAs• It’snotanerrorofomission• Norisitbecausethecompaniesdomostofthesoftwaredevelopment• NeitherdocompaniesexclusivelyhavetheexperienceneededtodesignacompetentISA• NorarethemostpopularISAswonderfulISAs• NeithercanonlycompaniesverifyISAcompatibility• Finally,proprietaryISAsarenotguaranteedtolast

BenefitsofaViableFreeandOpenISA

• Greaterinnovationviafree-marketcompetition frommanycoredesigners,closed-sourceandopen-source• Sharedopencoredesigns,shortertimetomarket,lowercostfromreuse,fewererrorsgivenmoreeyeballs• Processorsbecomingaffordableformoredevices,whichwouldhelpexpandtheInternetofThings(IoTs),whichcouldcostaslittleas$1• Softwarestackssurviveforlongtime upgradesoftwareonsystemsembeddedinconcrete50yearsago• Makearchitectureresearchandeducationmorereal withfullyopenhardwareandsoftwarestacks

RISC-VOrigins

• In2010,aftermanyyearsandmanyprojectsusingMIPS,SPARC,andx86asbasisofresearch,itwastimefortheComputerScienceteamatUCBerkeleytolookatwhatISAstousefortheirnextsetofprojects• Obviouschoices:x86andARM• x86impossible– toocomplex,IPissues

Intelx86“AAA”Instruction

• ASCIIAdjustAfterAddition• ALregisterisdefaultsourceanddestination• Ifthelownibbleis>9decimal,ortheauxiliarycarryflagAF=1,then• Add6tothelownibbleofALanddiscardoverflow• IncrementhighbyteofAL• SetCFandAF

• Else• CF=AF=0

• Asingle-byteinstruction

RISC-VOrigins

• In2010,aftermanyyearsandmanyprojectsusingMIPS,SPARC,andx86asbasisofresearch,itwastimefortheComputerScienceteamatUCBerkeleytolookatwhatISAstousefortheirnextsetofprojects• Obviouschoices:x86andARM• x86impossible– toocomplex,IPissues• ARMmostlyimpossible– complex,IPissues

• UCBerkeleystarteda“3-monthproject”duringsummerof2010todeveloptheirownclean-slateISA• AndrewWaterman,YunsupLee,DavePatterson,KrsteAsanovicprincipaldesigners

RISC-VBackground(cont’d)

• Fouryearslater,inMayof2014,UCBerkeleyreleasedfrozenbaseuserspec• Manychiptapeoutsandseveralresearchpublicationsalongtheway

• ThenameRISC-V(pronouncedrisk-five),waschosentorepresentthefifthmajorRISCISAdesigneffortatUCBerkeley• RISC-I,RISC-II,SOAR,andSPURwerethefirstfourwiththeoriginalRISC-Ipublicationsdatingbackto1981

• InAugust2015,articlesofincorporationwerefiledtocreateanon-profitRISC-VFoundationtogoverntheISA

RISC-VisNOTanOpen-SourceProcessor

• RISC-VisanISAspecification– NOTanopen-sourceprocessorcore• Mostofthecostofchipdesign isinsoftware,sowewanttomakesuresoftwarecanbereusedacrossmanychipdesigns• TheFoundationwillencouragebothopen-sourceandproprietaryimplementationsoftheRISC-VISAspecification

UCBerkeleyRISC-VCores:Raven-1 Raven-2

Raven-3

Raven-4

EOS14

EOS16

EOS18

EOS20

2011 2012 2013 2014 2015

May Apr Aug Feb Jul Sep Mar Nov Mar

EOS:IBM45nmSOIRaven:ST28nmFDSOIHurricane:ST28nmFDSOISWERVE:TSMC28nm

SWERVE

EOS22EOS24

1GHz50+DPGFLOPS/W

1.65GHz14DPGFLOPS/W

Hurricane-1

2016

IndustrialSupport– PlatinumFoundingMembers

IndustrialSupport– Gold,Silver,AuditorFoundingMembers

RumbleDevelopment

TheRISC-VISA

• RV32,RV64,RV128variantsfor32b,64b,128baddressspacesdefined• BaseISAonly<50integerinstructions,butsupportscompiler,linker,OS,etc.• Extensionsprovidefullgeneral-purposeISA,includingIEEE-754/2008floating-point• ComparableISA-levelmetricstootherRISCs• Designedforextension,customization• Twelve64-bitsiliconprototypeimplementationscompletedatBerkeleysofar(45nm,28nm)

RISC-VStandardBaseISADetails

• 32-bit,fixed-width,naturallyalignedinstructions• 31integerregistersx1-x31,plusx0zeroregister• rd/rs1/rs2infixedlocation,noimplicitregisters• Immediatefield(instr[31])alwayssign-extended• Floating-pointaddsf0-f31registersplusFPCSR,alsofusedmul-addfour-registerformat• DesignedtosupportPICanddynamiclinking

RV64GDefinition

• G=I,M,A,F,D• I = BaseIntegerISA• M = StandardIntegerMultiplication/DivisionExtension• A = StandardAtomicsExtension• F = StandardSingle-precisionFloating-pointextension• D = StandardDouble-precisionfloating-pointextension

• Thisisthestandard,generalpurposeversionoftheISA,whatisimplementedinQEMU

RISC-VPrivilegedSpecification

• FourPrivilegeModes:User,Supervisor,Hypervisor,Machine• MachineModerequired• Common:ProvideM,S,UforrunningUnix-likeOSes(QEMUdoesthis)• VirtualMemoryArchitecturedesignedtosupportcurrentUnix-likeoperatingsystems• Sv39(RV64)• Demand-paged39-bitvirtual-addressspaces• 3-levelpagetable• 4KiBpages,2MiBmegapages,1GiBgigapages

• AlsoSv32(RV32)andSv48,Sv57,Sv64(RV64)

RISC-VVirtualization

• ISAdesignedwithvirtualizationin-mindfromthebeginning,evenwhenonlyusingU+S+Mmodes• “Theprivilegedarchitectureisdesignedtosimplifytheuseofclassicvirtualizationtechniques,whereaguestOSisrunatuser-level,asthefewprivilegedinstructionscanbeeasilydetectedandtrapped.”– RISC-VPrivilegedArchitecturev1.9Manual

• AvoidingSomeClassicalVirtualizationPitfalls…

HandlingSensitive,butUnprivilegedInstructions• Inx86,fortheoriginalVMware– “TableIIliststhe[19]instructionsofthex86architecturethatunfortunatelyviolatedPopekandGoldberg’sruleandhencemadethex86non-virtualizeable”1

• InRISC-V,no“hidden”privilegedstatereads/writes• Smallsetofprivilegedinstructionsthatcanmodifyspaceofprivilegedstate(ControlStatusRegisters,orCSRs)

1.E.Bugnion,S.Devine,M.Rosenblum,J.Sugerman,andE.Y.Wang.2012.BringingVirtualizationtothex86ArchitecturewiththeOriginalVMwareWorkstation. ACMTrans.Comput.Syst. 30,4,Article12(November2012),51pages.

TrackingChangesinVirtualMachineMemory

• Inx86,fortheoriginalVMware– “…privilegedhardwareregisterscontaintheaddressofsegmentdescriptortablesandpagetables...but regularloadandstoreinstructionscanaccessthesestructuresinmemory.”1

• InRISC-V,stilluseregularloads/storestomodifymemorymanagementstate• PrivilegedSFENCE.VM instructionrequiredbyspec.aftermodifyingmemorymanagementstate

1.E.Bugnion,S.Devine,M.Rosenblum,J.Sugerman,andE.Y.Wang.2012.BringingVirtualizationtothex86ArchitecturewiththeOriginalVMwareWorkstation. ACMTrans.Comput.Syst. 30,4,Article12(November2012),51pages.

VirtualizingSegmentation

• Inx86,fortheoriginalVMware– Complicatedinteractionsbetweensegmentdescriptortablesandsegmentregisters,withvisibleandhiddenfields.Hiddenpiecesupdatedbyinstructionsorfaults.CausesproblemswithextrafaultsintroducedbyaVMM.1

• InRISC-V,nox86-stylesegmentation• Limitedbase-and-boundsmodewith2“segments”• Mostsoftwarewillusepaginginstead

1.E.Bugnion,S.Devine,M.Rosenblum,J.Sugerman,andE.Y.Wang.2012.BringingVirtualizationtothex86ArchitecturewiththeOriginalVMwareWorkstation. ACMTrans.Comput.Syst. 30,4,Article12(November2012),51pages.

RISC-VVirtualizationStacks

• Providecleansplitbetweenlayersofthesoftwarestack• ApplicationcommunicateswithApplicationExecutionEnvironment(AEE)viaApplicationBinaryInterface(ABI)• OScommunicatesviaSupervisorExecutionEnvironment(SEE)viaSystemBinaryInterface(SBI)• HypervisorcommunicatesviaHypervisorBinaryInterface(HBI)toHypervisorExecutionEnvironment(HEE)• AlllevelsoftheISAdesignedtosupportvirtualization

2 1.1draft: Volume II: RISC-V Privileged Architectures

ApplicationABIAEE

ApplicationABI

OSSBISEE

ApplicationABI

SBIHypervisor

ApplicationABI

OS

ApplicationABI

ApplicationABI

OS

ApplicationABI

SBI

HBIHEE

Figure 1.1: Di↵erent implementation stacks supporting various forms of privileged execution.

the OS, which provides the AEE. Just as applications interface with an AEE via an ABI, RISC-Voperating systems interface with a supervisor execution environment (SEE) via a supervisor binaryinterface (SBI). An SBI comprises the user-level and supervisor-level ISA together with a set ofSBI function calls. Using a single SBI across all SEE implementations allows a single OS binaryimage to run on any SEE. The SEE can be a simple boot loader and BIOS-style IO system in alow-end hardware platform, or a hypervisor-provided virtual machine in a high-end server, or athin translation layer over a host operating system in an architecture simulation environment.

The rightmost configuration shows a virtual machine monitor configuration where multiple multi-programmed OSs are supported by a single hypervisor. Each OS communicates via an SBI with thehypervisor, which provides the SEE. The hypervisor communicates with the hypervisor executionenvironment (HEE) using a hypervisor binary interface, to isolate the hypervisor from details ofthe hardware platform.

Our graphical convention represents abstract interfaces using black boxes with white text, toseparate them from actual components.

The various ABI, SBI, and HBIs are still a work-in-progress, but we anticipate the SBI and HBIto support devices via virtualized device interfaces similar to virtio [2], and to support devicediscovery. In this manner, only one set of device drivers need be written that can support anyOS or hypervisor, and which can also be shared with the boot environment.

Hardware implementations of the RISC-V ISA will generally require additional features beyond theprivileged ISA to support the various execution environments (AEE, SEE, or HEE), but these weconsider separately as part of a hardware abstraction layer (HAL), as shown in Figure 1.2. Note

ApplicationABIAEEHAL

Hardware

ApplicationABI

OSSBISEE

ApplicationABI

HALHardware

SBIHypervisor

ApplicationABI

OS

ApplicationABI

ApplicationABI

OS

ApplicationABI

SBI

HBIHEE

HardwareHAL

Figure 1.2: Hardware abstraction layers (HALs) abstract underlying hardware platforms from theexecution environments.

RISC-VHypervisorSpecification- WIP

• CurrentprivilegeddesigncanhaveanM-modemonitorthatprovidesphysicalresourcepartitioning,canactassimplehypervisor• UpcomingHypervisorExtensionSpecificationfor“full”Hypervisors• Rightnow,anemptyslotintheprivilegedspecification

• Wanttogetinvolved?• HypervisorSpecificationDraftwillmaketheroundssoononisa-dev@groups.riscv.org mailinglist

TheRISC-VEcosystem

• SoftwareTools• GCC/glibc/GDB• LLVM/Clang• Linux• Yocto• VerificationSuite

• HardwareTools• ZynqFPGAInfrastructure• Chisel

• SoftwareImplementations• Spike,“Golden-standard”ISASimulator• ANGEL,JavaScriptISASimulator• QEMU

• HardwareImplementations• RocketChipGenerator

• RV64Gsingle-issuein-orderpipeline• SodorProcessorCollection• BOOM(BerkeleyOut-of-OrderMachine)

github.com/riscvandgithub.com/ucb-bar

RISC-VTargetsupportforQEMU

• Maintainedathttps://github.com/riscv/riscv-qemu• QEMUfull-systememulation• QEMUonmodernx86iscurrentlythefastestRISC-Vimplementation• AbighelpinRISC-Vsoftwaredevelopment

Spike QEMU

TimelineofRISC-V“Firsts”

2014 2015 2016

riscv-qemuWorkStarted

1st LinuxBootonQEMU

FastestRISC-VImplementation

1st RISC-VImplementation

w/TCP/IPNetworking

1st PythonBringuponRISC-V

1st JavaBringuponRISC-V(HotspotZeroJVM)

1st RISC-VCoreBuiltonRISC-VSystem

withChisel

…allonQEMU!

RISC-VChipDevelopmentonRISC-V

+4

Instruction Mem

RegFile

IType SignExtend

DecoderData Mem

ir[24:20]

branch

pc+4

pc_s

el

ir[31:20]

rs1

ALU

ControlSignals

wb_s

el

RegFile

rf_we

n

val

mem

_rw

PC

mem

_val

addrwdata

rdata

Inst

JumpTargGen

BranchTargGen

ir[19:15]

ir[31:25],ir[11:7]

PC+4jalr

rs2

BranchCondGen

br_eq?br_lt?

co-p

roce

ssor

(CSR

) reg

ister

s

ir[11

:7]

jump

ir[31:12]

Execute Stage

br_ltu?PC

addr

ir[31:12]

JumpRegTargGen

Op2Sel

Op1SelAluFun

data

wa

wd

en

addr data

UType

Note: for simplicity, the CSR File (control and status registers) and associated datapath is not shown

RISC-V Sodor 1-Stage

exception

SType SignExtend

ir[31:20]

PC

rs2rs1

rs2

BuildscreenshotcourtesyMichaelKnyszek

TimelineofRISC-V“Firsts”

2014 2015 2016

riscv-qemuWorkStarted

1st LinuxBootonQEMU

FastestRISC-VImplementation

1st RISC-VImplementation

w/TCP/IPNetworking

1st PythonBringuponRISC-V

1st JavaBringuponRISC-V(HotspotZeroJVM)

1st RISC-VCoreBuiltonRISC-VSystem

withChisel

QEMURISC-V

Priv.Spec1.7Bump

QEMURISC-V

Priv.Spec1.9Bump

UpstreamingBegins

…allonQEMU!

1st RISC-VSystem

w/RemoteGDB

RISC-VTargetsupportforQEMU

• RISC-Vsupportstartedin2014,evolvesastheISAdoes• SupportsRV64IMAFD(“RV64G”)Full-systememulation• UserISAsupportlargelyunchangedsincethen(currentlyv2.0)• PrivilegedISAnearingstandardization(currentlyv1.9)

• FuturePriv.SpecupgradestoQEMUmuchsimplerduetostructuralsimilarityofpriv.modeemulationcodewithSpike• Pre-1.7->1.7bump,~1month• 1.7->1.9bump,3days

• I/Ocurrentlylimitedto“Host-TargetInterface”(HTIF)devices• EnoughtobootLinux,interactthroughconsole• Otherdevicespreviouslyshoehornedin(networking,displays,consoles)

• Mainlywaitingonplatformstandardizationandsoftwaresupport

hw/riscv/riscv_board.c

• Reference“board”designedtomatchSpike• Providessimplehardware,config:• HTIF-basedconsole(simple,non-standardconsoledeviceforearlyboot)• “Loopback”SoftwareInterruptEmulation• RTC/TimerscompliantwithRISC-Vv1.9privilegedspecification• ResetVector(BootROM)• ConfigurationString

0x8000_0000

0x0000_0000

0x0000_1000

0x4000_0000

... DRAM ...

Conf. StringReset Vector

RTCTimers

DRAM Base

DRAM Top

SoftInt Emu.

Base + Size

High

I/O:HTIF(old),Debug(new)

• Host-TargetInterface(HTIF)isarelicofBerkeleyTestChips• Two64-bitregisters,fromhostandtohost• Formerlyprovidednetwork,blockdevice,console,nowusedonlyforconsole,signalingtestcompletion• Debugging:HTIFbeingphasedoutonrealhardware,replacedwithdraftDebugUnitspecification(willbestandardized)• I/O:Realhardware/softwaresimulatorswillalsophaseoutHTIFandmovetostandarddevicesassoftwaresupportprogresses

SoftwareStackinsideRISC-VQEMU

• M-moderunssecurebootandmonitor(currentlybbl)• S-moderunsOS(OSalwaysrunsvirtualized)• U-moderunsapplicationontopofOSorM-mode

M-modemonitor

U-modesystemprocess

S-modeOS

Device2Interrupts

Device1Interrupts

OtherInterrupts

U-modeapp

BootUp

BinarysuppliedtoQEMUcontains:• bbl – “BerkeleyBootLoader”performsMachine-Modemanagementofthesystem,exposesSBItoOSes• vmlinux – Linuxkernelaspayloadforbbl• Includesaramdiskforrootfs (currentlylimiteddevicesupport)

Systembootsintohardcodedbootrom,jumpstobbl,bbl initializesthesysteminM-mode,setsupSupervisorExecutionEnvironment(SEE),thenloadsandrunssuppliedkernel,e.g.Linux

Testing/Debugging

• TheusualGDB,bruteforce,etc• Passestheriscv-testsstandardtestsuite• BootsLinux

• Withreasonablylargesoftwarestacksontop– Python,Java,etc.

• Fuzztestingagainstthe“GoldenStandard”

FuzzTestingwithriscv-torture• Generatesarandomsequenceofinstructionsbasedonsuppliedparameters(%ofmeminstructions,floatingpointinstructions,integerinstructions,etc.)• RunscodeonSpikeandotherimplementationofyourchoice

• Spikeisthe“GoldenReference”RISC-VSimulator,writtenbytheauthorsoftheRISC-VSpecs

• Dumpregisterstateattheendandcompare• Onfailure,binarysearchtopinpointinstructionwherethingsgowrong• Scriptsforrunningriscv-tortureonQEMUavailableathttps://github.com/sagark/riscv-qemu-torture• Accumulated384hoursoffailure-freetestingsinceAugustPriv.1.9update!

target-riscv SLoC*

• ARM- 45,438• MIPS- 37,501• x86- 30,437• RISC-V- 5,074

WorkinProgress/TODOs

• Functionality:Standarddevicesupport(comboofQEMU+Linux)• Upstreaming!– planningtostartinmid-September

• SubmittedagiantpatchinFebruaryasproof-of-existenceforgauginginterestinGSoC

• Somecleanupbasedongiant-patchfeedbackdone(e.g.usebuilt-inFPU)• LatestRISC-VPrivilegedSpec.v1.9bumpdone• TODO:

• Morecleanupbasedongiant-patchfeedback,checkpatch• Rebasetomaster(currentlyv2.5.0)• Small,logicalpatches

• Future:• SupportotherISAvariants,likeRV32,CompressedISA,QEMULinux-UserMode

Thanks!Questions?

SagarKarandikarsagark@eecs.berkeley.edu

KVMForum2016

https://github.com/riscv/riscv-qemu