Embed Size (px)
Citation preview
QEMUSupportfortheRISC-VInstructionSetArchitecture
KVMForum2016
https://github.com/riscv/riscv-qemu
Outline
• WhyRISC-V?• BenefitsofanOpenISA• RISC-VISABasics• VirtualizationSupport• QEMURISC-VTargetSupport• WorkinProgress/TODOsforUpstreaming/FutureWork
ISAsdon’tmatter
• Mostoftheperformanceandenergyofrunningsoftwareonacomputerisdueto:• Algorithms• Applicationcode• Compilers• OS/Runtimes• ISA• Microarchitecture(coreandmemoryhierarchy)• Circuitdesign• Physicaldesign• Fabricationprocess
+Inasystem,therearealsodisplays,radios,DC/DCconverters,sensors,actuators…
WhyInstructionSetArchitectureMatters
• Whycan’tIntelsellmobilechips?• 99%+ofmobilephones/tabletsbasedonARMv7/v8ISA
• Whycan’tARMpartnerssellservers?• 99%+oflaptops/desktops/serversbasedonAMD64ISA(over95%+builtbyIntel)
• HowcanIBMstillsellmainframes?• IBM360,oldestsurvivingISA(50+years)
ISAisthemostimportantinterfaceinacomputersystemwheresoftwaremeetshardware
WhysomanyISAsonanSoC?
• Applicationsprocessor(usuallyARM)• Graphicsprocessors• Imageprocessors• RadioDSPs• AudioDSPs• Securityprocessors• Power-managementprocessor• ….• AppsprocessorISA(e.g.ARM)toolargeformostaccelerators• IPboughtfromdifferentplaces,eachproprietaryISA• Home-grownISAcores• OveradozenISAsonsomeSoCs– eachwithuniquesoftwarestack
NVIDIATegraSoC
DoweneedallthesedifferentISAs?
Musttheybeproprietary?
WhatiftherewereonefreeandopenISAeveryonecoulduseforeverything?
ISAsshouldbeFreeandOpen
• WhileISAsmaybeproprietaryforhistoricalorbusinessreasons,thereisnogoodtechnicalreasonforthelackoffree,openISAs• It’snotanerrorofomission• Norisitbecausethecompaniesdomostofthesoftwaredevelopment• NeitherdocompaniesexclusivelyhavetheexperienceneededtodesignacompetentISA• NorarethemostpopularISAswonderfulISAs• NeithercanonlycompaniesverifyISAcompatibility• Finally,proprietaryISAsarenotguaranteedtolast
BenefitsofaViableFreeandOpenISA
• Greaterinnovationviafree-marketcompetition frommanycoredesigners,closed-sourceandopen-source• Sharedopencoredesigns,shortertimetomarket,lowercostfromreuse,fewererrorsgivenmoreeyeballs• Processorsbecomingaffordableformoredevices,whichwouldhelpexpandtheInternetofThings(IoTs),whichcouldcostaslittleas$1• Softwarestackssurviveforlongtime upgradesoftwareonsystemsembeddedinconcrete50yearsago• Makearchitectureresearchandeducationmorereal withfullyopenhardwareandsoftwarestacks
RISC-VOrigins
• In2010,aftermanyyearsandmanyprojectsusingMIPS,SPARC,andx86asbasisofresearch,itwastimefortheComputerScienceteamatUCBerkeleytolookatwhatISAstousefortheirnextsetofprojects• Obviouschoices:x86andARM• x86impossible– toocomplex,IPissues
Intelx86“AAA”Instruction
• ASCIIAdjustAfterAddition• ALregisterisdefaultsourceanddestination• Ifthelownibbleis>9decimal,ortheauxiliarycarryflagAF=1,then• Add6tothelownibbleofALanddiscardoverflow• IncrementhighbyteofAL• SetCFandAF
• Else• CF=AF=0
• Asingle-byteinstruction
RISC-VOrigins
• In2010,aftermanyyearsandmanyprojectsusingMIPS,SPARC,andx86asbasisofresearch,itwastimefortheComputerScienceteamatUCBerkeleytolookatwhatISAstousefortheirnextsetofprojects• Obviouschoices:x86andARM• x86impossible– toocomplex,IPissues• ARMmostlyimpossible– complex,IPissues
• UCBerkeleystarteda“3-monthproject”duringsummerof2010todeveloptheirownclean-slateISA• AndrewWaterman,YunsupLee,DavePatterson,KrsteAsanovicprincipaldesigners
RISC-VBackground(cont’d)
• Fouryearslater,inMayof2014,UCBerkeleyreleasedfrozenbaseuserspec• Manychiptapeoutsandseveralresearchpublicationsalongtheway
• ThenameRISC-V(pronouncedrisk-five),waschosentorepresentthefifthmajorRISCISAdesigneffortatUCBerkeley• RISC-I,RISC-II,SOAR,andSPURwerethefirstfourwiththeoriginalRISC-Ipublicationsdatingbackto1981
• InAugust2015,articlesofincorporationwerefiledtocreateanon-profitRISC-VFoundationtogoverntheISA
RISC-VisNOTanOpen-SourceProcessor
• RISC-VisanISAspecification– NOTanopen-sourceprocessorcore• Mostofthecostofchipdesign isinsoftware,sowewanttomakesuresoftwarecanbereusedacrossmanychipdesigns• TheFoundationwillencouragebothopen-sourceandproprietaryimplementationsoftheRISC-VISAspecification
UCBerkeleyRISC-VCores:Raven-1 Raven-2
Raven-3
Raven-4
EOS14
EOS16
EOS18
EOS20
2011 2012 2013 2014 2015
May Apr Aug Feb Jul Sep Mar Nov Mar
EOS:IBM45nmSOIRaven:ST28nmFDSOIHurricane:ST28nmFDSOISWERVE:TSMC28nm
SWERVE
EOS22EOS24
1GHz50+DPGFLOPS/W
1.65GHz14DPGFLOPS/W
Hurricane-1
2016
IndustrialSupport– PlatinumFoundingMembers
IndustrialSupport– Gold,Silver,AuditorFoundingMembers
RumbleDevelopment
TheRISC-VISA
• RV32,RV64,RV128variantsfor32b,64b,128baddressspacesdefined• BaseISAonly<50integerinstructions,butsupportscompiler,linker,OS,etc.• Extensionsprovidefullgeneral-purposeISA,includingIEEE-754/2008floating-point• ComparableISA-levelmetricstootherRISCs• Designedforextension,customization• Twelve64-bitsiliconprototypeimplementationscompletedatBerkeleysofar(45nm,28nm)
RISC-VStandardBaseISADetails
• 32-bit,fixed-width,naturallyalignedinstructions• 31integerregistersx1-x31,plusx0zeroregister• rd/rs1/rs2infixedlocation,noimplicitregisters• Immediatefield(instr[31])alwayssign-extended• Floating-pointaddsf0-f31registersplusFPCSR,alsofusedmul-addfour-registerformat• DesignedtosupportPICanddynamiclinking
RV64GDefinition
• G=I,M,A,F,D• I = BaseIntegerISA• M = StandardIntegerMultiplication/DivisionExtension• A = StandardAtomicsExtension• F = StandardSingle-precisionFloating-pointextension• D = StandardDouble-precisionfloating-pointextension
• Thisisthestandard,generalpurposeversionoftheISA,whatisimplementedinQEMU
RISC-VPrivilegedSpecification
• FourPrivilegeModes:User,Supervisor,Hypervisor,Machine• MachineModerequired• Common:ProvideM,S,UforrunningUnix-likeOSes(QEMUdoesthis)• VirtualMemoryArchitecturedesignedtosupportcurrentUnix-likeoperatingsystems• Sv39(RV64)• Demand-paged39-bitvirtual-addressspaces• 3-levelpagetable• 4KiBpages,2MiBmegapages,1GiBgigapages
• AlsoSv32(RV32)andSv48,Sv57,Sv64(RV64)
RISC-VVirtualization
• ISAdesignedwithvirtualizationin-mindfromthebeginning,evenwhenonlyusingU+S+Mmodes• “Theprivilegedarchitectureisdesignedtosimplifytheuseofclassicvirtualizationtechniques,whereaguestOSisrunatuser-level,asthefewprivilegedinstructionscanbeeasilydetectedandtrapped.”– RISC-VPrivilegedArchitecturev1.9Manual
• AvoidingSomeClassicalVirtualizationPitfalls…
HandlingSensitive,butUnprivilegedInstructions• Inx86,fortheoriginalVMware– “TableIIliststhe[19]instructionsofthex86architecturethatunfortunatelyviolatedPopekandGoldberg’sruleandhencemadethex86non-virtualizeable”1
• InRISC-V,no“hidden”privilegedstatereads/writes• Smallsetofprivilegedinstructionsthatcanmodifyspaceofprivilegedstate(ControlStatusRegisters,orCSRs)
1.E.Bugnion,S.Devine,M.Rosenblum,J.Sugerman,andE.Y.Wang.2012.BringingVirtualizationtothex86ArchitecturewiththeOriginalVMwareWorkstation. ACMTrans.Comput.Syst. 30,4,Article12(November2012),51pages.
TrackingChangesinVirtualMachineMemory
• Inx86,fortheoriginalVMware– “…privilegedhardwareregisterscontaintheaddressofsegmentdescriptortablesandpagetables...but regularloadandstoreinstructionscanaccessthesestructuresinmemory.”1
• InRISC-V,stilluseregularloads/storestomodifymemorymanagementstate• PrivilegedSFENCE.VM instructionrequiredbyspec.aftermodifyingmemorymanagementstate
1.E.Bugnion,S.Devine,M.Rosenblum,J.Sugerman,andE.Y.Wang.2012.BringingVirtualizationtothex86ArchitecturewiththeOriginalVMwareWorkstation. ACMTrans.Comput.Syst. 30,4,Article12(November2012),51pages.
VirtualizingSegmentation
• Inx86,fortheoriginalVMware– Complicatedinteractionsbetweensegmentdescriptortablesandsegmentregisters,withvisibleandhiddenfields.Hiddenpiecesupdatedbyinstructionsorfaults.CausesproblemswithextrafaultsintroducedbyaVMM.1
• InRISC-V,nox86-stylesegmentation• Limitedbase-and-boundsmodewith2“segments”• Mostsoftwarewillusepaginginstead
1.E.Bugnion,S.Devine,M.Rosenblum,J.Sugerman,andE.Y.Wang.2012.BringingVirtualizationtothex86ArchitecturewiththeOriginalVMwareWorkstation. ACMTrans.Comput.Syst. 30,4,Article12(November2012),51pages.
RISC-VVirtualizationStacks
• Providecleansplitbetweenlayersofthesoftwarestack• ApplicationcommunicateswithApplicationExecutionEnvironment(AEE)viaApplicationBinaryInterface(ABI)• OScommunicatesviaSupervisorExecutionEnvironment(SEE)viaSystemBinaryInterface(SBI)• HypervisorcommunicatesviaHypervisorBinaryInterface(HBI)toHypervisorExecutionEnvironment(HEE)• AlllevelsoftheISAdesignedtosupportvirtualization
2 1.1draft: Volume II: RISC-V Privileged Architectures
ApplicationABIAEE
ApplicationABI
OSSBISEE
ApplicationABI
SBIHypervisor
ApplicationABI
OS
ApplicationABI
ApplicationABI
OS
ApplicationABI
SBI
HBIHEE
Figure 1.1: Di↵erent implementation stacks supporting various forms of privileged execution.
the OS, which provides the AEE. Just as applications interface with an AEE via an ABI, RISC-Voperating systems interface with a supervisor execution environment (SEE) via a supervisor binaryinterface (SBI). An SBI comprises the user-level and supervisor-level ISA together with a set ofSBI function calls. Using a single SBI across all SEE implementations allows a single OS binaryimage to run on any SEE. The SEE can be a simple boot loader and BIOS-style IO system in alow-end hardware platform, or a hypervisor-provided virtual machine in a high-end server, or athin translation layer over a host operating system in an architecture simulation environment.
The rightmost configuration shows a virtual machine monitor configuration where multiple multi-programmed OSs are supported by a single hypervisor. Each OS communicates via an SBI with thehypervisor, which provides the SEE. The hypervisor communicates with the hypervisor executionenvironment (HEE) using a hypervisor binary interface, to isolate the hypervisor from details ofthe hardware platform.
Our graphical convention represents abstract interfaces using black boxes with white text, toseparate them from actual components.
The various ABI, SBI, and HBIs are still a work-in-progress, but we anticipate the SBI and HBIto support devices via virtualized device interfaces similar to virtio [2], and to support devicediscovery. In this manner, only one set of device drivers need be written that can support anyOS or hypervisor, and which can also be shared with the boot environment.
Hardware implementations of the RISC-V ISA will generally require additional features beyond theprivileged ISA to support the various execution environments (AEE, SEE, or HEE), but these weconsider separately as part of a hardware abstraction layer (HAL), as shown in Figure 1.2. Note
ApplicationABIAEEHAL
Hardware
ApplicationABI
OSSBISEE
ApplicationABI
HALHardware
SBIHypervisor
ApplicationABI
OS
ApplicationABI
ApplicationABI
OS
ApplicationABI
SBI
HBIHEE
HardwareHAL
Figure 1.2: Hardware abstraction layers (HALs) abstract underlying hardware platforms from theexecution environments.
RISC-VHypervisorSpecification- WIP
• CurrentprivilegeddesigncanhaveanM-modemonitorthatprovidesphysicalresourcepartitioning,canactassimplehypervisor• UpcomingHypervisorExtensionSpecificationfor“full”Hypervisors• Rightnow,anemptyslotintheprivilegedspecification
• Wanttogetinvolved?• HypervisorSpecificationDraftwillmaketheroundssoononisa-dev@groups.riscv.org mailinglist
TheRISC-VEcosystem
• SoftwareTools• GCC/glibc/GDB• LLVM/Clang• Linux• Yocto• VerificationSuite
• HardwareTools• ZynqFPGAInfrastructure• Chisel
• SoftwareImplementations• Spike,“Golden-standard”ISASimulator• ANGEL,JavaScriptISASimulator• QEMU
• HardwareImplementations• RocketChipGenerator
• RV64Gsingle-issuein-orderpipeline• SodorProcessorCollection• BOOM(BerkeleyOut-of-OrderMachine)
github.com/riscvandgithub.com/ucb-bar
RISC-VTargetsupportforQEMU
• Maintainedathttps://github.com/riscv/riscv-qemu• QEMUfull-systememulation• QEMUonmodernx86iscurrentlythefastestRISC-Vimplementation• AbighelpinRISC-Vsoftwaredevelopment
Spike QEMU
TimelineofRISC-V“Firsts”
2014 2015 2016
riscv-qemuWorkStarted
1st LinuxBootonQEMU
FastestRISC-VImplementation
1st RISC-VImplementation
w/TCP/IPNetworking
1st PythonBringuponRISC-V
1st JavaBringuponRISC-V(HotspotZeroJVM)
1st RISC-VCoreBuiltonRISC-VSystem
withChisel
…allonQEMU!
RISC-VChipDevelopmentonRISC-V
+4
Instruction Mem
RegFile
IType SignExtend
DecoderData Mem
ir[24:20]
branch
pc+4
pc_s
el
ir[31:20]
rs1
ALU
ControlSignals
wb_s
el
RegFile
rf_we
n
val
mem
_rw
PC
mem
_val
addrwdata
rdata
Inst
JumpTargGen
BranchTargGen
ir[19:15]
ir[31:25],ir[11:7]
PC+4jalr
rs2
BranchCondGen
br_eq?br_lt?
co-p
roce
ssor
(CSR
) reg
ister
s
ir[11
:7]
jump
ir[31:12]
Execute Stage
br_ltu?PC
addr
ir[31:12]
JumpRegTargGen
Op2Sel
Op1SelAluFun
data
wa
wd
en
addr data
UType
Note: for simplicity, the CSR File (control and status registers) and associated datapath is not shown
RISC-V Sodor 1-Stage
exception
SType SignExtend
ir[31:20]
PC
rs2rs1
rs2
BuildscreenshotcourtesyMichaelKnyszek
TimelineofRISC-V“Firsts”
2014 2015 2016
riscv-qemuWorkStarted
1st LinuxBootonQEMU
FastestRISC-VImplementation
1st RISC-VImplementation
w/TCP/IPNetworking
1st PythonBringuponRISC-V
1st JavaBringuponRISC-V(HotspotZeroJVM)
1st RISC-VCoreBuiltonRISC-VSystem
withChisel
QEMURISC-V
Priv.Spec1.7Bump
QEMURISC-V
Priv.Spec1.9Bump
UpstreamingBegins
…allonQEMU!
1st RISC-VSystem
w/RemoteGDB
RISC-VTargetsupportforQEMU
• RISC-Vsupportstartedin2014,evolvesastheISAdoes• SupportsRV64IMAFD(“RV64G”)Full-systememulation• UserISAsupportlargelyunchangedsincethen(currentlyv2.0)• PrivilegedISAnearingstandardization(currentlyv1.9)
• FuturePriv.SpecupgradestoQEMUmuchsimplerduetostructuralsimilarityofpriv.modeemulationcodewithSpike• Pre-1.7->1.7bump,~1month• 1.7->1.9bump,3days
• I/Ocurrentlylimitedto“Host-TargetInterface”(HTIF)devices• EnoughtobootLinux,interactthroughconsole• Otherdevicespreviouslyshoehornedin(networking,displays,consoles)
• Mainlywaitingonplatformstandardizationandsoftwaresupport
hw/riscv/riscv_board.c
• Reference“board”designedtomatchSpike• Providessimplehardware,config:• HTIF-basedconsole(simple,non-standardconsoledeviceforearlyboot)• “Loopback”SoftwareInterruptEmulation• RTC/TimerscompliantwithRISC-Vv1.9privilegedspecification• ResetVector(BootROM)• ConfigurationString
0x8000_0000
0x0000_0000
0x0000_1000
0x4000_0000
... DRAM ...
Conf. StringReset Vector
RTCTimers
DRAM Base
DRAM Top
SoftInt Emu.
Base + Size
High
I/O:HTIF(old),Debug(new)
• Host-TargetInterface(HTIF)isarelicofBerkeleyTestChips• Two64-bitregisters,fromhostandtohost• Formerlyprovidednetwork,blockdevice,console,nowusedonlyforconsole,signalingtestcompletion• Debugging:HTIFbeingphasedoutonrealhardware,replacedwithdraftDebugUnitspecification(willbestandardized)• I/O:Realhardware/softwaresimulatorswillalsophaseoutHTIFandmovetostandarddevicesassoftwaresupportprogresses
SoftwareStackinsideRISC-VQEMU
• M-moderunssecurebootandmonitor(currentlybbl)• S-moderunsOS(OSalwaysrunsvirtualized)• U-moderunsapplicationontopofOSorM-mode
M-modemonitor
U-modesystemprocess
S-modeOS
Device2Interrupts
Device1Interrupts
OtherInterrupts
U-modeapp
BootUp
BinarysuppliedtoQEMUcontains:• bbl – “BerkeleyBootLoader”performsMachine-Modemanagementofthesystem,exposesSBItoOSes• vmlinux – Linuxkernelaspayloadforbbl• Includesaramdiskforrootfs (currentlylimiteddevicesupport)
Systembootsintohardcodedbootrom,jumpstobbl,bbl initializesthesysteminM-mode,setsupSupervisorExecutionEnvironment(SEE),thenloadsandrunssuppliedkernel,e.g.Linux
Testing/Debugging
• TheusualGDB,bruteforce,etc• Passestheriscv-testsstandardtestsuite• BootsLinux
• Withreasonablylargesoftwarestacksontop– Python,Java,etc.
• Fuzztestingagainstthe“GoldenStandard”
FuzzTestingwithriscv-torture• Generatesarandomsequenceofinstructionsbasedonsuppliedparameters(%ofmeminstructions,floatingpointinstructions,integerinstructions,etc.)• RunscodeonSpikeandotherimplementationofyourchoice
• Spikeisthe“GoldenReference”RISC-VSimulator,writtenbytheauthorsoftheRISC-VSpecs
• Dumpregisterstateattheendandcompare• Onfailure,binarysearchtopinpointinstructionwherethingsgowrong• Scriptsforrunningriscv-tortureonQEMUavailableathttps://github.com/sagark/riscv-qemu-torture• Accumulated384hoursoffailure-freetestingsinceAugustPriv.1.9update!
target-riscv SLoC*
• ARM- 45,438• MIPS- 37,501• x86- 30,437• RISC-V- 5,074
WorkinProgress/TODOs
• Functionality:Standarddevicesupport(comboofQEMU+Linux)• Upstreaming!– planningtostartinmid-September
• SubmittedagiantpatchinFebruaryasproof-of-existenceforgauginginterestinGSoC
• Somecleanupbasedongiant-patchfeedbackdone(e.g.usebuilt-inFPU)• LatestRISC-VPrivilegedSpec.v1.9bumpdone• TODO:
• Morecleanupbasedongiant-patchfeedback,checkpatch• Rebasetomaster(currentlyv2.5.0)• Small,logicalpatches
• Future:• SupportotherISAvariants,likeRV32,CompressedISA,QEMULinux-UserMode
