Benchmarking Benchmarking for [Physical] Synthesisfor [Physical] Synthesis

Igor Markov Igor Markov and and Prabhakar KudvaPrabhakar Kudva

The Univ. of MichiganThe Univ. of Michigan // IBM IBM

In This Talk …In This Talk …

BenchmarkBenchmarkinging vs benchmarks vs benchmarksBenchmarking exposes new research QsBenchmarking exposes new research QsWhy industry should care about Why industry should care about

benchmarkingbenchmarkingWhat is What is (and is not)(and is not) being done being done

to improve benchmarking infrastructureto improve benchmarking infrastructureNot in this talk, but in a focus groupNot in this talk, but in a focus group

Incentives for verifying published workIncentives for verifying published workHow to accelerate a culture changeHow to accelerate a culture change

BenchmarkBenchmarkinging

Design benchmarksDesign benchmarks Data model / representation; Data model / representation; InstancesInstances

Objectives (QOR metrics) and constraintsObjectives (QOR metrics) and constraints Algorithms, methodologies; Algorithms, methodologies; ImplementationsImplementations

Solvers: Solvers: dittoditto Empirical and theoretical analyses, e.g.,Empirical and theoretical analyses, e.g.,

Hard vs easy benchmarks (Hard vs easy benchmarks (regardless of sizeregardless of size)) Correlation between different objectivesCorrelation between different objectives Upper / lower bounds for QOR, statistical behavior, etcUpper / lower bounds for QOR, statistical behavior, etc

Dualism between benchmarks and solversDualism between benchmarks and solvers For more details, see For more details, see http://gigascale.org/bookshelfhttp://gigascale.org/bookshelf

Industrial BenchmarkingIndustrial Benchmarking

Growing size & complexity of VLSI chipsGrowing size & complexity of VLSI chipsDesign objectivesDesign objectives

Area / power / yield / etcArea / power / yield / etc

Design constraintsDesign constraintsTiming / FP + fixed-die partitions / fixed IPs /Timing / FP + fixed-die partitions / fixed IPs /

routability / pin access / signal integrity…routability / pin access / signal integrity…

Can the same algo excel in all contexts?Can the same algo excel in all contexts?Sophistication of layout and logic motivate Sophistication of layout and logic motivate

open benchmarking for Synthesis and P&Ropen benchmarking for Synthesis and P&R

Design TypesDesign Types ASICASICss

Lots of fixed I/Os, few macros, millions of standard cellsLots of fixed I/Os, few macros, millions of standard cells Design densities : 40-80% (IBM)Design densities : 40-80% (IBM) Flat and hierarchical designsFlat and hierarchical designs

SoCSoCss Many more macro blocks, coresMany more macro blocks, cores Datapaths + control logicDatapaths + control logic Can have very low design densities : < 20%Can have very low design densities : < 20%

Micro-Processor (Micro-Processor (PP) Random Logic Macros() Random Logic Macros(RLMRLM)) Hierarchical partitions are LS+P&R instances (5-30K)Hierarchical partitions are LS+P&R instances (5-30K) High placement densities : 80%-98% (low whitespace)High placement densities : 80%-98% (low whitespace) Many fixed I/Os, relatively few standard cellsMany fixed I/Os, relatively few standard cells Note: “Partitioning w Terminals” DAC`99, ISPD `99, ASPDAC`00

Why Invest in BenchmarkingWhy Invest in Benchmarking AcademiaAcademia

Benchmarks can identify / capture Benchmarks can identify / capture new research problemsnew research problems

Empirical validation of novel researchEmpirical validation of novel research Open-source tools/BMs can be analyzed and tweakedOpen-source tools/BMs can be analyzed and tweaked

IndustryIndustry Evaluation and transfer of academic researchEvaluation and transfer of academic research Support for executive decisionsSupport for executive decisions

(which tools are relatively week & must be improved)(which tools are relatively week & must be improved) Open-source tools/BMs can be analyzed and tweakedOpen-source tools/BMs can be analyzed and tweaked

When is an EDA problem (not) solved?When is an EDA problem (not) solved? Are there good solver implementations?Are there good solver implementations? Can they “solve” existing benchmarks?Can they “solve” existing benchmarks?

Participation / Leadership NecessaryParticipation / Leadership Necessary

Activity 1Activity 1: Benchmarking platform / flows: Benchmarking platform / flowsActivity 2Activity 2: Establishing common evaluators: Establishing common evaluators

Static timing analysisStatic timing analysisCongestion / yield predictionCongestion / yield predictionPower estimationPower estimation

Activity 3Activity 3: Standard-cell libraries: Standard-cell librariesActivity 4Activity 4: Large designs w bells & whistles: Large designs w bells & whistlesActivity 5Activity 5: Automation of benchmarking: Automation of benchmarking

Activity 1Activity 1: Benchmarking Platform: Benchmarking Platform

Benchmarking “platform”: a reasonable subset ofBenchmarking “platform”: a reasonable subset of data modeldata model specific data representations (e.g., file formats)specific data representations (e.g., file formats) access mechanisms (e.g., APIs)access mechanisms (e.g., APIs) reference implementation (e.g., a design database)reference implementation (e.g., a design database) design examples in compatible formatsdesign examples in compatible formats

Base platforms available (next slide)Base platforms available (next slide) More participation necessaryMore participation necessary

regular discussionsregular discussions additional tasks / features outlinedadditional tasks / features outlined

Common Methodology PlatformCommon Methodology Platform

Synthesis (SIS, MVSIS…)

Placement(Capo, Dragon, Feng Shui, mPl,…)

Common Model(Open Access?)

Blif Bookshelf format

Blue Flow exists, Common model hooks: To be Done

Placement UtilitiesPlacement Utilities

http://vlsicad.eecs.umich.edu/BK/PlaceUtils/http://vlsicad.eecs.umich.edu/BK/PlaceUtils/ Accept input in the GSRC Bookshelf formatAccept input in the GSRC Bookshelf format Format convertersFormat converters

LEF/DEF LEF/DEF Bookshelf Bookshelf Bookshelf Bookshelf Kraftwerk (DAC98 BP, E&J) Kraftwerk (DAC98 BP, E&J) BLIF(SIS) BLIF(SIS) Bookshelf Bookshelf

Evaluators, checkers, Evaluators, checkers, postprocessors and plotterspostprocessors and plotters Contributions in these categories are welcomeContributions in these categories are welcome

Placement Utilities (cont’d)Placement Utilities (cont’d)Wirelength CalculatorWirelength Calculator (HPWL) (HPWL)

Independent evaluation of placement resultsIndependent evaluation of placement resultsPlacement PlotterPlacement Plotter

Saves gnuplot scripts (Saves gnuplot scripts ( .eps, .gif, …) .eps, .gif, …)Multiple views (cells only, cells+nets, rows,…)Multiple views (cells only, cells+nets, rows,…)

Probabilistic Congestion MapsProbabilistic Congestion Maps (Lou et al.) (Lou et al.)GnuplotGnuplot scripts scriptsMatlabMatlab scripts scripts

better graphics, including 3-d fly-by viewsbetter graphics, including 3-d fly-by views

.xpm files (.xpm files ( .gif, .jpg, .eps, …) .gif, .jpg, .eps, …)

Placement Utilities (cont’d)Placement Utilities (cont’d)Legality checkerLegality checkerSimple legalizerSimple legalizerLayout GeneratorLayout Generator

Given a netlist, creates a row structureGiven a netlist, creates a row structureTunable %whitespace, aspect ratio, etcTunable %whitespace, aspect ratio, etc

All available in binaries/PERL atAll available in binaries/PERL athttp://http://vlsicad.eecs.umich.edu/BK/PlaceUtilsvlsicad.eecs.umich.edu/BK/PlaceUtils//

Most source codes are shipped w CapoMost source codes are shipped w Capo

Activity 2Activity 2: Creating Evaluators: Creating Evaluators

Contribute measures/analysis tools for:Contribute measures/analysis tools for:Timing AnalysisTiming AnalysisCongestion/YieldCongestion/YieldPowerPowerAreaAreaNoise….Noise….

BenchmarkBenchmarkinging Needs for Timing Opt. Needs for Timing Opt.

A common, reusable STA methodologyA common, reusable STA methodology High-quality, open-source infrastructureHigh-quality, open-source infrastructureFalse paths; realistic gate/delay modelsFalse paths; realistic gate/delay models

Metrics validated against phys. synthesisMetrics validated against phys. synthesisThe simpler the better,The simpler the better,

but must be good predictorsbut must be good predictorsBuffer insertion profoundly impacts layoutBuffer insertion profoundly impacts layout

The use of linear wirelength in timing-driven layout The use of linear wirelength in timing-driven layout assumes buffers insertionassumes buffers insertion (min-cut vs quadratic) (min-cut vs quadratic)

Apparently, synthesis is affected tooApparently, synthesis is affected too

Vertical BenchmarksVertical Benchmarks ““Tool flow” Tool flow”

Two or more EDA tools, chained sequentiallyTwo or more EDA tools, chained sequentially(potentially, part of a complete design cycle)(potentially, part of a complete design cycle)

Sample contexts: Sample contexts: physical synthesisphysical synthesis, place & route, , place & route, retiming followed by sequential verificationretiming followed by sequential verification

Vertical benchmarksVertical benchmarks Multiple, redundant snapshots of a tool flowMultiple, redundant snapshots of a tool flow

sufficient info for detailed analysis of tool performancesufficient info for detailed analysis of tool performance Herman Schmit @CMU is maintaining Herman Schmit @CMU is maintaining

a resp. slot in the VLSI CAD Bookshelfa resp. slot in the VLSI CAD Bookshelf See See http://gigascale.org/bookselfhttp://gigascale.org/bookself Include flat gate-level netlistsInclude flat gate-level netlists Library information ( < 250nm)Library information ( < 250nm) Realistic timing & fixed-die constraintsRealistic timing & fixed-die constraints

Infrastructure NeedsInfrastructure Needs

Need Need common evaluatorscommon evaluators of delay / power of delay / powerTo avoid inconsistent / outdated resultsTo avoid inconsistent / outdated results

Relevant initiatives from Si2Relevant initiatives from Si2OLA (Open Library Architecture)OLA (Open Library Architecture)OpenAccess OpenAccess For more info, see For more info, see http://www.si2.orghttp://www.si2.org

Still: no reliable public STA toolStill: no reliable public STA toolSought: OA-based utilities for timing/layoutSought: OA-based utilities for timing/layout

Activity 3Activity 3: : Standard-cell LibrariesStandard-cell Libraries

Libraries carry technology informationLibraries carry technology information Impact of wirelength delays Impact of wirelength delays

increases in recent technology generationsincreases in recent technology generations Cell characteristics must be compatibleCell characteristics must be compatible

Some benchmarks in the BookshelfSome benchmarks in the Bookshelfuse use 0.250.25mm and and 0.350.35mm libraries libraries Geometry info is there, + timing (in some cases)Geometry info is there, + timing (in some cases)

Cadence test library?Cadence test library? Artisan libraries?Artisan libraries? Use commercial tools to create librariesUse commercial tools to create libraries

Prolific, Cadabra,…Prolific, Cadabra,…

Activity 4Activity 4: Need New Benchmarks: Need New BenchmarksTo Confirm / Defeat Tool TuningTo Confirm / Defeat Tool Tuning

Data on tuning from the ISPD03 paperData on tuning from the ISPD03 paper“Benchmarking for Placement”, Adya et al.“Benchmarking for Placement”, Adya et al.

Observe thatObserve that CapoCapo does well on does well on Cadence-Capo, grid-like circuitsCadence-Capo, grid-like circuits DragonDragon does well on does well on IBM-Place (IBM-Dragon)IBM-Place (IBM-Dragon) FengShuiFengShui does well on does well on MCNC benchmarksMCNC benchmarks mPLmPL does well on does well on PEKOPEKO

This is hardly a coincidenceThis is hardly a coincidence Motivation for more / better benchmarksMotivation for more / better benchmarks P.S. P.S. Most differences above have been explained,Most differences above have been explained,

all placers above have been improvedall placers above have been improved

Activity 4Activity 4: Large Benchmark Creation: Large Benchmark Creation

www.opencores.org has large designs has large designsMay be a good starting point –May be a good starting point –

use vendor tools to create blif filesuse vendor tools to create blif files(+post results)(+post results)

Note: there may be different ways to convert Note: there may be different ways to convert A group of design housesA group of design houses

( (IBM, Intel, LSI, HPIBM, Intel, LSI, HP))is planning a release of new largeis planning a release of new largegate-level benchmarks gate-level benchmarks for layoutfor layoutProbably no logic informationProbably no logic information

Activity 5Activity 5: Benchmarking Automation: Benchmarking Automation Rigorous benchmarking is laborious. Risk of errors is highRigorous benchmarking is laborious. Risk of errors is high How do we keep things simple / accessible?How do we keep things simple / accessible? Encapsulate software management in an ASPEncapsulate software management in an ASP

Web uploads for binaries and source in tar.gz w MakefilesWeb uploads for binaries and source in tar.gz w Makefiles Web uploads for benchmarksWeb uploads for benchmarks GUI interface for GUI interface for NxMNxM simulations; tables created automatically simulations; tables created automatically GUI interface for composing tool-flows; flows can be saved/reusedGUI interface for composing tool-flows; flows can be saved/reused Distributed back-end includes job schedulingDistributed back-end includes job scheduling Email notification of job completionEmail notification of job completion All files created are available on the Web (permissions & policies)All files created are available on the Web (permissions & policies) Anyone can re-run / study your experiment or interface with itAnyone can re-run / study your experiment or interface with it

Follow-on Action PlanFollow-on Action Plan

Looking for volunteers to Looking for volunteers to -test Bookshelf.exe-test Bookshelf.exe Particularly, in the context of synthesis & verificationParticularly, in the context of synthesis & verification Contact: Igor Contact: Igor imarkov@eecs.umich.eduimarkov@eecs.umich.edu

Create a joint benchmarking groupCreate a joint benchmarking groupfrom industry and academiafrom industry and academia Contact: Prabhakar Contact: Prabhakar kudva@us.ibm.comkudva@us.ibm.com Regular discussionsRegular discussions

Development basedDevelopment basedon common infrastructureon common infrastructure