Thoughts on Metrics & Thoughts on Metrics & BenchmarkingBenchmarking

Thoughts on Metrics & Thoughts on Metrics & BenchmarkingBenchmarking

Randal E. Bryant

– 2 –

Program ComplexityProgram Complexity

Complexity SourcesComplexity Sources Control

Concurrency Error handling & mitigation

Data Complex data structures Data structure invariants

Data

Control

EmbeddedSystems

My Systems

– 3 –

Role of BenchmarksRole of Benchmarks

GoodGood Provides objective standard for comparison Competition drives progress

BadBad Can become goal rather than means to achieve goal Poorly selected set can thwart progress

E.g., DIMACS SAT benchmarks

– 4 –

Benchmarking DesiderataBenchmarking Desiderata

Organic ProcessOrganic Process E.g., SPEC, SAT Continually update Large community involved

SelectionSelection Range of sizes From different sources (applications, organizations)

Desirable, but Uncommon:Desirable, but Uncommon: Parameterized Access to originating application

– 5 –

Some IdeasSome Ideas

Select Small Set of Driving ApplicationsSelect Small Set of Driving Applications Control intensive embedded system(s) Data intensive

Create Supporting BenchmarksCreate Supporting Benchmarks Scaled down versions or components of driving applications

IssuesIssues Language Ancilliary code: Libraries, OS Specs.