Keynote 3 Foster TrendsFunctionVerification

7/21/2019 Keynote 3 Foster TrendsFunctionVerification

http://slidepdf.com/reader/full/keynote-3-foster-trendsfunctionverification 1/76

Industry Pulse:

Trends in Functional Verification

Chief Scientist Verification

Design Verification Technology

Memo CODE 2013

Harry Foster



www.mentor.com© 2013 Mentor Graphics Corp. Company Confidential

Extrapolating From Current ConditionsDisregards Future Innovation

HF, MemoCODE, 20132

“In 1910, in the early history telephony, a Bell telephone statistician

projected a massive ramp-up in switchboard operator jobs as telephoneuse grew, until “every woman in America” would be required.”

Source: Future Savvy: Identifying trends to Make Better Decisions, Manage Uncertainty, and Profit From Change Adam Gordon, 2008




Functional Verification Market According to EDAC

700

965

0

200

400

600

800

1000

2010 2012

M i l l i o n s ( $

)

HF, MemoCODE, 20133

38% Growth Between 2010 2012

EDAC: Market Statistics Service 2007 Annual Summary Report




Functional Verification Market According to EDAC

380430

190

365

130

170

0

100

200

300

400

2010 2012

M i l l i o n s

( $ )

Simulation 13% GrowthEmulation 94% Growth

Formal 31% Growth

HF, MemoCODE, 20134

EDAC: Market Statistics Service 2007 Annual Summary Report




HF, MemoCODE, 20135

Conducted by Wilson Research Group

— Commissioned by Mentor Graphics

— Format followed 2002, 2004 Collett studies for trend analysis, as

well as the 2007 FarWest Research Study

Worldwide study

— Overall confidence of 95% plus/minus 4.05%

This was a blind study!

— To eliminate any bias in the results

This was a balanced study!

— No single vendor dominated responses

Wilson Research Group

2012 Wilson Research GroupFunctional Verification Study




Who Participated In The SurveyParticipant’s market segment

HF, MemoCODE, 20136

Wilson Research Group and Mentor Graphics, 2012 Functional Verification Study, Used with permission

0%

10%

20%

30%

40%

50%

60%

ASIC Vendor Fabless IC Vendor

ICManufacturer

SystemsCompany

DesignServicesCompany

IP VendorCompany

Other

S t u d y P a r t i c i p a

n t s

Non-FPGA 2012

FPGA 2012


http://slidepdf.com/reader/full/keynote-3-foster-trendsfunctionverification 7/76www.mentor.com© 2013 Mentor Graphics Corp. Company Confidential

Who Participated In The SurveyParticipant’s job title

HF, MemoCODE, 20137


0%

10%

20%

30%

40%

50%

60%

S t u d y P

a r t i c i p a n t s

Non-FPGA 2012

FPGA 2012



Overview

Beyond Theory

Beyond Standards

Beyond the Status Quo

HF, MemoCODE, 20138



BEYOND THEORY

Beyond Theory in Terms of Rising Complexity

HF, MemoCODE, 20139



Difference Between Theory and Practice

HF, MemoCODE, 201310

In theory there is no difference betweentheory and practice, but in practice there is.





Theory: Everything is clear, but nothing works.





Practice: Everything works, but nothing is clear.





The problem is sometimes theory meets practice:

Nothing works and nothing is clear.

http://www.google.com/url?sa=i&rct=j&q=boat%20flipped&source=images&cd=&cad=rja&docid=jSiimmTOKzVhSM&tbnid=40AAk08FNA-Z2M:&ved=0CAUQjRw&url=http://jacksonkayak.com/blog/2012/09/15/deep-water-re-entry-in-a-coosa/&ei=feUUUoeALMKZiQLG9YHYCg&bvm=bv.51156542,d.cGE&psig=AFQjCNHEoS-ugzmRBc6fMGwTV1YF7MIM0g&ust=1377187570643316



Beyond Theory in Terms of Rising Complexity


What does this really mean?

What makes things complex?

How do we measure complexity?



What Makes Something Complex?


System consisting of many interconnected parts — Examining the individual parts tells you nothing about the system

Complex does not necessarily mean complicated



Designs are Getting More Complex

HF, MemoCODE, 2013


0%

5%

10%

15%

20%

25%

30%

35%

40%

S t u d y P

a r t i c i p a n t s 2007

2010

2012

2007: Mean 90nm2010: Mean 65nm2012: Mean 45nm

Process Geometry

16




HF, MemoCODE, 2013

0%

5%

10%

15%

20%

25%

30%

N

o n - F P G A S t u d y P a r t i c i p a n t s

2002

2007

2010

2012

About a 1/3rd of designs below 5M gates About a 1/3rd of design between 5M - 20M gates About a 1/3rd of designs great than 20M gates


Number of gates of logic and datapath, excluding memories

17




HF, MemoCODE, 2013


0.4

2.7

6.1

11.1

0

2

4

6

8

10

12

2002 2007 2010 2012

N o n - F P

G A M e a n D e s i g n S

i z e G a t e s ( M )

Mean number of gates of logic and datapath, excluding memories trends

18




21% 22%

28%

6%

10% 13%

0%

10%

20%

30%

40%

50%

60%

NONE 1 2 3 4 5 or MORE

N o n - F P G A S t u d y P


Number of Embedded Processors for Non-FPGA Designs

2004

2007

2010

2012

Source: Wilson Research Group and Mentor Graphics, 2012 Functional Verification Study


79% of designs contain one or more embedded processors




1.06

1.46

1.96

2.25

0

1

2

3

2004 2007 2010 2012

N o n - F P G A M e a n

n u m b e r

o f E m b e d d e d P r o

c e s s o r s



Mean number of embedded processors continues to rise




FPGAs are Getting Complex Too!

HF, MemoCODE, 2013


44%

38%

11%

1%4%

2%

21% 22%

28%

6%

10% 13%

0%

10%

20%

30%

40%

NONE 1 2 3 4 5 or MORE

S u r v e y P a r t i c i p

a n t s

Number of embedded processors

FPGA

Non-FPGA

56% of FPGAs contain one or more embedded processors

21




How do we measure complexity?


Computational complexity theory used in computer science

There are no generally accepted metrics!

O(1)

O(logn)

O(n)

O(nlogn)

O(n^2)

O(2^n)

O(n!)

O p e r a t i o n s

Elements




Is bug density a good proxy?

Single, sequential data

streams — Floating point unit — Graphics shading unit — DSP convolution unit — MPEG decode

— . . .

Multiple, concurrent datastreams — Cross bar — Bus traffic controller — DMA controller — Standard I/F (e.g., PCIe)

— . . .

Channel

Compressed

Audio

Data Link LayerTX

RX

PHY

Sequential data streams

1x number of bugs

Concurrent data streams

5x number of bugs

Encoder Decoder


- Ted Scardamalia, internal IBM study




Packet-Based Design

Transaction

Layer Packet

Reformater

Data Link

Layer Packet

Reformater

Retry BufferArbiter

Tx

Rx

From

Fabric ToPHY

From Rx

Channel

Concurrency is Complicated to Verify





Maybe effort is a good proxy?


A f f o r d a

b l e

E x p e n s i v e

P r o h i b i t i v e

C o s t / E f

f o r t

Complexity

CriticalThreshold




Verification Consumes Majority of Project Time

0%

5%

10%

15%

20%

25%

1%-20% 21%-30% 31%-40% 41%-50% 51%-60% 61%-70% 71%-80% >80%

N o n - F P G A S t u

d y P a r t i c i p a n t s

Time (Percent)

Total Project Time Spent in Verification

2007

2010

2012

2007: Mean 49%2010: Mean 56%2012: Mean 56%

Wilson Research Group and Mentor Graphics, 2012 Functional Verification Study





More and More Verification EngineersMean peak number of design vs. verification engineers

7.8 8.1 8.5

4.8

7.6 8.4

2007 2010 2012

~ 1-to-1 ratioof peak design

and verificationengineers

58% 11%

4% 5%

VerificationEngineers

DesignEngineers






Where Verification Engineers Spend Their Time

HF, MemoCODE, 2013


16%

22%

23%

36%

4%Test Planning

Testbench Development

Creating and Running Test

Debug

Other

More time spent in debugthan any other task!

28




54%

2007

46%

2007

47%

2012

53%

2012

Doing Design Doing Verification

Design Engineer Project Time2007 - 2012


Designers Doing More and More Verification





0

20

40

60

80

100

2007 2012 2017 2022 2027 2032 2037

At this rate… In 25 years, ALL of a designer’s

time will bedevoted to

verification

Project Time 2007 - 2037

Design Verification

Time Design EngineersSpends Doing:

T i m e ( P e r c e n

t )


Time Designers Spends in Design vs. Verification




Design Reuse Trends


41%

33%

13% 13%

28%

35%

22%

15%

0%

10%

20%

30%

40%

NEW LOGIC REUSED LOGIC(Developed in-house)

PURCHASED IP ANALOG, RF AND/OR MIXED SIGNAL

N o n - F P G A S t u d y P a r t i c i p a n t s

Design Composition

2007

2012

Source: Wilson Research Group and Mentor Graphics.




Verification Reuse

HF, MemoCODE, 2013


50%

41%

8%

39%

44%

17%

0%

10%

20%

30%

40%

50%

NEW REUSED FROM OTHER DESIGNS

ACQUIRED EXTERNALLY


Testbench Composition

2007

2012

Mean testbench composition trends

32




With All This Effort, How are We Doing?





Project’s Schedule Completion Trends

0%

5%

10%

15%

20%

25%

30%

35%

More than 10%EARLY

ON-SCHEDULE 20% 40% >50% BEHINDSCHEDULE

S t u d y P a r t i c i p a n t s

2007

2010

2012

2007: 67% behind schedule2010: 66% behind schedule2012: 67% behind schedule

Behind Schedule Ahead of schedule






FPGA vs. Non-FPGA Completion Trends

HF, MemoCODE, 2013


0%

5%

10%

15%

20%

25%

30%

35%

More than 10%EARLY

ON-SCHEDULE 20% 40% >50% BEHINDSCHEDULE

S t u d y P a r t i c i p a n t s

Non-FPGA vs. FPGA completion compared to project's original schedule

Non-FPGA

FPGA

Non-FPGA: 67% behind schedule FPGA: 67% behind schedule

35




Required Number of Spins

HF, MemoCODE, 2013


0%

10%

20%

30%

40%

50%

1FIRST

SILICONSUCCESS

2 3 4 5 6 7SPINS or

MORE


Number of Required Spins

2004

2007

2010

2012

36




Types of Flaws

HF, MemoCODE, 2013


0%

10%

20%

30%

40%

50%

60%


Trends in Types of Flaws Resulting in Respins

2004

2007

2010

2012

* Multiple answers possible

37





www.mentor.com© 2013 Mentor Graphics Corp. Company Confidential© 2010 Mentor Graphics Corp. Company Confidential

www.mentor.comHF, MemoCODE, 201339

Cost of Find Functional Flaws

Silicon Debug, Doug Josephson and Bob Gottlieb, (Paul Ryan)D. Gizopoulos (ed.), Advances in Electronic Testing: Challenges and Methodologies, Springer, 2006

Relative Cost Of Finding Bugs

$1

$10$100

$1,000

$10,000

$100,000

$1,000,000

$10,000,000

I n i t i a l D

e s i g

n

D e s i g

n R e v i e

w

L a y o u

t

T a p e

R e l e

a s e

E a r

l y S i l i c

o n

S a m

p l i n g

V o l u m

e P r o d u c t i o

n

Design Cycle

C

ostTo

Fix




BEYOND STANDARDS

Beyond arguing over who won the standards war





Standardization of Languages

0%

20%

40%

60%

80%

VHDL Verilog Synopsys Vera System C SystemVerilog Specman e C/C++ OTHER Testbench


Languages Used for Verification (testbenches)

2007

2010

2012

SystemVerilog grew 8.3% between 2010 and 2012






SystemVerilog Adoption by Design Size

58.8%

71%

89%

0%

20%

40%

60%

80%

100%

< 5M 5 - 20M > 20M

N o n - F P G A S t u d

y P a r t i c i p a n t s

SystemVerilog Adoption by Design Size(Gate Count Excluding Memories)






Standardization in Base Class Libraries

0%

10%

20%

30%

40%

AccelleraUVM

OVM Mentor AVM

Synopsys VMM

SynopsysRVM

CadenceeRM

CadenceURM

Other



Testbench Methodologies and Base-Class Libraries

2010

2012

486% UVM growth between 2010 and 201246% UVM projected growth in the next twelve months






Standardization of the SoC Verification Process

Ten years ago, IC/ASIC verification was partitioned intotwo main steps:


Integration Verification

Block-Level Verification

Block Full Chip





Emerging from ad hoc to systematic processes


Block -Level Verification

Interconnect Verification


Application/ SW

Verification

IP Subsystem SoC System




BEYOND THE STATUS QUO

Beyond surviving by maintaining the status quo





The Verification Paradox

A good verification process lets you get the most out ofbest-in-class verification tools

Start Tools Ad Hoc

Processes

6-9%

Cost Increase

Start Process Tools20-30%

Cost Savings

Source: Cisco Momentum Research Group










Application/ SW

Verification




Use of Advanced Verification Techniques


62%

71%

68%

70%

41%

40%

37%

48%

0% 20% 40% 60% 80%

Constrained-Random Simulation

Functional coverage

Assertions

Code coverage

Non-FPGA Study Participants

2007

2012





Directed vs Constrained-Random Simulation



44%51%

56%49%

0%

20%

40%

60%

80%

100%

2010 2012

N o n - F P G A S t u d y P


Mean Directed vs. Constrained-Random Simulation Trends

Directed

Constrined-Random




Larger Designs Use More Formal

20%

26%

41%

0%

10%

20%

30%

40%

< 5M 5 - 20M > 20M


Formal Property Checking Adoption by Design Size(Gate Count Excluding Memories )






The Evolution of Formal Technology

FormalPropertyChecking

Fully Automatic

Formal

Automated Applications

LowEffort

High

Effort

FormalExperts

Everyone

1990s Today2000s







A57 A57 A57 A57




Application/ SW

Verification









Application/ SW

Verification

— IP Blocks connectivity — Access all memories

— Access all registers, such as control — Configurations work — Functional scenarios and use-cases — Verify multiple clock domain crossings




Mean Number of Clock Domains by Design Size

5.21

9.36

11.48

0

2

4

6

8

10

12

14

< 5M 5 - 20M > 20M

M e a n N u m b e r

o f C l o c k D o m a i n s

Mean Number of Clock Domains by Design Size(Gates Excluding Memories)











Application/ SW

Verification

— Boot OS — Load System Drivers

— Run Application SW




SoC Design & Verification Involves Lots of SW


$0

$20

$40

$60

$80

$100

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012

Total SW Engineering Costs + ESA Tool Costs

Total HW Engineering Costs + EDA Tool CostsSource: ITRS 2010, Impact of Design Technology on SoC Consumer Portable Implementation Cost

( $

M )

It’s the software, stupid! -Gary Smith

As Design Sizes Increase…Emulation Up,




As Design Sizes Increase…Emulation Up,FPGA Prototyping Down in 2012

18%

32%

50%

45%

57%

28%

0%

10%

20%

30%

40%

50%

60%

HW Acceleration/Emulation FPGA Prototyping


Adoption by design size for those doingHW acceleration/emulation and FPGA Prototyping

< 5M

5 - 20M

> 20M



Integrated Simulation/Emulation/Software




Integrated Simulation/Emulation/Software Verification Environments Emerge

HF, MemoCODE, 2013

Virtual Prototype

Processor Debug

JTAG

SW Debug

OVM/UVMSystemC/C++

Monitors

Testbench Acceleration

Assertions &Checkers

Physical Devices

Protocol Solutions

System Level

Virtual Devices &Transactors

Protocol Solutions

Design under Test

59




Coverage and Power





Application/ SW

Verification

Block Subsystem SoC System

Across all aspects of verification




Signoff criteria trends

The Rising Importance of Coverage

HF, MemoCODE, 2013

Wilson Research Group and Mentor Graphics, 2012 Functional Verification Study, Used with permission * Multiple answers possible

0% 10% 20% 30% 40% 50%

OTHER

WHEN THE PROJECT PLAN SAYS SIGN-OFF,REGARDLESS OF STATUS

WHEN WE CAN NO-LONGER THINK OF ANY MORETESTS TO WRITE

WHEN THE RATE OF BUGS FOUND PER WEEK DROPSBELOW A SPECIFIED GOAL

WHEN THE EMULATED OR PROTOTYPED DESIGN ISWORKING IN-SITU

WHEN COVERAGE SAYS WE HAVE ACHIEVED ATARGET

WHEN THE PROJECT PLAN SAYS SIGN-OFF, ASSUMING VERIFICATION LOOKS OK

WHEN ALL TESTS DOCUMENTED IN THE VERIFICATION PLAN ARE COMPLETE AND PASS

Non-FPGA Study Participants

20072012

61




Unified Coverage Interoperability Standard


New Accellera UCIS Standard Announced at DAC 2012

UCIS API

CoverageDatabase

Simulation

Formal

Emulation

Analysis Testplan




Trends in Power Management Verification

0% 20% 40% 60% 80%

Power domain powerdown/power up

Power domain statereset/restoration

Transitions between systempower states

Hardware power controlsequence generation

Interactions between power domains

Operation in each system power state

Application-level power management

Hypervisor/OS control of power

management

Aspects of Non-FPGA Power Managed Design That Are Verified

67% of the industry actively

manages power

Source: Wilson Research Group and Mentor Graphics, 2012 Functional Verification Study, Used with permission





Power Trends

HF - January 2013 Master Set, WRG & MG Study Results


0%

5%

10%

15%

20%

25%

30%

35%



Percentage of total simulations that were power-aware

About 10% of power managed designsperform no power-aware simulation!




Power Trends

HF - January 2013 Master Set, WRG & MG Study Results


0%

5%

10%

15%

20%

25%

30%

35%


Percentate of Verification Resources Focused on PowerManagement

Median Verification Resources: 20% -29%




Power Trends


6% 6%

24%

15%

27%

22%

0%

5%

10%

15%

20%

25%

N o n - F P G A S t u d y P a

r t i c i p a n t s

* Multiple answers possible

Notation used to describe power intent

Source: Wilson Research Group and Mentor Graphics, 2012 Functional Verification Study, Used with permission




Beyond the Status Quo





Application/ SW

Verification

IP Subsystem SoC System





BACK TO THE FUTURE

The Productivity Gap





Design Productivity Gap

SiliconDensity

DesignProductivity

C a p a c i t y

Time

Source: SEMATECH





VerificationProductivity

Verification Productivity Gap

Source: SEMATECH

SiliconDensity

DesignProductivity

C a p

a c i t y

Time





Closing The Verification Gap


Reuse Abstraction

Acceleration Methodology

?



www.mentor.com© 2013 Mentor Graphics Corp. Company Confidential72

100s of lines

of TLM

Billions of Transistors

100s of Millions of Gates

Millions ofLines of RTL

#include "systemc.h"SC_MODULE(adder) // module

(class) declaration {sc_in<int> a, b; // portssc_out<int> sum; voiddo_add() // process { sum =a + b; } SC_CTOR(adder) //constructor {SC_METHOD(do_add); //register do_add to kernelsensitive << a << b; //sensitivity list of do_add } };

Managing Complexity




www.mentor.com© 2013 Mentor Graphics Corp. Company Confidential73

RTL

clk

Cycle Accurateclk

Timed TLM

Untimed TLM

C/C++

Simulation

1x (7 days)

10,000x (1 min)

10x

100x

1,000x

Protocol

Protocol

Transaction

Functionarguments

Transaction

Productivity Gains Through Abstraction





What’s the advantage of SystemC compared with RTL?

There are two different aspects – quality and time schedule.

Today, a full chip on SystemC will run around 10 MHz, and

you will never reach that speed using RTL or a lower-levelabstraction. It’s similar to a prototype speedup. Previously

with RTL designs, our bug rate was in the range of 10 to 50

bugs per square millimeter. Now we are at less than one bug

per millimeter squared. So we have both quality and speed

of development.

Source: EETimes, 2007, Laurent Ducousso, who manages intellectual-property (IP) verification forSTMicroelectronics’ Home Entertainment Division

Bug Prevention vs. Bug Hunting





Summary

Beyond theory in terms of rising complexity Beyond arguing over who won the standards wars

Beyond surviving by maintaining the status quo




Documents

Keynote 3 Foster TrendsFunctionVerification