Max Metric Driven Ver Wp

7/29/2019 Max Metric Driven Ver Wp

http://slidepdf.com/reader/full/max-metric-driven-ver-wp 1/10

Introduction

MDV is a key to Silicon Realization, part o the greater EDA360 vision. EDA360

helps both design creators and design integrators close the “productivity gap”

(through improved approaches to design, verication, and implementation) as

well as the “protability gap” (by providing new capabilities or IP creation/

selection/integration and system optimization). Silicon Realization represents

everything it takes to get a design into working silicon, including the creation

and integration o large digital, analog, and mixed-signal IP blocks. Advanced

verication capabilities, such as MDV, are a must.The unctional verication landscape has changed beyond all recognition

over the last 10 years, and while design paradigms have become mature and

stable, verication methodologies and technologies have continued to evolve

and new fows and tools are still being invented. Against this rapidly changing

background, designs have been steadily growing with more IP and more

complex IP being integrated into larger and larger SoCs.

Realizing silicon in the ace o these challenges requires new approaches and a

very fexible workorce capable o adapting and changing on a regular basis. This

paper outlines a new approach to managing verication complexity—metric-

driven verication—and a valuable resource to back up this new approach, the

Cadence Incisive Verication Kit, which together will enable you to plan and

embrace new methodologies and approaches in a sae and controlled way. The

kit provides clear and realistic examples at each stage to guide you and your

teams in their new projects.

Silicon Realization is achieved when metrics can be applied to design intent,

when users can work more productively because they are working at a higher

level o abstraction, and when successive renements converge to achieve

verication closure. MDV manages intent, abstraction, and convergence—

leading to greater predictability, productivity, and eventually protability.

Maximizing Verication Eectiveness Using

Metric-Driven VericationAuthored by Nick Heaton – Senior Solution Architect, Cadence Design Systems, Inc.

This paper introduces the Cadence® Incisive® Verication Kit as a golden example o how to

maximize verication eectiveness by applying metric-driven verication (MDV) in conjunction

with the Universal Verication Methodology (UVM). MDV provides an overarching approach

to the verication problem by transorming an open-ended, open-loop verication process

into a manageable, repeatable, deterministic, and scaleable closed-loop process. Through this

transormation, verication project teams and managers greatly increase their chances o consistently

delivering working designs at improved quality levels in less time with ewer human resources.

ContentsIntroduction .................................1

Failing to Plan = Planning to Fail ...2

Metric-Driven Verication ............3

Building Strong Testbench

Foundations .................................4

Simulation isn’t the Only Way ......5

Low Power isn’t Just the Designer’s

Problem .......................................5

Reuse isn’t Just About Testbench

Components ................................6

Does Speed Matter? ....................7

What About Scalability? ...............8

Is Metric-Driven Verication Just or

RTL Hardware? ............................8

How Do I Get Up to Speed with All

this New Stu? ............................9

Summary ................................... 10

About The Author ...................... 10



Failing to Plan = Planning to Fail

Any management process needs clear and measurable goals, and verication is no exception. Failing to capture

these goals at the outset o a project means that there is no clear denition against which to measure either

progress or closure. You can only gauge improvement in what you can clearly measure. Historically this was a

problem with directed testing. Huge lists would be drawn up to dene the verication process, but these lists were

not executable or maintainable. This open-ended nature led to big project slips and huge stresses on project teams.

In addition, there is oten conusion about the denition o what constitutes a verication plan and what constitutes

a test plan. Let’s make our denition clear right away: a test plan denes a ully elaborated list o tests that will be

created and the completion o this list denes completion o verication. A test plan alone however tends to be a

poor mechanism since it already contains decisions about the execution o the process. In contrast, a verication

plan captures the “what” that needs to be veried but does not dene the execution—the “how.” This is a crucially

important distinction as it is expected that verication may be done using multiple technologies by multiple people.

We reer to the “what” as the “eatures” or “goals” that need to be veried—it is the design intention. When

planning a project, you should not presume the underlying tool by which a eature is veried. It will most likely come

rom several sources, and the results are represented by various orms o coverage.

A verication plan should capture the goals o the verication process such that the sign-o criteria are clearly

identied as well as milestones along the way. An example o a milestone would be RTL code reeze, which a team

might dene as having 75% o all eatures covered by that time.

In addition to capturing goals upront, a verication plan should be executable. In other words, it is essentialthat progress toward project closure can be easily and automatically measured. Typically, the goals get rened

as a project progresses, which means the verication plan is a living document that matures over the lietime o

a project.

Figure 1 shows a sample o a verication plan or the UART block in the Incisive Verication Kit environment.

As the sample shows, the verication plan can be a regular Microsot Word document that uses template heading

styles to identiy structure and specic eatures. This is easy to create in Microsot Word or using Incisive Enterprise

Planner as the ront-end editor o the plan.

The verication plan also identies abstract eatures and hierarchies o eatures that may closely resemble

the hierarchy o the specication. While not mandatory, a verication plan is o ten a useul convenience that

helps designers and verication engineers communicate. Verication plans can also include other (sometimes

Figure 1: Verifcation plan or the UART block

www.cadence.com 2

Maximizing Verication Eectiveness Using Metric-Driven Verication



commercially supplied) verication plans—as in the case o the Incisive Verication Kit—the UART has an AMBA ®

APB interace and, thereore, the plan includes an APB verication plan. This mechanism enables reuse o plans that

relate to standard interaces or perhaps blocks o IP that are reused across multiple projects.

Figure 2 shows an executable verication plan (vPlan) with coverage mapped onto it. This approach allows resource

decisions to be made in an inormed way such that progress toward closure proceeds according to the original plan.

The coverage is accumulated rom any number o runs o a verication fow and rom a variety o verication

engines (ormal, simulation, hybrid, or emulation).

Metric-Driven Veriication

Over the past ew years, the term "coverage-driven verication" has become widely adopted to reer to the

gathering o simulation coverage in all o its various orms, traditionally code and unctional coverage. This

approach has many limitations that aect its usability and scalability. Once major limitation is that it does notinclude checking or time-based aspects, which are essential in dening the closure criteria.

Metric-driven verication (MDV) broadens the scope o what is captured and measured to include checks,

assertions, sotware and time-based data points that are encompassed in the term “metrics.“

The second enhancement MDV oers over coverage-driven verication is the ability to create eature hierarchies

by using an executable verication plan (vPlan). This helps manage the wealth o data captured by all the tools

involved in the execution o a verication project.

Figure 2: Executable vPlan with coverage

Figure 3: vPlan with metrics

www.cadence.com 3




The third enhancement is support or parallel verication execution to optimize verication throughput. One o the

primary objectives o MDV is to reduce the dependency on the one resource that is not scalable—the verication

engineer. By enabling use o parallel processing via machines through automation, we improve productivity and

reduce the number o non-productive tests. Figure 3 contains a vPlan that comprises metrics rom a number o

dierent environments and dierent verication engines.

We can see how metrics are gathered at the UART level rom multiple runs o both ormal (static) and simulation

(dynamic) verication.

Building Strong Testbench Foundations

All o the high-level capabilities o MDV that deliver the abstracted and ltered view o the verication process

would amount to nothing without a common ramework under which verication environments could be

constructed and reused. The Universal Verication Methodology (UVM) has become the de-acto standard or the

construction o verication environments. The UVM is based on the Open Verication Methodology (OVM) 2.1.1

release (the OVM itsel an evolution rom the e Reuse Methodology (eRM), which has been in widespread use

since 2002 on thousands o projects). The UVM supports all o the important languages customers demand and

enjoys industry-wide support.

Given the investment needed in training and development o advanced verication environments, it is reassuring to

know that by adopting the UVM you are part o a global community working toward common verication goals.

MDV builds on top o the UVM, enables the management o complex SoC developments, and ocuses on the keychallenges rather than issues o verication environment implementation or reuse.

The Incisive Verication Kit completely adopts the UVM and includes real-world testbench examples in both e and

SystemVerilog. At the block level, the kit shows how MDV can be applied to the verication o a UART design. It

shows how a verication plan can be constructed upront, how a UVM environment can be rapidly constructed

using Universal Verication Components (UVCs), and how simulations can be run and coverage annotated against

the verication plan to monitor and manage the verication process.

Figure 4 details the architecture o one o the kit

testbenches, showing the hook-up o the UVCs to the

UART DUT.

One o the major strengths o the UVM is its approach

to reuse. Improving productivity is all about writing

complex testbench elements—once and only once—and thereater reusing these components as widely as

possible. One o the dimensions o reuse engineered into

the UVM is rom block to cluster.

Figure 5 shows the cluster-level testbench or the kit APB

subsystem, clearly illustrating the large amount o reused

content. Notice the reused serial interace UVCs and the

reused APB UVCs.

UVC for APB Interface UVC for Serial Interface

Coverage

BFM

Seq

SeqCoverage

BFM

APB Serial Tx/Rx

UART

Seq

Seq

Monitor Sequencer Monitor Sequencer

Figure 4: Details o a kit testbench architecture

UVC for Serial Interface

Coverage

BFM

UART #1 UART #2 SDIO AHB

APB

APB-Subsystem

Seq

Seq

Monitor Sequencer

UVC for Serial Interface

Coverage

BFM

Seq

Seq

Monitor Sequencer

UVC for SDIO Interface

Coverage

BFM

Seq

Seq

Monitor Sequencer

UVC for AHB Interface

Coverage

BFM

Seq

Seq

Monitor Sequencer

UVC for APB Interface

Coverage

BFM

Seq

Seq

Monitor Sequencer

Figure 5: Cluster-level testbench or the kit APB subsystem

www.cadence.com 4




One o the key purposes o the kit is to accelerate understanding o UVM concepts, and testbench architecture

diagrams like this are available as clickable demos in the kit. These diagrams enable you to quickly amiliarize

yoursel with both the methodology and language usage in the context o a realistic example.

This modular, layered approach also orms the basis or reuse. In addition to the creation o plug-and-play

hardware and sotware verication components that can be reused rom block to cluster to chip to system, UVM

components can also be reused across multiple projects and platorms.

As there is a requent need or verication components or standard interaces such as USB, PCI Express, AMBA

AXI, and so on, Cadence has created a comprehensive portolio o commercial verication IP (VIP) components.

These components are o particular interest when it comes to "compliance testing" (ensuring that the design's

interaces ully comply with their associated specications). Furthermore, each o these VIP components comes

equipped with its own pre-dened verication plan that can be quickly and easily incorporated into a master plan.

(A subset o this VIP port olio is demonstrated within the Incisive Verication Kit).

Simulation isn’t the Only Way

MDV uses a generic approach that doesn’t mandate

any specic verication execution engine; in other

words simulation isn’t the only tool you may choose

to use or establishing design correctness. Figure 6

illustrates how MDV changes an open-loop vericationprocess into a closed-loop process through a Plan-

Construct-Execute-Measure cycle using the results o

multiple verication engines.

For example, ormal tools are another means by

which design correctness can be established. Formal

properties in the orm o assertions are captured

and the tools attempt to exhaustively prove that, or

all possible circuit states, the property is true. While

the user typically writes these assertions manually,

automatically extracted assertions and assertion

libraries can also accelerate the verication process.

The Incisive Verication Kit provides examples o

these fows and shows how the checks and assertioncoverage points are easily included in the verication

plan. The results rom the ormal process are then

annotated onto the plan. MDV seamlessly provides verication engineers and managers with a consistent view o

the verication progress, regardless o where the results are coming rom. While exhaustively proving assertions is

a goal, there are requently occasions where the tools need additional guidance to complete their proo. Typically,

the user writes additional assertions that dene constraints. These narrow the scope o the ormal proo, and the

tools are then able to complete the proo. The question is: how do you know that these assumptions are valid?

Assertions are not only used or ormal proo, but will also execute in simulation. By reusing the constraints in your

simulation runs, you have a means o cross-checking whether the assumptions you made still hold true in your

simulation models. This isn’t a watertight mechanism, as users may not exhaustively simulate the environment in

all possible scenarios. However, i you adopt constrained-random simulation similar to the UVM, you increase your

chances o hitting the corner-case where assumptions may have been incorrect.

MDV provides a mechanism or users to include the coverage o the constraints, which, at a minimum, will identiy

that a constrained condition was triggered. You always have visibility into ailures, so the coverage you get gives

you a useul cross-check o these constraints. These techniques rom both the planning and execution side are

included as workshops within the kit.

Low Power isn’t Just the Designer’s Problem

Low-power design is no longer a “nice to have,” but is mandatory even or non-mobile applications, and it is a

crucial “must have” or all “green” products. But as a verication engineer, why should you care about low power?

Simulation

AMS

Formal

System

Plan

Execute

Measure Construct

Figure 6: MDV changes an open-loop verifcation process intoa closed-loop process

www.cadence.com 5




In the past, the verication engineer would veriy that the RTL implementation precisely matched the design

specication, and while that still holds true, it is only part o the problem. With the addition o new low-

power techniques that capture power intent in side les, the “golden” design is no longer just the RTL, but is a

combination o the RTL plus the power intent. While the power intent denes the physical implementation details,

there is also a power control module (PCM) that is—almost without exception—under sotware control.

The low-power verication challenge thereore has two new dimensions. Firstly, the physical implementation

changes can introduce unctional bugs as a side eect. These may only maniest themselves during corner-case scenarios. Secondly, the interaction between the power management sotware and the hardware requires

careul verication to ensure that all power modes and power-mode transitions have been veried in all operating

circumstances. Even a design with just a ew pieces o IP that can be powered down can have thousands o

operational states and power transitions that need exercising.

The Incisive Verication Kit provides comprehensive material and examples covering verication o a hardware

platorm containing a realistic set o power management eatures. It also contains workshops on how to use an

MDV approach to hardware/sotware co-verication.

Figure 7 shows a simplied architecture diagram o the kit platorm including a power control module (PCM) .

Reuse isn’t Just About Testbench Components

Universal Verication Components (UVCs) are reusable verication components that perorm a vital role in

accelerating testbench development through reuse. These components may comprise complex developments,

and commercial availability o UVCs or standard protocols provides substantial leverage or assembling complex

testbenches. In all o this detail, engineers sometimes overlook the act that many other pieces o the verication

process may be reused in multiple dimensions.

UVCs provide project-to-project reuse at multiple levels. They also provide block-to-cluster reuse, whichsubstantially boosts productivity when assembling a testbench or a hardware platorm or which sub-component

testbenches exist. Not only can the components be reused, but also sequence libraries, register denitions, and

verication plans. The Incisive Verication Kit has comprehensive cluster-level environments that ully illustrate how

to apply these reuse techniques.

Figure 8 shows the architectural overview o a chip-level testbench and it s corresponding chip-level vPlan, showing

all o its verication plan reuse.

APB

Bridge

OR1K SMC

UART #1 UART #2 SDIO GPIO

Power Shut-off Control

0.6V – 1.2V

HCLK

ALUT

ENET

BridgeAPB-Subsystem #1

Chip-Level Platform

APB

VCO

AHB

PCMAPB-Subsystem #2

Figure 7: Architecture diagram o the kit platorm including a power control module

www.cadence.com 6




Does Speed Matter?

As a verication engineer, which o the ollowing does your manager care more about: how ast your simulator runs

or whether you achieve coverage closure on schedule? MDV enables sophisticated analysis o the coverage rom

the complete regression runs and allows you to correlate the results or a particular eature against the execution o

the verication. In other words, MDV helps you answer the question, “Which simulations should I re-run in order to

quickly test eature Y in 2 hours?”

Eective verication is not about running the most simulations in the least time. It is about running the most eectivesimulations as much o the time as possible. Simulations that add to coverage are the most valuable, as they are more

likely to hit corner-case scenarios and thereore nd bugs. MDV enables you to identiy the random seeds and tests

that contribute the most overall coverage or coverage or a particular eature or group o eatures. Running the whole

regression suite more and more is like using a sledgehammer—eventually you will hit the target but it will take a lot o

eort. Conversely, MDV is like putting a laser in the hands o the verication engineer; it can be ocused and red into

specic areas where coverage holes exist. Figure 9 shows the analysis o coverage contribution or each run o a

regression against a specic vPlan eature.

APB

Bridge

OR1K SMC

UART #1 UART #2 SDIO GPIO

Power Shut-off Control

0.6V – 1.2V

Automatic vPlan creation

HCLK

ALUT

ENET

Bridge

APB

VCO

AHB

PCM

APB-Subsystem #2

Chip vPlan

vPlan

APB SubsystemvPlan

vPlan

APB SubsystemvPlan

vPlan

EthernetvPlan

CPF File

vPlan

Low PowervPlan

vPlan

Chip-Level Platform

UARTvPlan

vPlan

UARTvPlan

vPlan

APBvPlan

vPlan

APBvPlan

vPlan

APB-Subsystem #1

Figure 8: Architectural overview o a chip -level testbench and its corresponding chip-level vPlan

Figure 9: Analysis o coverage contribution or each run o a regression against a specifc vPlan eature

www.cadence.com 7




The Incisive Verication Kit provides workshops on how to use the analysis eatures o MDV so you can quickly get

up to speed with this powerul capability.

What About Scalability?

One o the key lessons learned in the development o the UVM is the power o automatic test generation. By having

constrained-random tests, each time a test scenario is run with a new seed, it is potentially going to nd new bugs as it

may traverse new states in the design. This scalability has massive potential when allied with inrastructure that supportsvast processing arms. Huge numbers o concurrent simulations can be launched by one person, all potentially hunting

bugs without the need or engineers to write hundreds and hundreds o directed test cases. This ability to scale the

verication task is very compelling as it truly starts to automate the progress toward verication closure.

Figure 10 shows an example o how MDV was applied to a real customer project. Using MDV, the customer not only

reduced the project timerame rom 12 months to 5 months but also ound more bugs and used less manpower.

For ormal verication a large set o ormal properties traditionally comprise the verication task, and as the design

grows and the suite o properties grow, so does the length o ormal run time needed to complete the proo. New

eatures within Incisive Formal Verier have enabled a scaled “regression” approach to complete these large ormal

proos through the use o compute arms. Incisive Formal Verier has the ability to split a large proo “deck” into a

number o distributed smaller proos, and thereby run the entire regression in a much shorter time. As these types

o tasks tend to be run repeatedly as chip sign o approaches, the option o reducing run-time by a actor o 10 or

20 can make the dierence between hitting a schedule or not.

Is Metric-Driven Veriication Just or RTL Hardware?

Metric-driven verication is a structured and comprehensive approach to dening verication goals and measuring

and analyzing progress toward those goals. It thereore is not conned explicitly to veriying hardware, but

represents a by-product o the escalating cost o ailure when developing and implementing complex silicon

devices. And so it’s perectly reasonable to ask, “What systematic and comprehensive approaches do embedded

sotware developers use to ensure adequate quality levels?” and “Could MDV be applied to embedded sotware?”

The answer to these questions is yes. MDV techniques can easily be applied to embedded sotware, and the

Incisive Verication Kit contains workshops and example fows o how to implement such technology. The

techniques demonstrated show how to veriy the sotware interace to the hardware, in a hardware-sotware

co-verication fow.

Figure 10: Example o MDV applied to a real customer project

1 2 3 4 5 6 7 8 9 10 11 12

Man Months

Improvement in Verification Efficiency

Traditional HDL TB

Tape-out

MDV EnvFirst Project

MDV EnvSecond Project

MDV EnvThird Project

Resource Limited

VerificationEnvironment

TestCreation

RunTests


TestCreation

Run RegressionTests


TestCreation

Run (Massive)Regression Tests

License LimitedTCVE Run (Massive)Regression Tests

www.cadence.com 8




One o the key points to keep in mind is the dierence between validation and verication. Systems are typically

designed and validated top-down, validation being the process to ensure that you are developing the right

product. This is very much a sotware-oriented challenge, but still requires some orm o hardware model—

usually a unctional virtual prototype built rom transaction-level models (TLMs). Verication is usually perormed

as a bottom-up process that exhaustively proves the product has been developed right. Verication is normally

perormed to diering degrees on multiple abstraction levels. In other words, it is exhaustive verication at the

block level, but perhaps integration and the programmer’s view might be veried at the cluster or system levels.

Figure 11 shows the continuum that exists between hardware/sotware, verication, and validation, and the

diering levels at which MDV might be applied.

As embedded sotware content and complexity grows, there is more demand that deeply embedded products

should work reliably while always powered on over longer and longer periods. The impact o sotware quality on

product reliability will continue to rise in the same way that hardware quality has become mandatory or any new

SoC design.

Deeply embedded sotware quality is a major concern or complex SoC products, especially as multi-core designs

become more prevalent. Thus, the application o a rigorous, scalable quality measure such as MDV will become

the norm.

How Do I Get Up to Speed with All this New Stu?

So this new methodology and technology is great, you might say. But how on earth do you get my entire team up

to speed in a realistic timerame, such that it doesn’t impact the schedule?

The Incisive Verication Kit contains a large number o comprehensive hands-on workshops, including all the

slides and lab instructions, so you can walk through a specic topic in your own time or perhaps ask a Cadence

Application Engineer to run through it with you. The act that these workshops are pre-packaged and delivered

with the tools greatly enhances your ability to manage the ramp-up o teams on selected methodologies, all based

on a realistic example SoC.

The current range o workshops delivered includes:

Assertion-based verication • Introduction

• Incisive Formal Verier productivity fows

• Incisive Formal Verier connectivity fows

UVM – SystemVerilog • Module-based SystemVerilog

• Class-based SystemVerilog

• Introduction to register modeling with reg_mem

MDV Techniques

Algorithmic

TransactionBased

Interfaces

Integration

S o f t w a r e

H a r d w a r e

ProtocolCompliance

Did I design the right product?

Nomally a top down process

Mostly software oriented

using TLM’s to represent HW

Did I design the product right?Nomally a bottom up process

Mostly hardware oriented

using RTL to represent HW

VALIDATION

VERIFICATION

Figure 11: The continuum that exists between hardware/sotware, verifcation, and validation

www.cadence.com 9




UVM – Specman® / e • Introduction to Specman

• Introduction to e

• IntelliGen debugging

UVM – mixed language (SystemVerilog and e) • SystemVerilog over e using OIG

• SystemVerilog with SystemC® TLMs

Metric-driven verication oundations • Introduction

–Metric Assertion

• Planning

• Inrastructure

• Management

• Automation

Metric-driven verication using verication IP • Using multiple UVCs

• Using the Compliance Management System

• HW/SW co-verication

Summary

This paper introduced the concept o metric-driven verication (MDV) and explained how it is a powerul layer o

methodology that sits above the Universal Verication Methodology (UVM). The Incisive Verication Kit provides

a comprehensive set o examples o how MDV can be applied across an entire range o verication technologies

in a coherent and consistent way. Once applied, MDV puts powerul capabilities in the hands o engineers and

managers. These MDV capabilities make the crucial dierence; they improve verication eectiveness and, hence,

managers and engineers alike can maximize their utilization o resources, use the breadth and depth o technology

available, improve project visibility, and reduce the number o engineering man months.

About The Author

Nick Heaton is an ASIC and EDA veteran with more than 25 years o experience in the design and verication o

complex SoCs. Nick graduated rom Brunel University, London in 1983 with First Class Honors in Engineering and

Management Systems, initially working as an ASIC designer or ICL in Bracknell. In 1993, he ounded specialist

ASIC Design and Verication Company Excel Consultants, servicing customers such as ARM® and Altera. In 2002,

Nick joined Verisity as Manager o Northern European Consulting Engineering.

Nick currently works in the Cadence Research & Development organization as a Senior Solution Architect with

special responsibility or the Incisive Verication Kit, a complex and realistic golden example o verication

methodologies and technologies across all o Cadence ront-end products..

Cadence is transorming the global electronics industry through a vision called EDA360.

With an application-driven approach to design, our sotware, hardware, IP, and services help

customers realize silicon, SoCs, and complete systems eciently and protably. www.cadence.com

© 2011 Cadence Design Systems, Inc. All rights reserved. Cadence, the Cadence logo, Incisive, and Specman are registered trademarks o Cadence

Design Systems, Inc. ARM and AMBA are registered trademarks o ARM, Ltd. SystemC is a registered trademark o the Open SystemC Initiative, Inc.

in the U.S. and other countries and is used with permission. All others are properties o their respective holders. 22209 05/11 MK/DM /PDF


Documents

Max Metric Driven Ver Wp