Using Agile to Improve the FPGA Development …...2014 2012: A v er ag e 43% 2014: A v er ag e 46% 4.0 4.4 2.6 3.8 0 2 4 6 8 10 2012 2014 M e a n P e a k N u m b e r o f F P G A E

Using Agile to Improve the FPGA Development Process

Nigel Elliot Digital Design & Verification

AE Group Leader

November 2015

Mentor Graphics Predictability through Agility

www.mentor.com Mentor Graphics Corp. Company Confidential

Agenda

Trends in Industry

Scheduling Challenges

Waterfall vs Agile

Applying Iterative Development to FPGAs

Agile for FPGA 2


Majority of FPGA Projects Miss Schedule

Agile for FPGA 3

Many FPGA Projects Miss Schedule

Behind Schedule Ahead of schedule Source: Wilson Research Group and Mentor Graphics, 2014 Functional Verification Study

H Foster, WRG Functional Verification Study, November 2014 12 © 2014 Mentor Graphics Corporation, all rights reserved.

0%

5%

10%

15%

20%

25%

30%

35%

More than 10% EARLY

10% EARLY ON-SCHEDULE 10% BEHIND SCHEDULE

20% 30% 40% 50% >50% BEHIND SCHEDULE

Des

ign

Pro

jec

ts

Actual FPGA design completion compared to FPGA project's original schedule

2012

2014

2012: 67% behind schedule

2014: 59% behind schedule


More Effort Spent on Verification

Agile for FPGA 4

FPGA Verification Consumes Majority of Project Time

Source: Wilson Research Group and Mentor Graphics, 2014 Functional Verification Study


0%

5%

10%

15%

20%

25%

1%-20% 21%-30% 31%-40% 41%-50% 51%-60% 61%-70% 71%-80% >80%

Des

ign

Pro

jects

Percentage of FPGA Project Time Spent in Verification

2012

2014

2012: Average 43%

2014: Average 46%

4.0 4.4

2.6

3.8

0

2

4

6

8

10

2012 2014

Mean

Pea

k N

um

ber

of

FP

GA

E

ng

inee

rs

Verification Engineers

Design Engineers

Number of Peak FPGA Engineers Increasing



4.9% CAGR for FPGA design engineers

20.9% CAGR for FPGA verification engineers

Where FPGA Designers Spend Their Time

© 2014 Mentor Graphics Corporation, all rights reserved.


H Foster, WRG Functional Verification Study, November 2014 10

49%

51%

0%

20%

40%

60%

80%

100%

2014

Me

an

tim

e F

PG

A d

esig

n e

ng

ineer

sp

en

ds in

desig

n v

s. v

eri

fic

ati

on

Doing Verification

Doing Design


FPGA Verification Trends

Agile for FPGA 5

Majority using functional and code coverage

Increasing amount leveraging power of constrained random verification

We will look at how using an iterative approach can provide predictability for these efforts

FPGA Verification Technique Trends



0% 10% 20% 30% 40% 50% 60% 70%

Constrained-Random Simulation

Functional coverage

Assertions

Code coverage

FPGA Design Projects

2012

2014


Sources of Scheduling Uncertainty

Agile for FPGA 6

Where FPGA Verification Engineers Spend Their Time


© 2014 Mentor Graphics Corporation, all rights reserved. H Foster, WRG Functional Verification Study, November 2014 11

13% 20%

19%

43%

4%

Test Planning

Testbench Development

Creating Test and Running Simulation

Debug

Other

Root Cause of FPGA Functional Flaws



* Multiple answers possible

0%

10%

20%

30%

40%

50%

60%

70%

80%

DESIGN ERROR CHANGES IN

SPECIFICATION

INCORRECT or

INCOMPLETE

SPECIFICATION

FLAW IN

INTERNAL

REUSED BLOCK,

CELL, MEGACELL

or IP

FLAW IN

EXTERNAL IP

BLOCK or

TESTBENCH

OTHER

De

sig

n P

roje

cts

Root Cause of FPGA Functional Flaws

2012

2014

How can you accurately schedule

for debug?

Even running simulations can be

difficult to schedule

Significant time spent debugging

Changes or problems with specification


Scheduling Predictability Why is it difficult to create accurate schedules?

Agile for FPGA 7

Debug

Bug fixes

Verification closure

Hardware bring-up

Hardware validation

Low Defining requirements

Synthesis

Timing closure

Implementation

Medium Write spec

Write RTL

Write TB

Write constraints

High

Complete When Done Complete When Fully Tested


Agile Overview

Waterfall — Sequential – going back to previous steps

can be difficult and costly — One big bang, production ready release at

the end of the project — Relies heavily on initial requirements — Most testing occurs near the end of the

project — Lessons learned only for the next project — Task driven development

Agile — Adaptive and iterative process — Many production ready releases during

project based on most important features — Allows for changes throughout process — Testing happens often and early — Many opportunities for feedback — Value driven development with every

iteration

Agile for FPGA 8

Specification

Design

Verification

Implementation

Hardware


Adaptive Process (Agile)

The vision drives feature estimates

Predictive Process (Waterfall) The plan creates the schedule and cost estimates

Predictive vs Adaptive Process Waterfall vs Agile

Agile for FPGA 9

PLAN DRIVEN

SCHEDULE

SCHEDULE COST FEATURES

SCOPE COST


Predictability through Agility

Leverage iterative nature of Agile to take the highest priority features from inception to completion

Focus on the customer’s most important features for each iteration and revisit with each new iteration

For each target feature in the iteration, fully design, verify, implement and test in hardware

Fully develop features to spread difficult to schedule tasks across the project

Fully develop features to improve visibility of nebulous tasks as well as end of project cloudiness

Use lessons learned from previous iterations to better predict future iterations

Pipe clean tools and process early to streamline future iterations

Agile for FPGA 10

Specification

Design

Verification Implementation

Hardware Feature 1 Feature 2 Feature 3


Adapting to Change with Agile

Specification Design Verification Implementation Hardware

Agile for FPGA 11

Specification

Design


Hardware

Specification

Design


Hardware

Specification

Design


Hardware

Waterfall

Project Timeline Competition Changes

Requirement Changes

Market Changes

Technology Changes

Timeline Changes

Agile


Visibility Considerations Waterfall vs Agile

Hardware

Implementation

Verification

Design

Specification

Agile for FPGA 12

Hardware

Implementation

Verification

Design

Specification

50%

Done?

Waterfall Agile

Featu

re 1

Featu

re 2

Featu

re 3

Featu

re 4

Featu

re n

50% Done


Agile FPGA Advantage FPGAs a natural fit for Agile over ASICs

Waterfall often must be used in cases where changes are costly (e.g. ASIC tape-out)

FPGA development flows provides software like low cost changes

FPGA’s re-programmability inherently enable fast changes throughout the flow

Leverage custom boards or off-the-shelf FPGA reference boards

Agile for FPGA 13


Requirement Changes


Agile for FPGA 14

Specification

Design


Hardware

Specification

Design


Hardware

Specification

Design


Hardware

Waterfall

Project Timeline Requirement Changes

Agile

Feature 1 Feature 2 Feature 3



Assess Requirements

Reassess Requirements

Reassess Requirements


Quality Considerations Waterfall vs Agile


Agile for FPGA 15

Specification

Design


Hardware

Specification

Design


Hardware

Specification

Design


Hardware

Waterfall

Project Timeline

Agile




New Project Timeline

Out of Time or Money


When is Verification Done? Coverage Closure Is Generally Not Linear…

Agile for FPGA 16

Difficult to predict for accurate scheduling

Challenging to understand state of progress and time left

80% coverage closure does not mean 20% effort remains

It is more likely that the last 20% of coverage takes 80% of time!

Coverage closure requires: — Adding new tests — Evaluating and modifying

coverage — Tuning randomization constraints — Fixing bugs uncovered during

closure

Time

Covera

ge

100%

50%

75%

90%


How Much Effort is Left?

Agile for FPGA 17


Coverage Closure Scheduling Challenges High level view

Agile for FPGA 18

Project #1 Project #2 Project #3 Testplan Section

Verification Status

Testplan Section

Verification Status

Testplan Section

Verification Status

Testbench 80% Testbench 80% Testbench 80%

Which project is further along?


Coverage Closure Scheduling Challenges

Agile for FPGA 19


Verification Status

Testplan Section

Verification Status

Testplan Section

Verification Status


Block A 80% Block A 90% Block A 100%

Block B 80% Block B 90% Block B 100%

Block C 80% Block C 60% Block C 40%

Which project is further along?

Furthest Along Full Tested

Highest Priority Releasable Products


Agile Coverage Closure Improving schedule predictability

Agile for FPGA 20


Verification Status

Testplan Section

Verification Status

Testplan Section

Verification Status


Block A 80% Block A 90% Block A 100%

Feature A - 1 80% Feature A - 1 100% Feature A - 1 100%




Block B 80% Block B 90% Block B 100%

Feature B - 1 80% Feature B - 1 80% Feature B - 1 100%




Block C 80% Block C 60% Block C 40%

Feature C - 1 80% Feature C - 1 70% Feature C - 1 80%




Fully close highest priority features

Removes uncertainty associated with coverage closure

Avoid “uncertain” tasks associated with that functionality once at 100%

Learn overall effort to close items and apply that to future features


Agile Timing Closure

Agile for FPGA 21

Waterfall approach can make it difficult to make changes in earlier phases

Agile’s iterative approach encourages continuous development throughout all phases

Revisiting early stages of design can result in higher impact

Requires the whole process to be repeated (e.g. verification)

Tuning timing as new features are added improves predictability of timing closure by breaking the problem into smaller pieces


Agile Timing Closure


Agile for FPGA 22

Specification

Design


Hardware

Specification

Design


Hardware

Specification

Design


Hardware

Waterfall

Project Timeline Timing Closure

Problems

Agile




Timing Closure

Problems

Timing Closure

Problems


Summarizing Waterfall vs Agile

Agile for FPGA 23


Agile Projects Are

3x More Successful

Summary Advantages of applying iterative development

Success 43%

Challenged 48%

Failed 9%

Agile

Agile for FPGA 24

Success 14%

Challenged 57%

Failed 29%

Waterfall

Challenged: • Late • Over budget • Lacking

functionality • Low quality

Source: The CHAOS Manifesto, The Standish Group, 2012

FPGAs well suited to Agile

Improved predictability

Improved visibility

Improved quality

Reduced risk

Improved adaptability

Realities of FPGA Projects


Documents

Using Agile to Improve the FPGA Development …...2014 2012: A v er ag e 43% 2014: A v er ag e 46% 4.0 4.4 2.6 3.8 0 2 4 6 8 10 2012 2014 M e a n P e a k N u m b e r o f F P G A E