Boeing “Value Front” Tool (VFT) Multi-Attribute Trade Study Tool

A Phantom Works System Assessment team effort

Mark SchankmanPhantom Works Affordability Engineering

Cost & Affordability

Decision Analysis Effectiveness Analysis

23 March 2005

Chart 2

23 March 2005

Agenda

• Goals of this Presentation

• Definitions

• Purpose of Tool

• Background to Value Front Tool

• Demonstration Trade Study

• Methods and Tool Process

• Summary and Questions

Chart 3

23 March 2005

Goals of this Workshop

• Familiarize you: Trade Study Methods and Tool

• Inform you:

– Use of Multi-Attribute Decision Methods for Design Trade Studies

• Solicit Your Ideas: Potential Applications of Trade Study Tool

• Have FUN !!

Chart 4

23 March 2005

Definition of Terms

• VFT: Value Front Tool

• LCC: Life Cycle Cost

• Uncertainty: Degree to which the cost, technical performance, or schedule is unknown(measured by probability distributions)

• CAIV: Cost as an Independent Variable (also known as: “Target Costing”)

Illustration: Design a Hot Beverage Container

Chart 5

23 March 2005

Uti

lity

(C

ust

om

er S

atis

fact

ion

) o

r P

erfo

rman

ce

Cost

PerformanceThreshold (minimum)

Cost Objective (Target)

PerformanceObjective (desired)

Cost Threshold(not to exceed)

Trade Space

Best Value Concepts Lie on the “Value Front Line”

“Value Front” = Concepts that Maximize Customer Satisfaction with the Least Increase in Cost

Value Front Tool Identifies “Best Value” Alternatives Within the Trade Space

Concepts that provide less value per $

Chart 6

23 March 2005

CAIV Analysis Includes Measure of Performance & Cost Uncertainty

Util

ity =

Cus

tom

er D

esira

bilit

y

Life Cycle Cost

Value Front = Best Value Concepts

Technical Uncertainty

Cost & ScheduleUncertainty

Trade Space

Chart 7

23 March 2005

Methods Needed to Evaluate “Best Value”

Value FrontTool

Customer Need Priorities(Decision Analysis)

CustomerDesirability

Attribute Value

O/SDevProd

Cost-Risk

Attribute Value Distribution

UncertaintySimulation

Chart 8

23 March 2005

Value Front Method and Tool Capabilitiesthat Meet Systems Engineering Needs

• Link performance, cost, effectiveness and “risk” and measures their ability to meet customer needs

• Identify “Best Value” solutions in the design space = Systems Engineering Process for Trade Studies

• Include uncertainty in trade study evaluations

(Boeing PRO 4819: ..“balance cost, performance, & risk)

Boeing success as Systems Integrator depends on tools that Deploy the Systems Engineering Process

Chart 9

23 March 2005

Systems Engineering Trade Study Process Key Steps Addressed by Value Front Tool

Value Front Tool Provides Discipline to the Systems Engineering Trade Study Process

Define thepurpose

Establishevaluation

criteria & weightsthat meet customer needs

Identifyalternativesolutions

Screenalternatives

Document results

Analyze results Select “Best Value”

alternative

Evaluate “down selected” alternatives

– Performance– Weight– Reliability– Effectiveness– Safety– Etc

Trades -77/28/00

Accept/reject results

Steps in Value

Front Tool

Chart 10

23 March 2005

Example Value Front Tool Analysis :

Which Drip Coffee Maker to Buy ?

Typical Consumer Reports®

Value Comparison•Chart from “Consumer Reports” Dec 2004, “Coffee Makers”

We all evaluate performance criteria (if just in our heads)

Value Comparison

• Performance Criteria scored

• Total utility score

Performance Criteria

Average Cost

Chart 12

23 March 2005

Value Front ToolInputs Required

• Performance Criteria relative importance (from QFD or other Decision Analysis methods)

• Performance Criteria values (e.g., Warranty period in years)

• Utility Curves (use threshold and objective values) (must represent customer’s expectations)

• Life Cycle Cost(Purchase Price + Operating cost ) **

** Assume Development Cost included in Purchase Price

Chart 13

23 March 2005Sample Evaluation Criteria Scores Braun KF400

Source: Consumer Reports© December 2004

Attributes evaluated for five coffee makers

Chart 14

23 March 2005

Choose Your Criteria Scores & Weights“Mr. Coffee©” Coffee Maker

X X X X X X

Chart 15

23 March 2005Boeing Value Front ToolMain Screen

Chart 16

23 March 2005Evaluation Criteria Scores from VFTFive Coffee Maker Comparison

Attribute Scores based on Consumer Reports®

UtilityScore

WT*ScoreUtilityScore

WT*ScoreUtilityScore

WT*ScoreUtilityScore

WT*ScoreUtilityScore

WT*Score

Attribute NameAttribute Weight

(expected)

Ease of use 0.20 0.90 0.17 0.90 0.17 0.80 0.15 0.97 0.18 0.60 0.11

Brewing 0.30 0.95 0.31 0.90 0.29 0.80 0.26 0.50 0.16 0.50 0.16

Programmable 0.10 0.00 0.00 1.00 0.09 1.00 0.09 0.10 0.01 0.10 0.01

Cleaning 0.20 0.40 0.08 0.80 0.15 0.80 0.15 0.87 0.16 0.50 0.09

Cool to touch/ temperature

0.10 0.60 0.06 1.00 0.09 0.80 0.08 0.00 0.00 0.60 0.06

Warranty period (years)

0.10 1.00 0.11 1.00 0.11 1.00 0.11 0.40 0.04 0.80 0.09

Total WeightedUtility Score

1.00 0.72 0.91 0.84 0.56 0.52

Utility Uncertainty Metric 27% 23% 23% 27% 25%

Proctor SilexBraun KF400 Compresso Braun KF180 Black & Decker

Coffee Maker

Price: $20 $170 $60 $35 $15“You Get What You Pay For”

Good

Poor

Chart 17

23 March 2005

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

$0 $50 $100 $150 $200 $250

Life Cycle Cost(Purchase Price + Operating Cost in Dollars)

Uti

lity

Sc

ore

(Wei

gh

ted

Ave

rag

e)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

$0 $50 $100 $150 $200 $250

Life Cycle Cost(Purchase Price + Operating Cost in Dollars)

Util

ity S

core

(W

eig

hte

d A

vera

ge)

Value Front

Value Front Tool Results Utility Value vs. Cost (CAIV)

Black & Decker

Proctor Silex

Braun KF400

Compresso

Braun KF180

Chart 18

23 March 2005

Method to Visualize Utility and Best Value Uncertainty

PerformanceAttribute (eg, range, speed)

Producibility

Reliability

Safety Mission Effectiveness(eg, survivability)

Risk - measured as the probability that attribute is less than expected (in Value Front Method)

Utility - Extent to which customer

is satisfied

Chart 19

23 March 2005

Utility Attribute Contribution for Concept 2

0.00

0.20

0.40

0.60

0.80

1.001

2

3

4

5

6

Value Front ToolUtility Plot for Compresso Coffee Maker

Brewing

Warranty

Ease of Use

Cool to Touch

Cleaning

Programmable

Utility - Extent to which customer

is satisfied

Utility Score = 1.0

Chart 20

23 March 2005Utility Radar Plots All Coffee Makers Evaluated

Braun KF400

CompressoBraun KF180

Black & Decker

Proctor Silex

Utility Attribute Contributionfor Concept 3

0.00

0.20

0.40

0.60

0.80

1.001

2

3

4

5

6

Utility Attribute Contributionfor Concept 1

0.00

0.20

0.40

0.60

0.80

1.001

2

3

4

5

6

Utility Attribute Contributionfor Concept 2

0.00

0.20

0.40

0.60

0.80

1.001

2

3

4

5

6

Utility Attribute Contributionfor Concept 4

0.00

0.20

0.40

0.60

0.80

1.001

2

3

4

5

6

Utility Attribute Contributionfor Concept 5

0.00

0.20

0.40

0.60

0.80

1.001

2

3

4

5

6

Chart 21

23 March 2005

Methodology Used in

Value Front Tool

Chart 22

23 March 2005

Trade Study Methods are Integrated withValue Front Tool

Plot Utility vs CostIdentify Value Front

Util

ity =

Des

irabi

lity

Cost

Value Front3

CAIV Analysis identifies “best value”

•Utility Analysis•QFD (house of quality)•Uncertainty simulation•CAIV analysis•Cost-risk estimating

Utility Analysis uses Inputs

from IPTs

Attribute Value

Score Attribute Utilityfor each Concept

1001000

0.2

0.4

0.6

0.8

1

0 806040200 80

Threshold Goal

Desirability

Risk

604020

2

Util

ity =

Des

irabi

lity

Requirements Analysis

Facilitated by Decision Analysis

Technical Attributes

Cus

tom

er N

eeds

Attribute Importance Score

1

Translate Customer Needs

to System Technical Attributes Methods

Used

Chart 23

23 March 2005

Front End “House of Quality” Inputs

Plot Utility vs CostIdentify Value Front

Util

ity =

Des

irabi

lity

Cost

Value Front3

CAIV Analysis identifies “best value”

•Utility Analysis•QFD (house of quality)•Uncertainty simulation•CAIV analysis•Cost-risk estimating

Utility Analysis uses Inputs

from IPTs

Attribute Value

Score Attribute Utilityfor each Concept

1001000

0.2

0.4

0.6

0.8

1

0 806040200 80

Threshold Goal

Desirability

Risk

604020

2

Util

ity =

Des

irabi

lity

Requirements Analysis

Facilitated by Decision Analysis

Technical Attributes

Cus

tom

er N

eeds

Attribute Importance Score

1

Translate Customer Needs

to System Technical AttributesOperations Analysis

Methods Used

1

Translate Customer Needs

to System Technical Attributes

EvaluationsCriteria

orWHATs

Alt Solutions or HOWs

Combined Scores

WeightingFactors

RoofTechnical Attributes

Cus

tom

er

Nee

ds

Attribute Importance

Score

Chart 24

23 March 2005

Requirements Analysis

Facilitated by Decision Analysis

Technical Attributes

Cus

tom

er N

eeds

Attribute Importance Score

1

Translate Customer Needs

to System Technical Attributes

Utility Analysis uses Inputsfrom IPTs

Attribute Value

Score Attribute Utilityfor each Concept

1001000

0.2

0.4

0.6

0.8

1

0 806040200 80

Threshold Goal

604020

2U

tility

=D

esira

bilit

y

Value Front ToolUtility Analysis

Plot Utility vs CostIdentify Value Front

Util

ity =

Des

irabi

lity

Cost

Value Front3

CAIV Analysis identifies “best value”

Operations Analysis Methods Used

Chart 25

23 March 2005

Transform Attribute Value to Customer Desirability (“Utility”)

Attribute Value

0

0.2

0.4

0.6

0.8

1.0

0 20 40 60 80 100

De

sir

ab

ility

(U

tilit

y S

co

re)

Threshold Objective

Risk

ExpectedValue

Distribution DeterminedBy Worst / Best value expected

Chart 26

23 March 2005

Value Front Tool Boeing Potential Program Users

Conceptual Trade Studies:

• JDAM Container trade (completed)

• LASER Test bed aircraft trade – Phase I used commercial Utility Tool

• Advanced Rotorcraft – Considered by Boeing-Philadelphia

Engineering Disciplines Affected:

• Systems Engineering

• Operations Analysis: Cost & Affordability Decision Analysis (Utility Analyses) Effectiveness

Chart 27

23 March 2005

Status of Value Front Tool

• Successfully applied tool for JDAM Container trade study

• User Instructions planned later in year

• Beta Test version with “self-contained” Tool Planned– Sensitivity analysis capability– Plotting enhancements– Help screens

Current Excel “Prototype” tool will be converted to “Visual Basic” Tool for “interactive” use with customers

Chart 28

23 March 2005

Boeing IDS Example Analysis :

Weapon Container Trade Study

Chart 29

23 March 2005Weapon Container Affordability Analysis

Utility (Value to Customer) vs. Cost Uncertainty Region Define Boundaries of Utility Score & Cost

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

Cost - Millions of Dollars

Uti

lity

Sco

re W

eig

hte

d A

vera

ge

ECS #1

ECS #3

Hardigg

PRC #3

PRC #4

1

1

2

3

Concept

4

5

Value Front Uncertainty Region(Bounded by 10% and 90% Uncertainty)

2

3

4

5

1

Greater cost uncertainty: bad choice ?

Chart 30

23 March 2005

Value Front ToolSummary

• Tool provides standardized, disciplined, methods to evaluate cost, performance, and risk of design & technology solutions

• Identifies “best value” solutions in the design space as an essential part of the trade study process

• Tool provides contractor a competitive advantage by quantifying a concept’s “value” to the customer.

• The metrics necessary to make design and technology investment decisions are all evaluated (i.e., cost, performance, effectiveness, uncertainty).

“Value Front” Methods & Tool Help DeployAffordability Best Practices supporting the Systems Engineering Process

Chart 31

23 March 2005

References

Cost & Affordability: Matt Anderson (314-232-0931)

Decision Analysis: David Hamilton (314-234-3067)

Decision Analysis: Mike Wheeler (314-234-2405)

Target Costing Society: http://www.cam-i.org/

Affordability Resource Database: http://apat.web.boeing.com

Affordability Best Practice (Boeing): PRO-4819

Chart 32

23 March 2005

Backup Charts

Chart 33

23 March 2005

TopLevel

Objectives

ObjectiveImportanceRatings

Operational Attributes

Operational Attribute Priorities

QFD Matrices Description

The Value Front Tool Utilizes the Normalized Priorities Derived In a

QFD Effort

(3X7) + (1X5)

Be Able to Fly Long Distances

Be Able to Easily Detect Moving Targets

Be Re-deployable

Range Loiter TimeSortie

GenerationRelativePriorities

9

7

5

9

3

86 45 26(157)

Priority

Normalized Priority .55 .29 .16

1 9 1

System Attributes

Operational Attributes

Chart 34

23 March 2005

Value Front Tool – Uncertainty Input TemplatesDeveloped at Boeing-Huntington Beach to Select Technical and

Cost Uncertainty Ranges for Cost-Risk Analysis

Risk category

COST ESTIMATE RISKNotional

lower value

Notional upper value

ACommercial off-the-shelf (COTS); catalog pricing, high volume production, thousands built.

0.990 1.050

B Firm fixed-price quotation based on multiple sources. 0.970 1.083C Firm fixed-price quotation based on single sources. 0.950 1.118

DIdentical item in full production, auditable detailed historical actuals available.

0.930 1.155

ECost-plus quotation from multiple sources (with cost/price analysis).

0.910 1.195

FCost-plus quotation from single sources (with cost/price analysis).

0.890 1.236

GIdentical item in full production, limited historical actuals available.

0.860 1.280

H In-house detailed pricing estimate (auditable). 0.850 1.326

I In-house and industry models/ CERs, calibrated to the product line, used extensively, and updated periodically.

0.830 1.375

J Analog to similar item in ful production. 0.810 1.428

KCost-plus quotation from multiple sources (without cost/price analysis).

0.790 1.483

LCost-plus quotation from single sources (without cost/price analysis).

0.770 1.543

MIn-house and industry models/ CERs, generic, product type included in CER database.

0.750 1.606

NRough-order-of-magnitude (ROM) estimate. Lacks full documentation and not auditable (from vendor or in-house).

0.730 1.674

O Engineering estimate (experienced in cost estimating). 0.710 1.747

PIn-house pricing estimate (limited or no database on product).

0.690 1.825

QIn-house and industry models/ CERs, generic, product type excluded from CER database.

0.670 1.909

REngineering estimate (little experience in cost estimating).

0.650 2.000

Risk category

Cost risk assessment template TECHNICAL RISK (NRE and REC) (design, technology, integration)

Notional lower value

Notional upper value

AQualified off- the- shelf item, fully meets defined requirements, stable, and no user uncertainty. All materials reflect current technologies in production.

0.94 1.06

BOff-the-shelf items meet defined requirements with qualifications. All materials reflect current technologies in production.

0.90 1.20

C

Design effort required using standard existing components within original specification levels, minor modifications, minimal secondary requirements need definitions. All materials and processes reflect current technologies.

0.85 1.35

D

Design effort required using existing components beyond original Specification levels—some require major modifications, some noncritical requirements undefined, some user uncertainty. All materials and processes reflect technologies verified through prototyping.

0.80 1.55

E

Moderate engineering, development required using existing design knowledge/capability, and some critical requirements not clearly defined. All materials and processes reflect technologies verified through prototyping.

0.80 1.60

FMajor engineering development required, several critical requirements not defined. Materials and processes verified through analysis only.

0.75 1.75

GNew or breakthrough advance in design-critical requirements not defined or very difficult to achieve. Some materials/processes require scientific research.

0.70 2.00

Cost RiskTechnical Risk

Chart 35

23 March 2005

“Best Value” (CAIV) Analysis is Facilitated withSystems Assessment Integrated Tools

“Value Front”Trade Study Tool• Customer Satisfaction Metric =

“utility” (0 – 1.0)

• Combines “utility” metric for each technical attribute

• Measures “utility” uncertainty based on technical risk

Analogy Cost- Risk Tool

• Estimates concept cost compared to baseline (known cost)

• Measures cost & schedule uncertainty

Life Cycle Cost

Value Front

Trade Study

Option

Cost

Threshold

Cost & Schedule Uncertainty

Technical Uncertainty

Util

ity =

Des

irabi

lity

Chart 36

23 March 2005

Example Attribute Criteria (source: Boeing Trade Study Workshop - Al Bruns)

• Product capability– Operational utility– Performance– Effectiveness– Survivability

• Producibility– Task time– Cycle time– Mistake proofing

• Supportability– Maintainability– Design for upgrade– Logistics– Training– Reliability

• Environmental• Safety• Political• Ease of use

Attributes should be measurable (but can be qualitative)

Chart 37

23 March 2005

Categories of Trade Studies Performed withValue Front Tool

• Preferred Design concept:– System, sub-system, or component trade studies

• Preferred Technology concept – Evaluate technology investment vs. risk (uncertainty)– Evaluate new/emerging technologies for “best value” that

meets Boeing strategic needs

• Evaluate manufacturing options

• Screen tool to “weed out” concepts with less value to customer (performance vs. cost)

Decisions can be Design or Technology Selection