Benchmarking the Benefits and Current Maturity of …...2020/03/26 · • DoD Digital Engineering Strategy ―Vision, mission, and goals can be generalized to any organization •

1

Benchmarking the Benefits and Current Maturity of Digital Engineering/Model-Based SE

Tom McDermott, Stevens Inst. of Technology Nicole Hutchison, Megan Clifford (Stevens)

Eileen Van Aken, Alejandro Salado, Kaitlin Henderson (Va Tech)

www.sercuarc.org

This material is based upon work supported, in whole or in part, by the U.S. Department of Defense through the Systems Engineering Research Center (SERC) under Contract H98230-08-D-0171. The SERC is a federally funded University Affiliated Research Center (UARC) managed by

Stevens Institute of Technology consisting of a collaborative network of over 20 universities. More information is available at www.SERCuarc.org

http://www.sercuarc.org/

Systems Engineering Research Center 2

Frameworks for Evaluating DE Value

• DoD Digital Engineering Strategy ―Vision, mission, and goals can be generalized to any organization

• INCOSE Model-Based Enterprise Capability Matrix ―Benchmark for Organizational and Industry-wide DE Maturity, published

January 2020

• DE/MBSE Value, Benefits, and Metrics Framework ―Developed by the SERC, four categories for DE value/benefits:

Quality, Velocity/Agility, User Experience, Knowledge Transfer

• DE/MBSE Enterprise Adoption Framework ―Developed by the SERC from the Baldrige CPE Framework:

Adoption


DoD Digital Engineering Strategy

STRATEGIC MANAGEMENT SYSTEM Digital Engineering (DE) Vision: Modernize how the Department designs, develops, delivers, operates, and sustains systems. DE Mission: Securely and safely connect people, processes, data, and capabilities across an end-to-end digital enterprise.

WAY

S /I

NIT

IATI

VES

END

S EN

DS

WAYS / IN

ITIATIVES M

EAN

S MEANS

An enduring, authoritative

source of truth is used over the

lifecycle

Plan and develop the authoritative source

of truth

Govern the authoritative source

of truth

Use technological innovation to

improve engineering practices

Infuse technological innovations to enable the end-to-end digital

enterprise

Advance human-machine interactions

Make use of data to improve awareness,

insights, and decision making

Models are used to inform enterprise

and program decision making

Formalize the planning for models to support engineering activities and decision making across the lifecycle

Use the authoritative source of truth across

the lifecycle

Use models to support engineering activities and decision making across the lifecycle

Improve the digital engineering

knowledge base

Transform culture and workforce

engineering across the lifecycle

Build and prepare the workforce

Lead and support digital engineering

transformation efforts

Process, Methods, Tools, Technology, People

Infrastructure and environments support improved

communication and collaboration

Develop, mature, and use digital

engineering IT infrastructures

Secure IT infrastructure and protect intellectual

property

Develop, mature, and use digital

engineering methodologies

Formally develop, integrate,

and curate models


Focus: Enterprise Transformation Measures

• Gartner: “Select just 5 to 9 metrics to track, report and act on. The value of a metric lies in its ability to influence business decision making.”

• The best metrics: ―Have a defined and defensible causal relationship to a business outcome ―Work as a leading, not lagging, indicator ―Address a specific defined audience – in a way they can understand ―Drive action when they change from green to yellow to red

• There are no universal metrics – must be enterprise specific • Good Digital Transformation metrics have some traits ―Measure people adoption, and enterprise process adoption ―Analyze breadth of usability, and issues with usability ―Measure productivity indicators ―Generate new value to the enterprise (revenue, operational efficiency, etc.)

https://www.gartner.com/smarterwithgartner/how-to-measure-digital-transformation-progress/


Summary DE Success Measures Framework

Quality: • Reduce Errors/Defects • Improve System Quality • Improve Traceability • Reduce Cost

Knowledge Transfer: • Better access to

information • Better communication/

info sharing • Collaboration

An enduring, authoritative

source of truth is used over the

lifecycle

Use technological innovation to

improve engineering

practices

Models are used to inform enterprise

and program decision making

Transform culture and workforce

engineering across the lifecycle

Infrastructure and environments support improved

communication and collaboration

Adoption: • Methods/Processes • Roles/Skills • Training/Tools • Leadership support • Change Mgmt Process • Resources

Velocity/Agility: • More Reuse • Improve Consistency • Increase Efficiency • Support Integration • Reduce Time

User Experience: • Manage Complexity • Improved System

Understanding • Automation


MBSE Benefits: Literature Review Results

• Searched papers that mention a benefit of MBSE and what the source of that benefit was: measured gains, observed gains, perceived gains (no source for benefit), reference. ―Total Papers that mention MBSE: 847 oPapers that mention benefits: 360 ―Measured gains: 2 ―Observed gains: 27 ―Perceived gains: 236 ―Reference: 114 ―Misc.: 2

*Kaitlin Henderson (VT) PhD studies


Literature Review Benefit Categories (48)

Category List of Benefits

Quality

Reduce errors/ defects Improved risk analysis Improved capability

Improved traceability Improved system design More stakeholder involvement

Improved system quality Better requirements generation Strengthened testing

Reduce risk Increased accuracy of estimates Reduce cost

Increased rigor Improved predictive ability Better analysis capability

Increased effectiveness Improved deliverable quality

Velocity/ Agility

Improved consistency Increased productivity Higher level support for integration Increased capacity for reuse Increased transparency Increased uniformity

Increased efficiency Increased confidence Increased precision

Reduce rework Increased flexibility Early V&V

Reduce time Better requirements management Reduce ambiguity

Reduce waste Ease of design customization Easy to make changes

User Experience

Higher level support for automation Improved system understanding Reduce effort

Reduce burden of SE tasks Better data management/capture

Better manage complexity Better decision making

Knowledge Transfer

Better accessibility of info Improved architecture Better communication/ info sharing Better knowledge management/ capture Multiple viewpoints of model Improved collaboration


Literature Review Quantitative Results (most cited)

Category Perceived Observed Measured

Quality

Reduce Errors/Defects (16) Reduce Errors/Defects (26) Reduce Errors/Defects (2)

Improve Traceability (61) Improve Traceability (9) Improve Traceability (1)

Improve System Quality (21) Improve System Quality (0) Improve System Quality (1)

Reduce Risk (22) Reduce Risk (2) Reduce Risk (1)

Increased Rigor (0) Increased Rigor (0) Increased Rigor (1)

Reduce Cost (33) Reduce Cost (4) Reduce Cost (0)

Velocity/ Agility

Consistency (44) Consistency (6) Consistency (1)

Reuse (37) Reuse (5) Reuse (1)

Efficiency (13) Efficiency (2) Efficiency (0)

Improve Standardization Improve Standardization Improve Standardization (0)

Collaboration/Info Sharing (68) Collaboration/Info Sharing (11) Collaboration/Info Sharing (0)

Integration/V&V (11) Integration/V&V (3) Integration/V&V (1)

Reduce Time (24) Reduce Time (8) Reduce Time (1)

User Experience

Automation (0) Automation (0) Automation (2)

Reduce SE Task Burden (0) Reduce SE Task Burden (0) Reduce SE Task Burden (1)

Manage Complexity (48) Manage Complexity (2) Manage Complexity (0)

Productivity (14) Productivity (0) Productivity (0)

System Understanding (24) System Understanding (2) System Understanding (0)

Knowledge Transfer

Information Access (27) Information Access (5) Information Access (2)

Knowledge Capture/Sharing (13) Knowledge Capture/Sharing (0) Knowledge Capture/Sharing (2)

Architecture/Sys Understanding (23) Architecture/Sys Understanding (2) Architecture/Sys Understanding (1)


mbsematuritysurvey.sercuarc.org

Final Survey data as of February 1, 2020. Quantitative metrics reflect “effective” participants. Information derived from text

responses reflects “all” participants


MBSE Survey Overview Topics Summary of Survey Questions 7. Model

Sharing and Reuse

19. Support model libraries for reuse 20. Have interfaces around models for stakeholder use 21. Shared models are used to consistently manage

programs across lifecycle 22. Identify practices for data/model discovery, reuse?

8. Modeling Environments

23. Our modeling environment is secure 24. Our modeling environment protects IP 25. Have defined processes for tools, data

interoperability 26. Identify benefits from collaborating on models

across disciplines

9. Organizational Implementation

27. Identify most challenging org obstacles for MBSE 28. Identify best organizational enablers for MBSE 29. Identify biggest changes our org needs for MBSE

10. Workforce 30. Have defined critical roles to support MBSE 31. Identify top MBSE roles in your organization 32. Have sufficient staffing to fill MBSE-related roles

11. MBSE Skills 33. Have defined critical skills for MBSE 34. Identify the most critical skills for MBSE

12. Demographics

Organizational size, domain, MBSE experience, role

Topics Summary of Survey Questions 1. MBSE

Usage 1. MBSE strategy is integrated with product strategy at the

enterprise level 2. MBSE processes & tools are integrated with product-level

processes and tools 3. Most important reasons for integrating MBSE

2. Model Manage-ment

4. Have a taxonomy for modeling across organization 5. Have defined processes/tools for model management 6. Have standard guidance for model management/tools 7. Identify business value from consistent model management

3. Technical Manage-ment

8. Use modeling as the basis for technical processes 9. MBSE process supports our technical review process 10. Identify benefits or challenges of MBSE in technical

reviews

4. Metrics 11. Modeling provides measurable improvement across projects

12. Have consistent metrics across programs/enterprise 13. Identify any metrics that have proven useful

5. Model Quality

14. Have defined processes/tools for V&V of models 15. Have defined processes/tools for data/model quality

assurance

6. Data Manage-ment

16. Have effective approaches for managing the data interface between tools 17. Data is managed independent of tools for portability 18. Identify new data management roles/processes

Survey content is derived from the draft INCOSE Digital Engineering Capabilities Definition

*Questions in italics elicit free text responses


Demographics - My organization is:

Aerospace and defense, 56

Automotive, 7

Consultancy and Services, 6

Tools & MBSE services, 4

Agricultural, 4

Manufacturing , 2 Meteorology, 1

Ship building, 1 Rail, 1 R&D, 1

Medical device, 1 Health IT, 1 Genomics, 1

Food & beverage packaging, 1

Electronics Manufacturing, 1

Construction and infrastructure, 1

Industry-Reported Market Areas


Demographic – type, size, experience

0

5

10

15

20

25


-29 -63

-86 -98 -92

-118 -77

30 -153

-61 -73

-91 -111

18 -85

-107 243

195 -10

-70 -165

-53 -24

-250 -150 -50 50 150 250

1. Mature use strategy2. Mature process/tool strategies4. Consistent lexicon & taxonomy across enterprise5. Mature model management processes6. Standard program & business guidance for models8. Models are the basis for technical processes9. MBSE is the basis for Technical Reviews11. Modeling provides measurable improvements12. Have consistent metrics across enterprise14. Consistent data/model V&V processes15. Consistent data/model quality assurance processes16. Processes to manage data interface between tools17. Data is portable across organizations & tools19. Support model libraries for model reuse20. Libraries support discoverable knowledge21. Consistent use of shared models23. Trust that environment is secure24. Trust that environment protects IP25. Have processes and tool selection & interoperability30. Have clearly defined roles supporting MBSE32. Have sufficient staffing for all roles33. Have defined critical skills supporting MBSE35. Training is linked to critical skills

Overall Survey Scores

MBSE Usage

Model Management

Technical Management

Metrics

Model Quality

Data Management

Model Sharing & Reuse

Modeling Environment

Workforce

MBSE Skills

AGREE DISAGREE


Demographics – Participant roles

Systems Engineer, 92

Project, Product or other

Management, 29

Executive Management, 17

Modeling and Simulation

Professional, 13

Other, 12

Acquisition Professional, 4

Other Engineering or

Software Disciplines, 3


Overall Survey Scores by Role

MBSE Usage

Model Management

Technical Management

Metrics

Model Quality

Data Management

Model Sharing & Reuse

Modeling Environment

Workforce

MBSE Skills

-40

-30

-20

-10

0

10

20

30

40

1. M

atur

e us

e st

rate

gy

2. M

atur

e pr

oces

s/to

ol st

ateg

ies

4. C

onsis

tent

lexi

con

& ta

xono

my

acro

ss e

nter

prise

5. M

atur

e M

odel

Man

agem

ent P

roce

sses

6. S

tand

ard

Prog

ram

& B

usin

ess G

uida

nce

for M

odel

s

8. M

odel

s are

the

basis

for T

echn

ical

Pro

cess

es

9. M

BSE

is th

e ba

sis fo

r Tec

hnic

al R

evie

ws

11. M

odel

ing

prov

ides

mea

sura

ble

impr

ovem

ents

12. H

ave

cons

isten

t met

rics a

cros

s ent

erpr

ise

14. C

onsis

tent

dat

a/m

odel

V&

V pr

oces

ses

15. C

onsis

tent

dat

a/m

odel

QA

proc

esse

s

16. P

roce

sses

to m

anag

e da

ta in

terf

ace

betw

een

tool

s

17. D

ata

is po

rtab

le a

cros

s org

aniza

tions

& to

ols

19. S

uppo

rt m

odel

libr

arie

s for

mod

el re

use

20. L

ibra

ries s

uppo

rt d

iscov

erab

le k

now

ledg

e

21. C

onsis

tent

use

of s

hare

d m

odel

s

23. T

rust

that

env

ironm

ent i

s sec

ure

24. T

rust

that

env

ironm

ent p

rote

cts I

P

25. H

ave

proc

esse

s for

tool

sele

ctio

n &

inte

rope

rabi

lity

30. H

ave

clea

rly d

efin

ed ro

les s

uppo

rtin

g M

BSE

32. H

ave

suffi

cien

t sta

ffing

for a

ll ro

les

33. H

ave

defin

ed c

ritic

al sk

ills s

uppo

rtin

g M

BSE

35. T

rain

ing

is lin

ked

to c

ritic

al sk

ills

Executive Management

Program/Project/Other Management

System Engineers and Other roles

AGRE

E DI

SAGR

EE


MBSE Usage and Model Management Survey Results

Our MBSE processes and tools are integrated with our overall product-level processes and tools.

Our organization has standard business and program guidance that defines our model management processes and tools.


Technical Management Survey Results

Our organization uses modeling as the basis for our technical processes consistently across the enterprise. Our MBSE process fully supports our technical review process.

Free-text question: Please identify any benefits or challenges your organization has found in the use of MBSE (or 'digital engineering') in the technical review process.


0

2

4

6

8

10

12

14

16

MBS

E m

etho

ds/p

roce

sses

Org

aniza

tiona

l cul

ture

Com

mun

icat

ing

succ

ess s

torie

s/pr

actic

es

Cust

omer

/sta

keho

lder

buy

-in/e

ngag

emen

t

MBS

E te

rmin

olog

y/on

tolo

gy/li

brar

ies

Lega

cy/c

urre

nt p

roce

sses

Peop

le w

illin

g to

use

MBS

E to

ols

Trai

ning

MBS

E to

ols

Cost

to u

se M

BSE

tool

s

Dem

onst

ratin

g be

nefit

s/re

sults

mor

e st

akeh

olde

r inv

olve

men

t

Lead

ersh

ip su

ppor

t/co

mm

itmen

t

Incr

ease

d tr

acea

bilit

y

mul

tiple

vie

wpo

ints

of m

odel

Com

mun

icat

ion/

info

shar

ing

Incr

ease

d ca

paci

ty fo

r reu

se

impr

oved

syst

em u

nder

stan

ding

Redu

ce ri

sk

Redu

ce ti

me

Redu

ce e

rror

s

Redu

ced

SE ta

sk b

urde

n

Incr

ease

d co

nsist

ency

Dem

onst

ratin

g be

nefit

s/re

sults

Obstacles vs. Enablers

Comparing Obstacles vs. Enablers in Tech Review Processes

Benefits of DE/MBSE Adoption of DE/MBSE

Participants were asked to respond to the prompt, “Please identify any benefits or challenges your organization has found in the use of MBSE (or 'digital engineering') in the technical review process.”


DE/MBSE Metrics Survey Result

Modeling activities in our organization provide measurable improvements within and across projects.

We have consistent metrics across our program(s)/enterprise that include our modeling activities.

Free-text question: Please identify any metrics that have proven to be useful for measuring the performance of your MBSE activities.


Model Quality Survey Results

Our organization has defined processes and tools for V&V of models at appropriate levels and program phases.

Our organization has defined processes and tools for data and model quality assurance.


Data Management Survey Results

Our organization has effective approaches for managing the data interface between tools.

Data is managed independent of tools and allows for portability across different organizational structures and related disciplines.


Model Sharing & Reuse Survey Result

Our organization supports model libraries for the purpose of model reuse.

Shared models are being used to consistently manage systems across the lifecycle.

Free-text question: Please identify any practices your organization has implemented to improve data and model discovery and reuse, either within or

between teams. Include examples of appropriate model reuse if possible.


Data/Model Discovery & Reuse

Participants were asked to respond to the prompt, “Please identify any practices your organization has implemented to improve data and model discovery and reuse, either within or between teams. Include examples of appropriate model reuse if possible.”

0

5

10

15

20

25

30

Libr

arie

s

Tool

Spe

cific

Chal

leng

es

Uni

que

vers

us In

tegr

ated

Solu

tions

EA F

ram

ewor

k

Mod

el C

urat

ion

Fram

ewor

k

Mod

el o

r Too

l Int

egra

tion

Crea

ted

MBS

E Ro

les

Data

Man

agem

ent P

roce

sses

Org

aniza

tiona

l Tax

onom

y an

dO

ntol

ogy

Oth

er

Total Responses None or NA TBD or Too Immature

Analyzable responses

Q10 Data and Model Discovery and Reuse 97 13 26 58

Foundational

Advanced


Modeling Environment Survey Results

Our organization takes steps to make sure that our modeling environment protects our intellectual property.

Our organization has defined processes and work instructions that cover tool selection, use, and related data interoperability concerns.


Workforce and Skills Survey Results

Our organization has clearly defined the critical roles to support MBSE. Our organization has clearly defined critical skills for MBSE.


Survey Results of Critical Skills

020406080

100120

Syst

ems

Engi

neer

ing…

Tool

Exp

ertis

e

Digi

tal

Engi

neer

ing

Non

-Tec

hnic

alSk

ills

Attr

ibut

es

Appl

ied

Expe

rienc

e

Soft

war

eDe

velo

pmen

t

Chal

leng

es

Mul

ti-Di

scip

linar

y

Org

aniza

tiona

lCo

ntex

t

Proj

ect

Man

agem

ent

Question 34 The most critical skills for MBSE are:


Survey Results of Critical Skills

05

1015202530

Mod

elin

g

Data

Sci

ence

Sim

ulat

ion

MBS

EEn

viro

nmen

t

Mod

elGo

vern

ance

Most Critical Digital Skills


05

1015202530

Gene

ral T

ool

Expe

rtise

Mod

elin

gLa

ngua

ges

Tool

Inte

grat

ion

Abili

ty to

Lear

n N

ewTo

ols

Most Critical Tool Expertise


020406080

100120

Syst

ems

Engi

neer

ing…

Tool

Exp

ertis

e

Digi

tal

Engi

neer

ing

Non

-Tec

hnic

alSk

ills

Attr

ibut

es

Appl

ied

Expe

rienc

e

Soft

war

eDe

velo

pmen

t

Chal

leng

es

Mul

ti-Di

scip

linar

y

Org

aniza

tiona

lCo

ntex

t

Proj

ect

Man

agem

ent



• Survey questions related to benefits are: ―Q3. What do you see as the most important reasons for integrating MBSE processes

with program and business management processes? ―Q7. Please provide one or more descriptions of the business value you are realizing

from consistent model management processes and tools. ―Q26. Please identify any additional benefits you find from collaborating on models

across disciplines.

Survey Free-Response Analysis - Benefits

Non-response

No value achieved

Response containing

benefits Q3 Reason for integrating

MBSE 41 N/A 104

Q7 Value from consistent model mgt.

23 18 70

Q26 Benefit from collaboration

24 6 52

Totals 88 24 226


List of DE/MBSE Benefit Categories (Literature Review)

Category

List of Benefits

Quality

Reduce errors/ defects Improved risk analysis Improved capability

Improved traceability Improved system design More stakeholder involvement Improved system quality Better requirements generation Strengthened testing Reduce risk Increased accuracy of estimates Reduce cost Increased rigor Improved predictive ability Better analysis capability Increased effectiveness Improved deliverable quality

Velocity/ Agility

Improved consistency Increased productivity Higher level support for integration Increased capacity for reuse Increased transparency Increased uniformity Increased efficiency Increased confidence Increased precision Reduce rework Increased flexibility Early V&V Reduce time Better requirements management Reduce ambiguity Reduce waste Ease of design customization Easy to make changes

User Experience

Higher level support for automation Improved system understanding Reduce effort

Reduce burden of SE tasks Better data management/capture

Better manage complexity Better decision making

Knowledge Transfer

Better accessibility of info Improved architecture Improved collaboration Better knowledge management/ capture

Better communication/ info sharing

Multiple viewpoints of model

Category

List of Benefits

Quality

Reduce errors/ defects

Improved risk analysis

Improved capability

Improved traceability

Improved system design

More stakeholder involvement

Improved system quality

Better requirements generation

Strengthened testing

Reduce risk

Increased accuracy of estimates

Reduce cost

Increased rigor

Improved predictive ability

Better analysis capability

Increased effectiveness

Improved deliverable quality

Velocity/ Agility

Improved consistency

Increased productivity

Higher level support for integration

Increased capacity for reuse

Increased transparency

Increased uniformity

Increased efficiency

Increased confidence

Increased precision

Reduce rework

Increased flexibility

Early V&V

Reduce time

Better requirements management

Reduce ambiguity

Reduce waste

Ease of design customization

Easy to make changes

User Experience

Higher level support for automation

Improved system understanding

Reduce effort

Reduce burden of SE tasks

Better data management/capture

Better manage complexity

Better decision making

Knowledge Transfer

Better accessibility of info

Improved architecture

Improved collaboration

Better knowledge management/ capture




Most Frequently-Reported Benefits from Survey Analysis by Question

Q3 Reason for integrating MBSE Q7 Value from consistent model mgt. Q26 Benefit from collaboration

Reduce cost (17) Increased capacity for reuse (18) Better communication/ information sharing (13)

Reduce time (17) Improved consistency (16) Improved system understanding (10)

Better accessibility of info (16) Improved system understanding (9) Better accessibility of info (6)

Increased efficiency (14) Reduce time (9) Improved consistency (5)

Improved consistency (13) Better communication/ information sharing (7)

Reduce errors (5)

Increased traceability (11) Better accessibility of info (7) Reduce time (5)

Improved system understanding (10) Reduce cost (7) Increased capacity for reuse (5)

• Integration Benefits • Model Mgmt. Benefits • Collaboration Benefits


Top Benefits Reported from Survey Questions

0

2

4

6

8

10

12

14

16

18

20

Survey Questions on Benefits- Top Responses

Q3 Reason for integrating MBSE




All Benefit Categories Reported from Survey Questions

0

2

4

6

8

10

12

14

16

18

Redu

ce c

ost

Redu

ce ti

me

Bett

er a

cces

sibili

ty o

f inf

oIn

crea

sed

effic

ienc

yIm

prov

ed c

onsis

tenc

yIn

crea

sed

trac

eabi

lity

Impr

oved

syst

em u

nder

stan

ding

Bett

er c

omm

unic

atio

n/ in

fo sh

arin

gHi

gher

leve

l sup

port

for i

nteg

ratio

nRe

duce

err

ors

Bett

er m

anag

e co

mpl

exity

Impr

oved

syst

em q

ualit

yIn

crea

sed

capa

city

for r

euse

Impr

oved

syst

em d

esig

nBe

tter

dec

ision

mak

ing

Impr

oved

col

labo

ratio

nIn

crea

sed

effe

ctiv

enes

sBe

tter

ana

lysis

cap

abili

tyBe

tter

requ

irem

ents

gen

erat

ion

Redu

ce ri

skIm

prov

ed d

eliv

erab

le q

ualit

yBe

tter

dat

a m

gt./

capt

ure

Bett

er k

now

ledg

e m

gt./

cap

ture

Redu

ce re

wor

kM

ultip

le v

iew

poin

ts o

f mod

elRe

duce

am

bigu

ityHi

gher

leve

l sup

port

for a

utom

atio

nEa

rly V

&V

Redu

ce e

ffort

Incr

ease

d pr

oduc

tivity

Impr

oved

arc

hite

ctur

eIn

crea

sed

rigor

Stre

ngth

ened

test

ing

Bete

r req

uire

men

ts m

gt.

Impr

oved

risk

ana

lysis

Incr

ease

d pr

ecisi

onIm

prov

ed c

apab

ility

Incr

ease

d fle

xibi

lity

Redu

ce w

aste

Incr

ease

d un

iform

ityM

ore

stak

ehol

der i

nvol

vem

ent

Easy

to m

ake

chan

ges

Redu

ce b

urde

n of

SE

task

sIm

prov

ed p

redi

ctiv

e ab

ility

Incr

ease

d tr

ansp

aren

cy

Q3 Reason for integrating MBSE




Analysis of Survey Question on Metrics

• Question 13 Please identify any metrics that have proven to be useful for measuring the performance of your MBSE activities was also analyzed against the literature review benefit categories

Adoption metrics: metrics related to the successful adoption and implementation of MBSE instead of measuring the value of MBSE itself

Non-

response No

metrics

Response containing

metrics Q13 Metrics

for MBSE 19 33 44

Adoption metrics

Lit review benefit metrics

Unable to categorize

Q13 Metrics for MBSE

17 72 11

Number of complete responses

Number of unique

comments


Most cited benefits and metrics categories from survey data

Top survey response metrics (Q13 only) Survey response benefits (Q3, Q7, and Q26) Better requirements generation 7 Better requirements generation 7

Reduce errors 7 Reduce errors 19 Increased traceability 6 Increased traceability 17

Better requirements mgt. 6 Better requirements mgt. 3 Improved system design 5 Improved system design 9

Reduce cost 5 Reduce cost 25 Reduce time 5 Reduce time 31

Increased capacity for reuse 5 Increased capacity for reuse 30 Better analysis capability 4 Better analysis capability 6 Improved system quality 2 Improved system quality 14 Increased effectiveness 2 Increased effectiveness 6

Higher level support for automation 2 Higher level support for automation 3 Higher level support for integration 2 Higher level support for integration 14

Top survey response metrics (Q13 only)

Survey response benefits (Q3, Q7, and Q26)


7


7

Reduce errors

7

Reduce errors

19

Increased traceability

6

Increased traceability

17

Better requirements mgt.

6

Better requirements mgt.

3


5


9

Reduce cost

5

Reduce cost

25

Reduce time

5

Reduce time

31


5


30


4


6


2


14


2


6


2


3


2


14


Most cited benefits and metrics categories from survey data

Remaining survey response metrics (Q13 only) Top Survey response benefits (Q3, Q7, and Q26) (not cited) 0 Improved consistency 34 (not cited) 0 Increased capacity for reuse 30

Better accessibility of info 1 Better accessibility of info 29 Improved system understanding 1 Improved system understanding 29

(not cited) 0 Better communication/ info sharing 29 (not cited) 0 Increased efficiency 20 (not cited) 0 Improved collaboration 13

Better knowledge mgt./ capture 1 Better knowledge mgt./ capture 8 Reduce ambiguity 1 Reduce ambiguity 8

Better manage complexity 1 Better manage complexity 7 Better decision making 1 Better decision making 7

(not cited) 0 Reduce rework 7 Improved deliverable quality 1 Improved deliverable quality 6

Better data mgt./ capture 1 Better data mgt./ capture 6 Reduce risk 1 Reduce risk 5

Increased uniformity 1 Increased uniformity 5 Multiple viewpoints of model 1 Multiple viewpoints of model 4

Strengthened testing 1 Strengthened testing 4 Reduce effort 1 Reduce effort 3

Improved capability 1 Improved capability 1

Remaining survey response metrics (Q13 only)

Top Survey response benefits (Q3, Q7, and Q26)

(not cited)

0

Improved consistency

34

(not cited)

0


30


1


29


1


29

(not cited)

0


29

(not cited)

0

Increased efficiency

20

(not cited)

0

Improved collaboration

13

Better knowledge mgt./ capture

1

Better knowledge mgt./ capture

8

Reduce ambiguity

1

Reduce ambiguity

8


1


7


1


7

(not cited)

0

Reduce rework

7


1


6

Better data mgt./ capture

1

Better data mgt./ capture

6

Reduce risk

1

Reduce risk

5


1


5


1


4


1


4

Reduce effort

1

Reduce effort

3

Improved capability

1

Improved capability

1


Challenges in Enterprise Adoption of MBSE

• Successful adoption of Model-based Systems Engineering (MBSE), like many other large-scale enterprise change initiatives, can present significant challenges - requires intentional focus on many aspects within an organization -not just the technical details of processes and tools. ―The Digital Engineering Work Group pain points relate to technical aspects such as

tools, reference models, standards, and data, as well as other organization-level challenges such as implementation/deployment approach, IT infrastructure, and training/skills of the workforce.

―Cloutier (2019) reported the top five inhibitors to successful adoption of MBSE were: cultural and general resistance to change, availability of skills, the MBSE learning curve, lack of perceived value of MBSE, and lack of management support.

• This breadth of factors demonstrates the importance of a holistic, enterprise-wide perspective in designing and implementing the approach to adopt MBSE.

• Examining MBSE adoption from the lens of the Baldrige Criteria for Performance Excellence (CPE) can generate insight and more complete understanding of MBSE adoption.


List of enterprise success factors from the survey analysis

Category

List of Success Factors

Leadership Leadership support/commitment

Leadership understanding of MBSE

Communication Awareness of MBSE benefits/value Communicating success stories/practices Need for change

Resources Cost to use MBSE tools General resources for MBSE implementation

Workforce

General MBSE awareness and knowledge People willing to use MBSE tools Teamwork

MBSE learning curve People in SE roles Training Workforce knowledge/skills

Change Processes

Champions Competing priorities Legacy/current processes Change management process design

Integration to support MBSE implementation

Vision and strategy for MBSE

Community of practice Demonstrating benefits/results

MBSE Processes MBSE methods/processes MBSE tools Security of data and IP MBSE terminology/ontology/libraries

Projects/programs to apply MBSE

Organizational Environment

Alignment with business strategy Organizational culture Success metrics

Organizational characteristics Rewards/recognition Supportive infrastructure

External Environment

Alignment with customer requirements

Customer/stakeholder buy-in/engagement

External regulations Use in SE community

Category

List of Success Factors

Leadership

Leadership support/commitment

Leadership understanding of MBSE

Communication

Awareness of MBSE benefits/value

Communicating success stories/practices

Need for change

Resources

Cost to use MBSE tools

General resources for MBSE implementation

Workforce

General MBSE awareness and knowledge

People willing to use MBSE tools

Teamwork

MBSE learning curve

People in SE roles

Training

Workforce knowledge/skills

Change Processes

Champions

Competing priorities

Legacy/current processes

Change management process design

Integration to support MBSE implementation

Vision and strategy for MBSE

Community of practice

Demonstrating benefits/results

MBSE Processes

MBSE methods/processes

MBSE tools

Security of data and IP

MBSE terminology/ontology/libraries

Projects/programs to apply MBSE

Organizational Environment

Alignment with business strategy

Organizational culture

Success metrics

Organizational characteristics

Rewards/recognition

Supportive infrastructure

External Environment

Alignment with customer requirements

Customer/stakeholder buy-in/engagement

External regulations

Use in SE community


Analysis of Survey Responses: Obstacles

• There were 166 respondents providing comments to the question on obstacles, parsed into 303 unique response comments.

05

101520253035404550

Org

aniza

tiona

l cul

ture

Wor

kfor

ce k

now

ledg

e/sk

ills

Lead

ersh

ip su

ppor

t/co

mm

itmen

tAw

aren

ess o

f MBS

E be

nefit

s/va

lue

Chan

ge m

anag

emen

t pro

cess

des

ign

Inte

grat

ion

to su

ppor

t MBS

E im

plem

enta

tion

MBS

E m

etho

ds/p

roce

sses

MBS

E to

ols

Com

petin

g pr

iorit

ies

Dem

onst

ratin

g be

nefit

s/re

sults

Gen

eral

reso

urce

s for

MBS

E im

plem

enta

tion

Trai

ning

Proj

ects

/pro

gram

s to

appl

y M

BSE

Lega

cy/c

urre

nt p

roce

sses

Gen

eral

MBS

E aw

aren

ess a

nd k

now

ledg

eLe

ader

ship

und

erst

andi

ng o

f MBS

ECo

st to

use

MBS

E to

ols

MBS

E le

arni

ng c

urve

Supp

ortiv

e in

fras

truc

ture

Alig

nmen

t with

cus

tom

er re

quire

men

tsCu

stom

er/s

take

hold

er b

uy-in

/eng

agem

ent

MBS

E te

rmin

olog

y/on

tolo

gy/li

brar

ies

Nee

d fo

r cha

nge

Alig

nmen

t with

bus

ines

s str

ateg

yEx

tern

al re

gula

tions

Org

aniza

tiona

l cha

ract

erist

ics

Rew

ards

/rec

ogni

tion

Secu

rity

of d

ata

and

IPVi

sion

and

stra

tegy

for M

BSE

Q27: Obstacles to Implementing MBSE Code

# Comments Obstacles

Organizational culture 44 Workforce knowledge/skills 30 Leadership support/commitment 25 Awareness of MBSE benefits/value 18 Change management process design 13 Integration to support MBSE implementation 13

MBSE methods/processes 13 MBSE tools 13 Competing priorities 11 Demonstrating benefits/results 11 General resources for MBSE implementation 11

Training 11 Projects/programs to apply MBSE 10 Legacy/current processes 9 General MBSE awareness and knowledge 8

Leadership understanding of MBSE 8


Analysis of Survey Responses: Enablers

• There were 156 respondents providing comments to the question on enablers, parsed into 223 unique response comments.

Code

# Comments Enablers

Leadership support/commitment 27 People willing to use MBSE tools 21 Workforce knowledge/skills 19 Champions 15 Demonstrating benefits/results 15 Training 13 People in SE roles 12 MBSE tools 11 Alignment with customer requirements 10

Change management process design 7

Communicating success stories/practices 6

Community of practice 6 Integration to support MBSE implementation 6

Use in SE community 6

0

5

10

15

20

25

30

35

40

45

50

Lead

ersh

ip su

ppor

t/co

mm

itmen

tPe

ople

will

ing

to u

se M

BSE

tool

sW

orkf

orce

kno

wle

dge/

skill

sCh

ampi

ons

Dem

onst

ratin

g be

nefit

s/re

sults

Trai

ning

Peop

le in

SE

role

sM

BSE

tool

sAl

ignm

ent w

ith c

usto

mer

requ

irem

ents

Chan

ge m

anag

emen

t pro

cess

des

ign

Com

mun

icat

ing

succ

ess s

torie

s/pr

actic

esCo

mm

unity

of p

ract

ice

Inte

grat

ion

to su

ppor

t MBS

E…U

se in

SE

com

mun

ityGe

nera

l res

ourc

es fo

r MBS

E…M

BSE

term

inol

ogy/

onto

logy

/libr

arie

sN

eed

for c

hang

ePr

ojec

ts/p

rogr

ams t

o ap

ply

MBS

EAw

aren

ess o

f MBS

E be

nefit

s/va

lue

MBS

E m

etho

ds/p

roce

sses

Org

aniza

tiona

l cul

ture

Cust

omer

/sta

keho

lder

buy

-…Te

amw

ork

Gene

ral M

BSE

awar

enes

s and

kno

wle

dge

Lead

ersh

ip u

nder

stan

ding

of M

BSE

Org

aniza

tiona

l cha

ract

erist

ics

Alig

nmen

t with

bus

ines

s str

ateg

yCo

st to

use

MBS

E to

ols

Rew

ards

/rec

ogni

tion

Succ

ess m

etric

sVi

sion

and

stra

tegy

for M

BSE

Q28: Enablers to Implementing MBSE


Analysis of Survey Responses: Changes

• There were 153 respondents providing comments to the question on changes, parsed into 273 unique response comments.

Code

# Comments Changes

MBSE methods/processes 45 Training 31 MBSE tools 20 General resources for MBSE implementation 16

Leadership support/commitment 16 Change management process design 14

Workforce knowledge/skills 13 Organizational culture 12 Integration to support MBSE implementation 11

Demonstrating benefits/results 10 MBSE terminology/ontology/libraries 9 Supportive infrastructure 9 Awareness of MBSE benefits/value 7 Leadership understanding of MBSE 6 Vision and strategy for MBSE 6

0

5

10

15

20

25

30

35

40

45

50

MBS

E m

etho

ds/p

roce

sses

Trai

ning

MBS

E to

ols

Gene

ral r

esou

rces

for M

BSE…

Lead

ersh

ip su

ppor

t/co

mm

itmen

tCh

ange

man

agem

ent p

roce

ss d

esig

nW

orkf

orce

kno

wle

dge/

skill

sO

rgan

izatio

nal c

ultu

reIn

tegr

atio

n to

supp

ort M

BSE…

Dem

onst

ratin

g be

nefit

s/re

sults

MBS

E te

rmin

olog

y/on

tolo

gy/li

brar

ies

Supp

ortiv

e in

fras

truc

ture

Awar

enes

s of M

BSE

bene

fits/

valu

eLe

ader

ship

und

erst

andi

ng o

f MBS

EVi

sion

and

stra

tegy

for M

BSE

Alig

nmen

t with

cus

tom

er re

quire

men

tsN

eed

for c

hang

eCo

mm

unic

atin

g su

cces

s sto

ries/

prac

tices

Gene

ral M

BSE

awar

enes

s and

kno

wle

dge

Peop

le w

illin

g to

use

MBS

E to

ols

Cust

omer

/sta

keho

lder

buy

-in/e

ngag

emen

tSu

cces

s met

rics

Alig

nmen

t with

bus

ines

s str

ateg

yCh

ampi

ons

Cost

to u

se M

BSE

tool

sPr

ojec

ts/p

rogr

ams t

o ap

ply

MBS

ERe

war

ds/r

ecog

nitio

nTe

amw

ork

Use

in S

E co

mm

unity

Com

mun

ity o

f pra

ctic

eLe

gacy

/cur

rent

pro

cess

esM

BSE

lear

ning

cur

veO

rgan

izatio

nal c

hara

cter

istic

sPe

ople

in S

E ro

les

Secu

rity

of d

ata

and

IP

Q29: Changes to Improve MBSE Implementation


Comparing Obstacles vs. Enablers vs. Changes

05

101520253035404550

Org

aniza

tiona

l cul

ture

Wor

kfor

ce k

now

ledg

e/sk

ills

Lead

ersh

ip su

ppor

t/co

mm

itmen

tAw

aren

ess o

f MBS

E be

nefit

s/va

lue

MBS

E to

ols

Chan

ge m

anag

emen

t pro

cess

des

ign

Inte

grat

ion

to su

ppor

t MBS

E im

plem

enta

tion

MBS

E m

etho

ds/p

roce

sses

Dem

onst

ratin

g be

nefit

s/re

sults

Trai

ning

Gene

ral r

esou

rces

for M

BSE

impl

emen

tatio

nCo

mpe

ting

prio

ritie

sPr

ojec

ts/p

rogr

ams t

o ap

ply

MBS

ELe

gacy

/cur

rent

pro

cess

esLe

ader

ship

und

erst

andi

ng o

f MBS

EGe

nera

l MBS

E aw

aren

ess a

nd k

now

ledg

eCo

st to

use

MBS

E to

ols

Supp

ortiv

e in

fras

truc

ture

MBS

E le

arni

ng c

urve

Alig

nmen

t with

cus

tom

er re

quire

men

tsCu

stom

er/s

take

hold

er b

uy-in

/eng

agem

ent

MBS

E te

rmin

olog

y/on

tolo

gy/li

brar

ies

Nee

d fo

r cha

nge

Org

aniza

tiona

l cha

ract

erist

ics

Alig

nmen

t with

bus

ines

s str

ateg

yEx

tern

al re

gula

tions

Visio

n an

d st

rate

gy fo

r MBS

ERe

war

ds/r

ecog

nitio

nSe

curit

y of

dat

a an

d IP

Peop

le w

illin

g to

use

MBS

E to

ols

Cham

pion

sPe

ople

in S

E ro

les

Com

mun

icat

ing

succ

ess s

torie

s/pr

actic

esU

se in

SE

com

mun

ityCo

mm

unity

of p

ract

ice

Team

wor

kSu

cces

s met

rics

Obstacles vs. Enablers vs. Changes


Framework for MBSE Adoption

• Insight from analysis of both obstacles and enablers, mapped to the Baldrige CPE, can be used to define a more comprehensive set of adoption practices for maturity in MBSE – e.g.:

― Leaders communicate a clear reason and need for MBSE adoption ― Leaders understand MBSE ― Leaders support and are committed to MBSE ― People understand the benefits of MBSE ― MBSE is aligned with the overall business strategy ― MBSE is used for the right projects/programs ― MBSE adoption is aligned with what customers need ― Customers and stakeholders buy-in to MBSE ― Data management processes support MBSE ― The IT infrastructure supports MBSE use ― Clear metrics are defined to track results & progress of MBSE ― Systems engineers have skills needed to support MBSE use ― Training is provided to develop needed skills ― People are rewarded/recognized for using MBSE ― The organizational culture is aligned with MBSE use

Baldrige CPE Framework

http://www.nist.gov/baldrige


Summary: Top DE Metrics Areas (numbers reflect survey citations)

Category Most Cited Benefits Quality Reduce Cost (25)

Reduce Defects/Errors/Rework (19) Increased Traceability (17) Improved System Quality (14)

Velocity/Agility Improved Consistency (34) Reduce Time (31) Improved Capacity for Reuse (30) Increased Efficiency (20) Better Collaboration/Information Sharing (8)

User Experience Improved System Understanding (28) Better Manage Complexity (9) Reduced Waste (4) Support for Automation (3)

Knowledge Transfer Better Accessibility of Information (29) Better Communication/ Information Sharing (29) Improved System Understanding (29)

Adoption Methods/Processes (45) Roles/Skills, People Willing to Use (31) Leadership support/Commitment (27) Training/Tools, People Willing to Use (21) Change Management Process Design (14)

44

Questions?

Thank you!

Benchmarking the Benefits and Current Maturity of Digital Engineering/Model-Based SEFrameworks for Evaluating DE ValueDoD Digital Engineering StrategyFocus: Enterprise Transformation MeasuresSummary DE Success Measures FrameworkMBSE Benefits: Literature Review ResultsLiterature Review Benefit Categories (48)Literature Review Quantitative Results (most cited)mbsematuritysurvey.sercuarc.orgMBSE Survey OverviewDemographics - My organization is:Demographic – type, size, experienceOverall Survey ScoresDemographics – Participant rolesOverall Survey Scores by RoleMBSE Usage and Model Management Survey ResultsTechnical Management Survey ResultsComparing Obstacles vs. Enablers in �Tech Review ProcessesDE/MBSE Metrics Survey ResultModel Quality Survey ResultsData Management Survey ResultsModel Sharing & Reuse Survey ResultData/Model Discovery & ReuseModeling Environment Survey ResultsWorkforce and Skills Survey ResultsSurvey Results of Critical SkillsSurvey Results of Critical SkillsSlide Number 28List of DE/MBSE Benefit Categories �(Literature Review)Most Frequently-Reported Benefits from Survey Analysis by QuestionTop Benefits Reported from Survey QuestionsAll Benefit Categories Reported from Survey QuestionsAnalysis of Survey Question on MetricsMost cited benefits and metrics categories from survey dataMost cited benefits and metrics categories from survey dataChallenges in Enterprise Adoption of MBSEList of enterprise success factors from the survey analysisAnalysis of Survey Responses: ObstaclesAnalysis of Survey Responses: EnablersAnalysis of Survey Responses: ChangesComparing Obstacles vs. Enablers vs. ChangesFramework for MBSE AdoptionSummary: Top DE Metrics Areas�(numbers reflect survey citations)Questions?

Documents

Benchmarking the Benefits and Current Maturity of …...2020/03/26 · • DoD Digital Engineering Strategy ―Vision, mission, and goals can be generalized to any organization •