Using System Archetypes to Identify Failure Patterns in ... · What is "Systems Thinking"? • A new perspective which provides an understanding that comes from focusing on the big

page 1

Sponsored by the U.S. Department of Defense © 2005 by Carnegie Mellon University

Diane GibsonLinda Levine, PhDWilliam E. Novak

May 2, 2006

Using SystemArchetypes toIdentify FailurePatterns inAcquisition

© 2006 by Carnegie Mellon University

Overview

Introduction to Systems Thinking

Applying Archetypes to Acquisition:The Far Orbit Satellite System (FOSS)

Conclusions and Future Directions

© 2006 by Carnegie Mellon University 3

What is "Systems Thinking"?• A new perspective which provides an understanding that comes from

focusing on the big picture and the interactions internal and externalto it instead of dividing up the picture into smaller elements.

For example -- if you look at the body of an aardvark, you see one picture if you killit, cut it up and examine the parts; you see another if you look at the animal as oneentity, and examine how it interacts with different levels of the environment in whichit lives (or if you study the interactions of the organs which make up the systemcalled "aardvark".)

• it involves thinking in 'circles' rather than straight lines• cause and effect are interactive; what

is a cause one time is an effect another


What is a System?System = a collection of elements where

• the whole is bigger than the sum of the parts

• everything is interconnected & effects everything else

• there is a purpose & the elements are arranged meet thispurpose

• the interactions of elements are characterized by reciprocalflows of influence

• this is also what causes the system to change

• the same action may have different effects, different actionsmay have the same effects, & delays often lead tounexpected outcomes

• cause & effect may be far apart in time & space


Causal Loop Diagrams - Language of Systems

• Consist of lines with arrows, drawn in circular format, to showcause-effect (influence) relationships

• The direction of the relationship is labeled (S)ame or(O)pposite, based on the direction of the causation.

• Loops are labeled either� (R)einforcing - to change by making stronger/weaker; created with an even #

of opposites.� or (B)alancing - equilibrium achieved between contrasting or opposing

elements; created with an odd # of opposites.

• Time delays are important.

• Additional loops can beadded to show sideeffects or otherconsequences.

Variable

Variable

Variable

Delay

Variable

S S

O

S

S

B2R1


Systems Archetypes

• Story, diagram, template and/or graph

• that describes and depicts specific patterns

• which reveal the underlying system structure and associatedproblems and mental models

• which facilitates addressing the underlying causes of behavior

• which are common across many organizations


Using Archetypes

• Viewpoints (Lenses)

• Examine problem structure (Patterns)

• Explain dynamic events

• Forecast potential behaviors

• Make changes to a system

• Present information about problems & solutions


Different Archetypes

• Shifting the Burden

• Drifting Goals

• Limits to Success/Growth


Shifting the Burden - StoryWhen a quick fix is applied to a problem, and it seems to work,that is the end of further investigation.

THE REAL PROBLEM...

However, the quick fix• leads to side effects, which may not be recognized as related

to the original problem;• consumes resources, reducing the ability of the organization

to address other causes and ultimately, to implement afundamental solution;

• and since the fundamental problem isn’t addressed, it returns,often in a more virulent form.


ShiftingThe

Burden

CausalLoop

Diagram

SymptomaticSolution

FundamentalSolution

Side Effect

ProblemSymptom

B1

B2

R3

S

S

O

S

O

O


Shifting The BurdenComments like the following should give you a clue that you may beexperiencing a shifting the burden pattern..

• Just once more won’t do any harm.

• The next time this happens we’ll look at it afresh.

• I wish we could stop this, it’s not doing any good.

• When this emergency ends, we’ll return to our normal standards.

• I know it doesn’t help in the long run, but what can we do?

• It’s not my responsibility!


Shifting the Burden – Interventions

• Identify problem symptoms, side effects and the quick fixes.

• Focus on the fundamental solution; maintain the symptomaticsolution while working on a fundamental solution.

• Examine the impact of solutions on different groups in theorganization. Gather multiple viewpoints to distinguishbetween the symptomatic and fundamental solutions.

• With an addiction pattern, discover what in the basic structureof the system contributed to making the fix an addiction.

• Use the archetype to explore potential side effects of anyproposed solution.


Drifting Goals – StoryA person or organization seeks to achieve a goal. But, it will takea lot of time and money to do so.So, they decided to reduce the goal, just a little...Voila! The difference quickly goes away with little delay or cost!

THE REAL PROBLEM?

Reducing the goal may continue and become a habit.

Someone must acknowledge that a problem exists or the goal maycontinue to erode without recognition.

NOTE: sometimes the goal may actually be too high. You may need to take the time to evaluate.


DriftingGoals

CausalLoop

Diagram

Pressure toLower Goal

ActualValue

Gap

B1

B2

S

S

O

O

Goal

S

Corrective Action

S

Delay


Limits to Success – Story

People engage in certain activities, leading to success. As effortsincrease, success increases. The growth engine works.After time, growth slows down. The reaction – redouble efforts,and do more of the same.Paradoxically, as you push harder, the system pushes back.

THE REAL PROBLEM...

Over time, limiting forces, intimately connected to the growthengine, come into play. As they begin to dominate, growth andperformance improvements stabilize or decline.Doing what initially worked creates its opposite. Torestart growth, a way must be found to deal with thelimiting forces.


Limits to Success –

Causal Loop Diagram

LimitingAction

B2S

S

O

Effort Performance

S

R1

Constraints

S


Overview





FOSS Program Case Study -1

Far Orbit Satellite System

Satellite space surveillance ground segment – cost of $2.1B($1.8B with a $350M award fee)

A Cost-Plus EMD contract awarded to Genadyne, teamed withAerocorp.

Increment 1 was due in September 1999, Increment 2 was duein October 2002, and Increment 3 due in 2009. The 24 yearlifecycle cost was estimated at $10B.

Per acquisition reform, the program embraced TSPR, usedperformance-based requirements, had minimal government SPOstaff (since contractor carried the risk), and relied on contractordata and metrics.

The program planned to use spiral development, software reuse,and open systems architecture for modular updates & COTS use.

Nov ‘96


Underbidding the Contract

“The SPO and the Office of the Secretary of Defense (OSD) hadestimated FOSS EMD costs to be $3.2 billion. The contactor hadbid $1.8 billion with a $347 million award fee pool for a totalcontract award of $2.1 billion. The large gap between theGovernment estimate and the contractor’s bid raised someconcern…”

“The government model estimated that the software could bebuilt at X rate and the contractor maintained that they could buildit at 3X rate. This was not resolved at source selection. Since theTSPR paradigm was in full ascendancy, it was decided to let thecontractor do the work with their assumption.”


ContractRevenue

Need

SAccuratelyBid Contracts

O

B2

UnderbidContracts

S SContract

Wins

S

Slips,Overruns,Shortfalls

S

IncrementalFunding

B1

O

Losses byAccurately

BiddingContractors

S

O

R

Delay

Based on “Shifting the Burden”

Underbidding the Contract - Diagram



Aerocorp was given $105M for Increment 1 work.

A COTS vendor went out of business forcing a replan, but thegovernment still paid the front-loaded award fees.

Increment 1 was on a very tight schedule. It was assessed to beat high schedule risk, although only low technical risk.

Jul ‘97


Sacrifice Quality

“The government model estimated that the software could be builtat X rate and the contractor maintained that they could build it at3X rate... The metric was lines of code. Unfortunately, lines ofcode does not mean lines of good code… they had a quota theywere reporting on the lines of code they had achieved. At the lowerlevels, the engineers were writing code, but all of the qualitychecks were falling off the table. Things like peer code checkswere not happening.”

“Some SPO technical staff reviewed contractor software tests andbecame aware of dubious practices of ‘passing’ test milestoneseven though the tests had revealed significant software problems.Basically, the test failures were noted and the corrections weredeferred for future tests.”


Sacrifice Quality - Diagram

Cost andSchedulePressure

GapBetween

Desired andActualQuality

S

ProductQuality

Standard

O

B2

O

Pressure toInvest inQuality

S

ActualProductQuality

S

Delay

Based on “Drifting Goals”

Pressure toReduceQuality

S

B1

S


The Bow Wave Effect

“Another Increment 1 software issue involved spiral development.Aerocorp software engineers would claim that they were meetingmilestone events, even though content was being shifted fromone spiral to the next. Aerocorp said that it was appropriateunder the spiral paradigm to ‘pick the low-hanging fruit first’ anddefer difficult requirements. They said this practice insuredsuccessive spirals could be implemented and tested promptly,and that valuable test results could be fed back into the nextspiral.”


The Bow Wave Effect - Diagram

SchedulePressure

SystemRisk

S

OtherDesign

DecisionsMade

S

B1

O

Developmentof Simpler

FunctionalityS

R1

O

SystemModifiability

O

Ability toIntegrate

NewCapability

S

S

R2

Based on “Shifting the Burden”

O

B2

SDevelopmentof Complex

Functionality

Delay



The problems with passing tests continued to increase, but thecontractor said they were normal.

The contractor found that the reusable software couldn’t be used.

Jan ‘99

Jun ‘99


Firefighting

“When Aerocorp began system testing in the summer of 1999,they plunged into a persistent test-fix-test cycle. It wasimpossible to verify system performance because the systemwould not run long enough before crashing. Instead, the testingbecame part of continued system development. The systemarchitecture would not support performance requirements; thetiming of system events was off. …it had both softwareintegration and hardware-software integration problems.”


Firefighting - Diagram

“Firefighting” concept from “Past the Tipping Point”

SchedulePressure

ReworkS

O

B

AvailableResources

O

QualityO

Errors

O

O

R



Increment 1, due in September 1999, was far from done due toarchitecture, design, and process issues.

The SPD proposed declaring a breach against the 1999 delivery,but the PEO agreed with the contractor saying, "It could stillhappen”—but Increment 1 failed repeatedly during OT&E testing.

FOSS declared an Acquisition Program Baseline (APB) breach.They brought in a Management Assessment Team (MAT) forrecovery planning. The MAT concluded that management anddevelopment processes were inadequate, Increment 1 had hightechnical risk, the design should be reengineered, and theprogram would slip 2 years.

The SPD restructured FOSS to MAT recommendations. Theyassessed the contractor $35M for late delivery. The SPOassigned a colonel to closely monitor Increment 1.

Aug ‘99

Oct ‘99

Jul ‘00



In IBR the contractor estimated $500M funding shortfall due toMAT restructuring, replanning, improved testing. The SPD gotthat reduced to a more acceptable $277M, and the AF CAIGvalidated that.

OSD CAIG estimated a $510M shortfall, not $277M, so OSDincreased funding by $484M.

A payload subsystem failure occurred requiring significantredesign, and contractor requested $125M more.

The new SPD calculated an EAC of $318M over the currentbudget and a 2-year slip.

Jul ‘00

Nov ‘00

May ‘01

Jul ‘01


Don’t Bring Me Bad News

“The Genadyne VP [said they] would need $125 million more inFY02… Disputing the… Genadyne Vice President, Blake told Mr.Collins that with aggressive cost controls, the $45 million wouldsuffice. Blake then retired.”

“Broderick passed the bad news informally up his chain of command.His superiors warned him that cost overruns of the scale he wassuggesting might cause program cancellation. They urged cautionand deliberation…

“Col. Broderick wanted to get the bad news out, and again told the AirForce leadership in the Pentagon that FOSS was at least $1 billionshort. He figured this would be a ‘significant emotional event’ for Mr.Collins...”

“Later that same day, Mr. Collins was in Lt Gen Ashton’s office andplaced a speakerphone call to Col Broderick. Col Broderick recalledtheir voices as very stressed. Again the issue was about reducing thebill. Broderick said, ‘I can’t do that.’”


# of Issueswith

System

S

# ofResolved

Issues

# ofReported

Issues

S

OB1

“PunishedMessengers”

S

Fear ofReporting

Issues

SB2

O

Don’t Bring Me Bad News - Diagram



SPD met with with the Air Force Deputy Assistant Secretary ofAcquisition & Management and the contractor VP about theshortfall, and the VP said the bad news was premature, but hadno alternate estimate. The Deputy Assistant Secretary told theAir Force leadership of the shortfall.

FOSS initiated a Nunn-McCurdy breach acknowledging a morethan 25% cost overrun.

FOSS Increment 1 finally passed OT&E, 2 years late.

Oct ‘01

Nov ‘01

Dec ‘01


Overview





Systems Thoughts

"Humankind has succeeded over time in conquering the physicalworld and in developing scientific knowledge by adopting an analyticalmethod to understand problems ... this sort of linear and mechanisticthinking is becoming increasingly ineffective to address modernproblems... because, today, most important issues are interrelated inways that defy linear causation."

"Systems thinking allows people to make their understanding of socialsystems explicit and improve them in the same way that people canuse engineering principles to make explicit and improve theirunderstanding of mechanical systems."

Daniel Aronson


Observations -1

• Not all problems fit an archetype

• Non-linear nature of systems thinking

• People in the same structure tend to behave in similar ways


Observations -2

Short-Term Thinking – Fixing Symptoms instead of Problems• Penchant for the “quick fix” and “low-hanging fruit”• Unwillingness to invest in long-term solutions addressing root

cause

The Role of Intent and Motive• Intent isn’t relevant—only behavior matters

Cross-Program Archetypes and Dynamic Games• Learning from past actions and outcomes of analogous

situations


Next Steps/Future Directions

Integration of Archetypes• Link related archetypes to build more complete models

System Dynamics Modelling• Inputs to system dynamics simulations of acquisition

High-leverage Interventions• Provide early warning of developing failure patterns

“Big Picture” Thinking• Improve decision-making by avoiding oversimplification


Additional Information

System Archetype Basics: From Story to Structure, Daniel H. Kim andVirginia Anderson, Pegasus Communications, Waltham, MA, 1998.

“Large-Scale Projects as Complex Systems: Managing ‘Scope Creep,’”Andrea Shapiro and Carol Lorenz, The Systems Thinker, vol. 11, num.1, February 2000, Pegasus Communications.

Far Orbit Satellite System (FOSS), Case Study, Dr. Christopher Roman,Defense Acquisition University (DAU), Ira Monarch, SEI.

Defense Acquisition Performance Assessment Report, LtGen. RonaldKadish, USAF (Ret) Panel Chairman, Defense Acquisition PerformanceAssessment Project Assessment Panel, January 2006.


Contact Information

Diane GibsonSenior Member of Technical StaffSoftware Engineering Process Management Program(412) [email protected]

Bill NovakSenior Member of Technical StaffAcquisition Support Program(412) [email protected]

Software Engineering InstituteCarnegie Mellon UniversityPittsburgh, PA 15213-3890

Documents

Using System Archetypes to Identify Failure Patterns in ... · What is "Systems Thinking"? • A new perspective which provides an understanding that comes from focusing on the big