© 2003 by Carnegie Mellon University page 1 Processes for COTS-Based Systems LA SPIN 3 September 2003 Software Engineering Institute Carnegie Mellon University

© 2003 by Carnegie Mellon University page 1

Processes for COTS-Based Systems

LA SPIN3 September 2003

Software Engineering InstituteCarnegie Mellon UniversityPittsburgh PA 15213

Sponsored by the U.S. Department of Defense

Tricia Oberndorf


Outline

• Background

• Process Overview

• High-level Process

• Example Low-level Activities Descriptions

• Converting This to a Process: EPIC


Background

Building systems today:• Few custom-built to order• Commercial products expected to play major role

Diverse things influence the shape and function of a COTS-based system (CBS):• Stakeholder needs• Characteristics of products and marketplace• Degree of interaction with legacy systems

Together, these compel many changes in system development and maintenance methods for CBSs.


COTS Spheres of Influence

VendorsMarket segmentsAvailable COTS ProductsEmerging technology

MarketplaceMarketplace

Use casesQuality attributesBusiness model

Stakeholder Needs/ Business Processes

Stakeholder Needs/ Business Processes

PatternsInterfacesInfrastructure

Architecture/ Design


Risk managementProject management

Change managementLicense negotiation

Programmatics/Risk

COSTS

Programmatics/Risk

COSTS


Marketplace Affects CBS Approach

Traditional Engineering Approach

Requirements

Architecture & Design

Implementation

Requirements-driven Negotiation-driven

Required CBS Approach

Simultaneous Definition

and TradeoffsMarketplace

Stakeholder Needs/Business Processes

Architecture Design

Programmatics/Risk


Conceptual Bases: Requirements1

“Nail down the requirements first” will not work – the marketplace will not cooperate• Think of it as a new sphere of influence:

Constraints come from stakeholder needsAND marketplace imperatives

System that is created mustaccommodate competing sources of gravity


Conceptual Bases: Requirements2

Requirements (cont.)• Must distinguish between negotiable and non-

negotiable• Keep the non-negotiable set as small as possible• Understand how to prioritize and trade-off the

negotiables

A risk-driven spiral approach is key:• Frequent use of prototypes• Considerable interaction with system’s end users• Gradual refinement of understanding of system


Conceptual Bases: Knowledge

CBS development and maintenance is dependent on an evolving Body of Knowledge.

Architecture anddesign constraints

End-userbusiness processes

Acquisitionconstraints

BoK

Prioritized negotiables Marketplace offerings

Evolving System View

Non-negotiables

Legacy system context

Risks


Process Overview

Three classes of activities:• Iterative: short-term, engineering-oriented

- Discovery: gather and refine system knowledge- Assembly: construct a prototype- Assessment: determine success of iteration & plan

• Pervasive: long-term, organizational scope- e.g., CM, license management, vendor relationship

management, contract tracking and oversight• Executive: event-driven, deal with decision making

- e.g., cost estimating, contract negotiation, project oversight

Today we focus on the Iterative.


Discovery

Gather and Refine activities occur simultaneously.

Gather:• Sources take many forms.

- Stakeholders, business processes, legacy systems, COTS marketplace

• Each is independent of the others.• There is no optimal order.

Refine:• Analysis and harmonization of the gathered knowledge• Yields the technical definition of the emerging system• Finds gaps, negotiates conflicts


Discovery Activities

Data from: stakeholders products …

RefineGather

Negotiate conflicts

Gap - need more data

Continue Discovery

Knowledge is sufficient for constructing executable

Go on?

Harmonized data

Mismatch

Agreement

Gather data zGather data y

Gather data xAnalyze


Assembly

Produces a prototype that reflects what’s been discovered

Reflects a traditional sequential process• Guided by local “ candidate requirements”

- Iteration Objectives, Detailed Iteration Plan plus hypotheses and desired behaviors expect to derive from prototype

- Vets “candidate requirements”• Produces an executable version of full system

- Likely to have less than complete functionality- Executable, not paper or specification or small piece

Does not preclude prototyping “in the small” throughout


Evaluation and Assessment

Occur at two levels:• Evaluation of the prototype in its own context

- Measures actual outcome against expected outcome- Considers degree to which prototype satisfies its

local “requirements”• Assessment of the iteration itself

- Addresses issues at project (not iteration) level- Considers whether objectives were good ones- Determines whether iteration results indicate

changes in project schedule, budget, etc.


On-going Iterations of A/APCS

Prototype

Deployed System

Body of Knowledge

Body ofKnowledge

BoK


Top-Level View of A/APCS

Manage Program: based on global program constraints, generates iteration objectives and fielding decisions and forwards the current System View to the next iteration

Perform Iteration: based on iteration objectives, the market-place, and current System View, generates a new version of the system, evaluates it, and updates the new System View

Assess Iteration: based on iteration objectives and the evaluation of the system, assess the overall iteration


Detail of Perform Iteration

Manage Program

Assess Iteration

PlanIteration

1

ConstructSystem

2

Evaluate System

3

Perform Iteration*


Detail of Construct SystemDetailed iteration plan

“Go” decisionfor Assembly

System view (defined in this iteration)

Discovery activities

System (for

evaluation and potential deployment)

Consolidateddata to analyze

Description of needed knowledgeGather

Data2.1

RefineData

2.2

AssembleCandidateSystem 2.3

*System view (defined in previous iteration)

Replanning needed

COTS products


Low-level Activities of Refine Data

Description of needed knowledge

System View (defined in previous iterations)

Emerging revisions to System View

Additional knowledge

neededGaps

Resolutions

Resolutions & Gaps

Data to analyze

Detailed iteration plan

Agreements

(to all)

System view

Agreements, Gaps &Conflicts

Conflicts

Gaps

Assembly “Go”

Replanning needed

Analyze Data

Negotiate Conflicts

Form System View

Determine Data to be Gathered

2.2.1

2.2.2

2.2.3

2.2.4


Low-level Activity Descriptions

Construct System (Step 2)2.1 Gather Data:Gather Stakeholder InputsGather COTS Product Data Gather Business Process DataGather External System Data

2.2 Refine Data: Analyze DataNegotiate ConflictsDetermine Data to be GatheredForm System View

2.3 Assemble Candidate System:Create Detailed DesignPerform Needed ModificationsWrite Custom CodeIntegrate ComponentsTest


Example: Gather COTS Product DataPurpose: Gather product data by evaluating COTS productsRole: Evaluators, stakeholdersInputs/Entry Conditions:Description of the kind of product data that is needed, including the level of detail needed as well as gaps that must be filled. This description forms the requirements for the product evaluation.Outputs/Exit Conditions: Product data to the requisite level of detail, consolidated and formattedDescription:This activity consists of methodically evaluating one or more COTS products. As with all of the “gather” activities, the input is the Need for Knowledge generated by the Refine Data activity. Appropriate evaluation criteria for this product data gathering effort are established; these then provide the basis for the quantification methods that guide the data collection effort.


Example (cont.)Notes on Planning: …

Actions:1. Define criteria2. Prioritize criteria3. Prepare for evaluation4. Collect data from the sources5. Record results6. Consolidate and format data (described in section 7.1.5) Notes on the Actions:1. A criterion consists of both a capability statement …2. It is essential to rank criteria by importance. …3. One necessary preparation is to assemble the needed …4. Different data collection techniques may be needed. …5. The raw scores earned by each of the examined products …6. Product evaluation data is often diverse and …


Converting This to a Process

While developing this framework, we engaged with a customer who needed a process PDQ.

We took these principles and, using RUP as a backbone, turned them into a process.

Original name: ITSEP

New name: Evolutionary Process for Integrating COTS-based systems (EPIC)• A negotiation-driven process that helps identify and

manage the differences between what users want and what COTS products can deliver

• Commercial and government EPIC pilots underway


EPIC Conceptual Framework

TimeTrade Space


and Tradeoffs

Marketplace

Stakeholder Needs/Business Processes


Programmatics/Risk

• Trades are negotiation-driven with knowledge of marketplace • Requirements formed based on knowledge of market/architecture • Continuous awareness of end-user process changes• Project business and contractual approaches support trades

Trade Space

Decisions

Converge


Knowledge Grows Incrementally

• Risk-based spiral development focuses and integrates diverse information

- Prioritized stakeholder needs, end-user processes- Organization and system architecture, design constraints- Identified risks, programmatic constraints- Marketplace offerings, product characteristics, other buyer

usage

• Frequent, evolving executable representations demonstrate current understanding

Executable

Time


Continuous Stakeholder Negotiation

• Quick resolution to discovered mismatches- Business process owners and end users- System engineers and developers- Vendors and suppliers- Project managers- Change agents

Time

Stakeholder Buy-in Increases

• Stakeholder needs mature with increased understanding of marketplace implications

• Business processes change to leverage available products• End users committed to system solution


EPIC: A Negotiation-Driven Approach

Drive strategic vision to

sustainable solution

Time

Solu

tio

n A

ltern

atives Stakeholder needs

Business processes

ProgrammaticsRisk


and TradeoffsMarketplace Architecture

Design

Time

Solu

tio

n A

ltern

atives Stakeholder needs

Business processes

ProgrammaticsRisk


and TradeoffsMarketplace Architecture

Design

Increase stakeholder buy-in and reconcile end-user processes with

COTS-based system

Accumulate knowledge through risk-based spirals

Coordinates operational change, system & software engineering, and project management to field initial capability in 6-18 months


Iterations Redefined for CBS

Assess iteration

Gather information

SimultaneousDefinition

and Tradeoffs

(1 to 8 weeks per outer cycle)

Plan iterationAssemble executable representation

Executable

Refine into harmonized set

by identifying and analyzing mismatches and negotiating among stakeholders


Phases Bounded by Anchor Points

LifeCycle Architecture

Initial Operational Capability

Elaboration Construction Transition

Refine, experiment & select solution

Try/select COTS

Prototype business process changes

Implement selected solution

Apply/track COTS

Prepare to change business process

Field and support solution

Track/update COTS

Change business process

… … …

LifeCycle Objectives

Inception

Gather and define project scope

Survey/try COTS

Agree to business process changes

…Plan iteration

Gather information

Assess iteration


ExecutableExecutable

Programmatics/Risk

Business Processes

Architecture/ •Design

Marketplace

Stakeholder needs/


and Tradeoffs

Assemble executablePlan iteration

Gather information

Assess iteration



Programmatics/Risk

Business Processes


Marketplace

Stakeholder needs/


and Tradeoffs


Gather information

Assess iteration



Programmatics/Risk

Business Processes


Marketplace

Stakeholder needs/


and Tradeoffs


Gather information

Assess iteration



Programmatics/Risk

Business Processes


Marketplace

Stakeholder needs/


and Tradeoffs


Gather information

Assess iteration



Programmatics/Risk

Business Processes


Marketplace

Stakeholder needs/


and Tradeoffs


Gather information

Assess iteration



Programmatics/Risk

Business Processes


Marketplace

Stakeholder needs/


and Tradeoffs


Gather information

Assess iteration



Programmatics/Risk

Business Processes


Marketplace

Stakeholder needs/


and Tradeoffs


Gather information

Assess iteration



Programmatics/Risk

Business Processes


Marketplace

Stakeholder needs/


and Tradeoffs

Assemble executable

6 to 18 months


Inception Phase: Goal

Achieve concurrence among stakeholders on project’s life-cycle objectives

• Match segments of marketplace with critical use cases• Identify implications of potential business process

changes• Establish acquisition strategy for appropriate vendor

relationships

Establish feasibility for the project through the business case that shows there is one or more candidate solutions

Survey & Try COTS


Inception Phase: Activities

• Gather high-level information (critical use cases)- “Must have” stakeholder needs and required

business processes- Market drivers, technologies, products- Architectural constraints and design alternatives- Programmatic constraints and risks- Incentives /inhibitors to business process changes

• Refine information to form high-level candidate solutions

• Assemble executable representation(s)

Plan iteration

Gather information

Assess iteration



Programmatics/Risk

Business Processes


Marketplace

Stakeholder needs/


and Tradeoffs

Assemble executable


Elaboration Phase: Goal

Define a high-fidelity solution with predictable cost/schedule

• Select and acquire COTS products • Business process implementation supported by all user

roles

Achieve sufficient stability of the solution across architecture, requirements, and marketplace

Mitigate development and rollout risks

Try & Select COTS


Elaboration Phase: Activities• Gather detailed knowledge Refine knowledge

to form a stable baseline- Gaps/mismatches identified and negotiated- COTS integration mechanisms defined and

validated - Business process changes implementation

refined- Balanced across all 4 spheres

• Assemble architectural prototypes- Business process changes prototyped

...

Amplify each candidate solution

Amplify selected solution

Plan iteration

Gather information

Assess iteration



Programmatics/Risk

Business Processes


Marketplace

Stakeholder needs/


and Tradeoffs

Assemble executable

Plan iteration

Gather information

Assess iteration



Programmatics/Risk

Business Processes


Marketplace

Stakeholder needs/


and Tradeoffs

Assemble executable


Construction Phase: Goal

Achieve a product quality release ready for the user community

Prepare functional units for needed business process changes• Organization changes• Training

Manage vendor relationships

Balance development stability and potential obsolescence with marketplace volatility

Apply & Track COTS


Construction Phase: Activities

• Gather marketplace information continually

• Refine selected solution as necessary

• Assemble selected solution for fielding- Develop any custom components- Tailor products (non-source code

customizations) - Develop production quality COTS integration

code/data sets- Implement business process changes for initial

fielding

Plan iteration

Gather information

Assess iteration



Programmatics/Risk

Business Processes


Marketplace

Stakeholder needs/


and Tradeoffs

Assemble executable


Transition Phase: Goal

Roll out and support selected solution set to the user community• Initial fielding (beta testing)• Full fielding• On-going support until solution release retired

Implement business process changes across user community

Balance operational stability with marketplace volatility

Manage relationships with vendors

Track & Update COTS


Transition Phase: Activities

• Gather marketplace information continually

• Refine solution to accommodate the impact of new products, releases, or technology

• Assemble maintenance releases- Re-tailor products (non-source code

customizations)- Re-develop COTS integration code/data sets- Re-integrate and re-test

Plan iteration

Gather information

Assess iteration



Programmatics/Risk

Business Processes


Marketplace

Stakeholder needs/


and Tradeoffs

Assemble executable


Closing

This presentation has discussed:• a framework for COTS-based systems processes• a specific process based in that framework

The principles are broadly applicable, and you are invited to try them out.


For More Information

David J. CarneySEIPittsburgh, [email protected]

Tricia OberndorfSEIPittsburgh, [email protected]

Patrick PlaceSEIPittsburgh, [email protected]

Ceci AlbertSEIWashington, [email protected]

Lisa BrownswordSEIWashington, [email protected]


Acronyms

A/APCS Assembly/Acquisition Process for COTS-based Systems

CBS COTS-based systemCM configuration managementCOTS commercial off-the-shelfEPIC Evolutionary Process for Integrating COTS-

based systems ITSEP Integrating Technology by a Structured

Evolutionary ProcessSEI Software Engineering Institute


BACKUPS


Guiding Principles1

• The requirements process must celebrate flexibility: requirements definition must be delayed and negotiated, and the true requirements minimized.

• Use of any COTS product necessitates documenting, and probably re-engineering of existing end-user process.

• Any COTS–based system will be in a necessary state of continual system re-formation and evolution throughout its useful lifetime.


Guiding Principles2

• Hands-on product evaluations are mandatory; these must be budgeted, scheduled, and accepted as a fundamental project activity, as central as requirements and design.

• Prototyping of the integrated collection of COTS products from the earliest possible moment throughout system lifetime is a basic necessity.

• Satisfactory contractual commitments can only result from the active participation of end users, testers, and other stakeholders, continuously throughout the entire period from project inception through sustainment and maintenance.

Documents

© 2003 by Carnegie Mellon University page 1 Processes for COTS-Based Systems LA SPIN 3 September 2003 Software Engineering Institute Carnegie Mellon University