View
216
Download
3
Tags:
Embed Size (px)
Citation preview
When Do You Need Systems of When Do You Need Systems of Systems Engineering: Systems Engineering: A Quantitative AnalysisA Quantitative Analysis
Jo Ann Lane17 March 2009
University of Southern CaliforniaCenter for Systems and Software Engineering
March 2009 USC CSSE Annual Research Review 2
OverviewOverview
• Key definitions
• Scope of research
• Methodology
• Model implementation
• Results of research
• Conclusions and future work
March 2009 USC CSSE Annual Research Review 3
What is a “System of Systems”?What is a “System of Systems”?
• Very large systems developed by creating a framework or architecture to integrate constituent systems
• SoS constituent systems independently developed and managed– New or existing systems in various stages of development/evolution– May include a significant number of COTS products– Have their own purpose– Can dynamically come and go from SoS
• SoS exhibits emergent behavior not otherwise achievable by component systems
• Typical domains– Business: Enterprise-wide and cross-enterprise integration to support
core business enterprise operations across functional and geographical areas
– Military: Dynamic communications infrastructure to support operations in a constantly changing, sometimes adversarial, environment
Based on Mark Maier’s SoS definition [Maier, 1998]
March 2009 USC CSSE Annual Research Review 4
Types of SoSTypes of SoS• Virtual [Maier, 1998]
– Lacks a central management authority and a clear SoS purpose– Often ad hoc and may use a service-oriented architecture where
the constituent systems are not necessarily known
• Collaborative [Maier, 1998]– Constituent system engineering teams work together more or
less voluntarily to fulfill agreed upon central purposes – No SoSE team to guide or manage activities of constituent
systems
• Acknowledged [Dahmann, 2008]– Have recognized objectives, a designated manager, and
resources at the SoS level (SoSE team)– Constituent systems maintain their independent ownership,
objectives, funding, and development approaches
• Directed [Maier, 2008]– SoS centrally managed by a government, corporate, or Lead
System Integrator (LSI) and built to fulfill specific purposes– Constituent systems maintain ability to operate independently,
but evolution subordinated to centrally managed purpose
This research focused on identifying the “home-ground” for these two types of SoSs...
March 2009 USC CSSE Annual Research Review 5
Scope of ResearchScope of Research
• Research question – When is it cost effective to establish and use a system of
systems engineering (SoSE) team to oversee and guide the evolution of a system of systems (SoS)?
• Hypothesis – There exists a threshold where it is more cost effective to
manage and engineer capability changes to an SoS using an SoSE team and this threshold can be determined by modeling the SoS system complexity and desired capability interdependency characteristics.
Focus is on software-intensive SoSs owned by the United States Department of Defense (DoD)...
March 2009 USC CSSE Annual Research Review 6
Statement of Topic and Contribution Statement of Topic and Contribution (continued)(continued)
• Research contribution – Provides guidance to DoD leadership with respect to the
management of sets of inter-related systems that are functioning as a system of systems.
– Guidance also applies to SoSs in other domains that are managed as collaborative or acknowledged SoSs
– Model for management and engineering guidance also provides• A method for conducting trade-off analyses for different approaches
for implementing a given SoS capability for a given SoS
• A model that can evolve into an SoSE cost model through calibration for a given SoS or SoS domain
• A cost model that can better model complex systems
March 2009 USC CSSE Annual Research Review 7
MethodologyMethodology• Using COSYSMO, developed a process model that can
compare the SoS management strategies as SoS characteristics are varied– SoS size (number of constituent systems)
– Size of SoS capability (number of equivalent nominal requirements)
– Scope of SoS capability (number of constituent systems affected by SoS capability)
– Constituent system volatility (level of constituent system change being engineered at the same time as SoS capability)
• Process model based on data from– 18 large-scale DoD SoS programs
– 16 DoD systems that participate as constituent systems in one or more SoSs
• Analyze model outputs to determine under what conditions an SoSE team is cost effective
March 2009 USC CSSE Annual Research Review 8
SoSE Process Model OverviewSoSE Process Model Overview• Purpose
– Estimate and compare the effort required to implement an SoS capability using two different management approaches
• Collaborative (no SoSE team)• Acknowledged (SoSE with limited authority/control)
• Assumptions and constraints– All constituent systems currently exist and have their own evolutionary paths
based on system-level stakeholder needs/desires– Model assumes SoSE and traditional SE teams are using relatively mature
processes– SoS capabilities are software-intensive– No SoS capability/requirements volatility– SoS internal volatility represented by constituent system volatility– No accommodation of schedule factors or the asynchronous nature of SoS
constituent system upgrades– Management of SoS internal interfaces reduces complexity for systems
March 2009 USC CSSE Annual Research Review 9
Systems Engineering Requirements CategoriesSystems Engineering Requirements Categories
• Requirements related to SoS capabilitiesa) Acknowledged SoS: Initially engineered at SoS level by SoSE
team with support from constituent system engineers for those systems impacted by the SoS capability, then allocated to constituent systems for further SE
b) Collaborative SoS: Not engineered at the SoS level, but must be engineered fully at the constituent system level through collaborative efforts with other constituent system engineers
• Non-SoS requirements related to constituent system stakeholder needs– Must be monitored by SoSE team to identify changes that might
adversely impact SoS– Represents on-going volatility at the constituent system level that
is occurring in parallel with SoS capability changes
March 2009 USC CSSE Annual Research Review 10
SoSE Model StructureSoSE Model Structure
Focus is on software-intensive SoSs owned by the US DoD, the number and volatility of constituent systems within an SoS, and the complexity of typical capability enhancements to the SoS...
March 2009 USC CSSE Annual Research Review 11
System Capability
Effort for a “collaborative”
SoS
Effort using an “acknowledged”
SoSE teamEquivalent set of
“sea-level” requirements
Conversion to COSYSMO size units
Calculations based on SoS characteristics/size and capability implementation approach using
COSYSMO algorithm
Overview of SoSE SDM FlowOverview of SoSE SDM Flow
March 2009 USC CSSE Annual Research Review 12
Model Parameters by SDM ConstructModel Parameters by SDM Construct
• Stocks– Inputs
• SoS Equivalent Requirements– Outputs
• SoSE Effort• SoS Upgrade Effort with
SoSE• SoS Upgrade Effort without
SoSE
• Flows– Capability Rate– SoSE Effort Rate– SE Effort Rate with SoSE– SE Effort Rate without SoSE
• Converter Parameters– COSYSMO effort multipliers
• COSYSMO SoSE EM• COSYSMO SE EM with
SoSE• COSYSMO SE EM without
SoSE• COSYSMO SE EM
– SoS complexity factors• Number of systems in SoS• Number of systems affected
by capability• Average system rate of
change
General Form of COSYSMO EquationGeneral Form of COSYSMO Equation Effort (person months) = [38.55 * EM * (size)1.06] / 152
March 2009 USC CSSE Annual Research Review 13
SoSE Effort MultiplierSoSE Effort Multiplier
2.50
March 2009 USC CSSE Annual Research Review 14
Effort Multiplier for SoSE Monitoring of Effort Multiplier for SoSE Monitoring of Constituent System RequirementsConstituent System Requirements
0.47
March 2009 USC CSSE Annual Research Review 15
SE Effort Multiplier for SoS Requirements SE Effort Multiplier for SoS Requirements withwith SoSE Support SoSE Support
1.06
March 2009 USC CSSE Annual Research Review 16
SE Effort Multiplier SoS Requirements SE Effort Multiplier SoS Requirements withoutwithout SoSE Support SoSE Support
1.79
March 2009 USC CSSE Annual Research Review 17
SE Effort Multiplier for System-Specific SE Effort Multiplier for System-Specific (Non-SoS) Requirements(Non-SoS) Requirements
0.72
March 2009 USC CSSE Annual Research Review 18
Effort CalculationsEffort Calculations
SoSE Effort
SoSE Effort = 38.55*[((SoSCR/SoSTreq)*(SoSTreq)1.06 *EMSoS-CR)+ ((SoSMR/SoSTreq)*(SoSTreq)1.06 * EMSoS-MR)/152]
Where:
Total SoSE requirements = SoS Capability Requirements + SoS “Monitored” Requirements
SoS “monitored” reqs = [∑SE non-SoS requirements being addressed current upgrade cycles for all SoS constituent systems] * “Oversight Factor”
“Oversight Factor” = 5% , 10%, 15% (these values are based on expert judgment from various CSSE affiliates and the SoS SE Guidebook team)
Based on COCOMO II approach for combining components with different EMs (SoS changes and Constituent System oversight)
March 2009 USC CSSE Annual Research Review 19
Effort Calculations Effort Calculations (continued)(continued)
Single System Effort with Support from SoSE Team
Total single system reqsw-SoSE = SoS requirements allocated to system + SE reqs in upgrade cycle
Single system SE Effort with SoSE Team
= 38.55*[1.15*( (SoSCSalloc / CSTreqSoSE)*( CSTreqSoSE)1.06* EMCS-CRwSOSE) +
(CSnonSoS / CSTreqSoSE)*( CSTreqSoSE)1.06* EMCSnonSOS] /152
Based on COCOMO II approach for combining components with different EMs plus including a 15% “tax” to support SoSE team in their engineering effort for the SoSE requirements. 15% represents half of the system design effort in the EIA 632 tasks.
March 2009 USC CSSE Annual Research Review 20
Effort Calculations Effort Calculations (continued)(continued)
Single System Effort with No SoSE Team Support
Total single system reqs wo-SoSE = SoSE capability reqs + SE non-SoS requirements
Single system SE Effort without SoSE Team =
38.55*[(( SoSCR / CSTreqwoSoSE)*( CSTreqwoSoSE)1.06* EMCS-CRnSOSE) +
((CSnonSoS / CSTreqwoSoSE)*( CSTreqwoSoSE)1.06* EMCSnonSOS)] /152
Based on COCOMO II approach for combining components with different EMs (SoS changes and non-SoS changes)
March 2009 USC CSSE Annual Research Review 21
Range of SoS Complexity Factor ValuesRange of SoS Complexity Factor Values
SoSE Model Parameter
Description Range of Values
SoS Size Number of constituent systems within the SoS
2-200
SoS Capability Size Number of equivalent nominal requirements as defined by COSYSMO
1-1000
Constituent System Volatility
Number of non-SoS changes being implemented in each constituent system in parallel with SoS capability changes
0-2000
Scope of SoS Capability
Number of constituent systems that must be changed to support capability
One to SoS Size (total number of constituents systems within the SoS)
SoSE Oversight Factor
Oversight adjustment factor to capture SoSE effort associated with monitoring constituent system non-SoS changes
5%, 10%, and 15%
March 2009 USC CSSE Annual Research Review 22
Results of ResearchResults of ResearchScenario 1 (SoS Size Varies) Scenario 2 (SoS Size Varies)
Scenario 3 (SoS Size Varies) Scenario 4 (SoS Size Varies)
Relative Cost of Collaborative and Acknowledged SoSECapability Affects Half of the Systems
System Volatility = 100 Reqs and SoS Capability = 100 Reqs
-300.00
0.00
300.00
600.00
900.00
1200.00
1500.00
1800.00
0 50 100 150 200 250
Number of Systems
Sav
ing
s (P
erso
n M
on
ths)
OSF 5%
OSF 10%
OSF 15%
Relative Cost of Collaborative and Acknowledged SoSECapability Affects Half of the Systems
System Volatility = 100 Reqs and SoS Capability = 50 Reqs
-200.00
0.00
200.00
400.00
600.00
800.00
0 50 100 150 200 250
Number of Systems
Sa
vin
gs
(P
ers
on
Mo
nth
s)
OSF 5%
OSF 10%
OSF 15%
Relative Cost of Collaborative and Acknowledged SoSECapability Affects One-Fourth of the Systems
System Volatility = 100 Reqs and SoS Capability = 100 Reqs
-400.00
0.00
400.00
800.00
1200.00
1600.00
2000.00
0 50 100 150 200 250
Number of Systems
Sa
vin
gs
(Pe
rso
n M
on
ths
) OSF 5%
OSF 10%
OSF 15%
Relative Cost of Collaborative and Acknowledged SoSECapability Affects Half of the Systems
System Volatility = 100 Reqs and SoS Capability = 25 Reqs
-200.00
-100.00
0.00
100.00
200.00
300.00
400.00
0 50 100 150 200 250
Number of Systems
Sa
vin
gs
(P
ers
on
Mo
nth
s)
OSF 5%
OSF 10%
OSF 15%
March 2009 USC CSSE Annual Research Review 23
Results of Research Results of Research (continued)(continued)
Scenario 5 (SoS Size Varies) Scenario 6 (SoS Size Varies)
Scenario 7-a (SoS Size = 10) Scenario 7-b (SoS Size = 100)
Relative Cost of Collaborative and Acknowledged SoSECapability Affects Half of the Systems
System Volatility = 2000 Reqs and SoS Capability = 100 Reqs
-15000.00
-10000.00
-5000.00
0.00
0 50 100 150 200 250
Number of Systems
Sav
ing
s (P
erso
n M
on
ths)
OSF 5%
OSF 10%
OSF 15%
Relative Cost of Collaborative and Acknowledged SoSECapability Affects All of the Systems
System Volatility = 2000 Reqs and SoS Capability = 100 Reqs
-10000.00
-8000.00
-6000.00
-4000.00
-2000.00
0.00
2000.00
0 50 100 150 200 250
Number of Systems
Sav
ing
s (P
erso
n M
on
ths)
OSF 5%
OSF 10%
OSF 15%
Relative Cost of Collaborative and Acknowledged SoSESoS Capability Scope Varies
System Volatility = 1000 Reqs and SoS Capability = 1000 Reqs
-1500.00
-1000.00
-500.00
0.00
500.00
1000.00
1500.00
0 1 2 3 4 5 6 7 8 9 10 11 12
Number of Systems Affected by Capability
Sa
vin
gs
(P
ers
on
Mo
nth
s)
OSF 5%
OSF 10%
OSF 15%
Relative Cost of Collaborative and Acknowledged SoSESoS Capability Scope Varies
System Volatility = 1000 Reqs and SoS Capability = 1000 Reqs
-5000.00
0.00
5000.00
10000.00
15000.00
20000.00
25000.00
0 20 40 60 80 100 120
Number of Systems Affected by Capability
Sav
ings
(Per
son
Mon
ths) OSF 5%
OSF 10%
OSF 15%
March 2009 USC CSSE Annual Research Review 24
Results of Research Results of Research (continued)(continued)
Scenario 8-a (SoS Size = 10) Scenario 8-b (SoS Size = 100)
Scenario 9 (SoS Size = 10) Scenario 10 (SoS Size = 5)
Relative Cost of Collaborative and Acknowledged SoSESoS Capability Scope Varies
System Volatility = 1000 and SoS Capability = 1 Req
-300.00
-200.00
-100.00
0.00
100.00
0 1 2 3 4 5 6 7 8 9 10 11 12
Number of Systems Affected by Capability
Sav
ing
s (P
erso
n M
on
ths)
OSF 5%
OSF 10%
OSF 15%
Relative Cost of Collaborative and Acknowledged SoSESoS Capability Scope Varies
System Volatility = None and SoS Capability = 1000 Reqs
-1000.00
-500.00
0.00
500.00
1000.00
1500.00
0 1 2 3 4 5 6 7 8 9 10 11 12
Number of Systems Affected by Capability
Sav
ing
s (P
erso
n M
on
ths)
OSF 5%
OSF 10%
OSF 15%
Relative Cost of Collaborative and Acknowledged SoSESoS Capability Scopre Varies
System Volatility = None and SoS Capability = 1000 Reqs
0.00
5000.00
10000.00
15000.00
20000.00
25000.00
0 20 40 60 80 100 120
Number of SYstems Affected by Capability
Sav
ings
(Per
son
Mon
ths)
OSF 5%
OSF 10%
OSF 15%
Relative Cost of Collaborative and Acknowledged SoSESoS Size = 5 SoS Capability Scope Varies
System Volatility = 1000 Reqs and SoS Capability = 1000 Reqs
-1000.00
-500.00
0.00
500.00
0 1 2 3 4 5 6
Number of Systems Affected by Capability
Sav
ing
s (P
erso
n
Mo
nth
s) OSF 5%
OSF 10%
OSF 15%
March 2009 USC CSSE Annual Research Review 25
Results of Research Results of Research (continued)(continued)
Scenario 11 (SoS Size = 5) Scenario 12 (SoS Size = 5)
Relative Cost of Collaborative and Acknowledged SoSESoS Size = 5 SoS Capability Scope Varies
System Volatility = None and SoS Capability = 1000 Reqs
-800.00
-600.00
-400.00
-200.00
0.00
200.00
0 1 2 3 4 5 6
Number of Systems Affected by Capability
Sav
ings
(Per
son
Mon
ths)
OSF 5%
OSF 10%
OSF 15%
Relative Cost of Collaborative and Acknowledged SoSESoS Size = 5 SoS Capability Scope Varies
System Volatility = 1000 and SoS Capability = 1 Req
-160.00
-120.00
-80.00
-40.00
0.00
0 1 2 3 4 5 6
Number of Systems Affected by Capability
Sa
vin
gs
(P
ers
on
Mo
nth
s)
OSF 5%
OSF 10%
OSF 15%
March 2009 USC CSSE Annual Research Review 26
There exists a threshold where it is more cost effective to manage and engineer
changes to an SoS using an SoSE team and this threshold can be determined by modeling the SoS’ interdependency and
complexity characteristics.
When is it cost effective to create and empower an SoSE team to oversee and guide the
evolution of an SoS?
SoSE Model
Model parameters: SoS size Scope/size of SoS change CS volatility SoSE oversight
ConclusionsConclusions
March 2009 USC CSSE Annual Research Review 27
Conclusions Conclusions (continued)(continued)
• SoSE team is cost effective when– SoS contains more than a “few” systems– SoS capability changes typically affect a “significant
percentage” of constituent systems– SoS capability requirements are a “significant percentage”
of the total requirements being addressed by constituent systems in an upgrade cycle
– SoS oversight activities and the rate of capability modifications/changes being implemented are sufficient to keep an SoSE team engaged (i.e., little-to-no slack time)
• SoSE team is NOT cost effective when– The number of systems in an SoS is “small”– The constituent system volatility is high and the SoS
changes are small
March 2009 USC CSSE Annual Research Review 28
Conclusions Conclusions (continued)(continued)
• The “oversight factor” (the amount of effort spent by the SoSE team to monitor non-SoS changes in the constituent systems) is a key factor in determining the cost effectiveness of the SoSE team– More work is needed to determine a more accurate
“oversight factor”– This factor may be variable across multiple SoSs
• There may be reasons other than cost to engage an SoSE team– Importance of SoS– Critical SoS performance requirements requiring extensive
analysis at the SoS level
March 2009 USC CSSE Annual Research Review 29
Future WorkFuture Work
• Expand SoSE model to– Include schedule factors to allow trade-offs between
“faster” and “cheaper”
– Include quality factors based on complexities and the resulting rework due to inadequate SoS engineering
– Allow users to specify specific constituent system configurations to allow capability alternative trade-offs
• Investigate the factors in going from an Acknowledged SoS to a Directed SoS
March 2009 USC CSSE Annual Research Review 30
Backup ChartsBackup Charts
March 2009 USC CSSE Annual Research Review 31
Translating capability objectives Translating capability objectives
Translating capability objectives
Addressing new requirements
& options
Addressing new requirements
& options
Addressingrequirements
& solution options
Understanding systems &
relationships(includes plans)
Understanding systems &
relationships(includes plans)
Understanding systems &
relationships
External Environment
Developing, evolving and maintaining
SoS design/arch
Developing, evolving and maintaining
SoS design/arch
Developing& evolving
SoS architecture
Assessing (actual)
performance to capability objectives
Assessing (actual)
performance to capability objectives
Assessing performance to capability objectives
Orchestrating upgrades
to SoS
Orchestrating upgrades
to SoS
Orchestrating upgrades
to SoS
Monitoring & assessing
changes
Monitoring & assessing
changes
Monitoring & assessing
changes
Traditional SE and SoSE ActivitiesTraditional SE and SoSE Activities
Traditional SE (Defense Acquisition Guide
[DoD, 2006] View)
SoSE (SoS SE Guidebook View Based on
Interviews and Analysis of 18 DoD SoSs in Various Stages)
March 2009 USC CSSE Annual Research Review 32
Key COSOSIMO Research FindingsKey COSOSIMO Research Findings
• Limitations of COSYSMO for “Directed” SoSE effort estimation– Missing cost factors
• Cost/schedule compatibility of proposed SE approach• Level of overall risk resolution• Number of constituent systems and associated organizations• Constituent system maturity and stability• Constituent system readiness
– Need to adjust for SoSE oversight of constituent system SE– Need ability to assign different EMs to various parts of SE
March 2009 USC CSSE Annual Research Review 33
Using System Dynamics Models to Explore Using System Dynamics Models to Explore Alternatives or Influences in the Development of Large Alternatives or Influences in the Development of Large
Software-Intensive SystemsSoftware-Intensive Systems
• System dynamics modeling tool: visual modeling tools that allow one to conceptualize, simulate and analyze models of dynamic systems and processes
• Consist of causal loops or stock and flow diagrams
• Models are executable, allowing use to explore behaviors of the model as variables representing process influences are changed
• Examples:– Hybrid/plan-driven ICM [Madachy
et al., 2007]– Intergovernmental collaboration
[Cresswell et al., 2002] – Inter-Organizational Baseline
Alignment [Greer et al., 2005]– Requirements volatility [Ferreira,
2002]– Under-allocation of resources in
early phases of a project [Black and Repenning, 2001]
– Interactions between concurrently developed projects [Ford and Sterman, 2003]
March 2009 USC CSSE Annual Research Review 34
Model Validity Rationale Model Validity Rationale
• SoSE model description– Comparison model based on a modified version of the validated
academic systems engineering cost model, COSYSMO
– Modifications based upon key findings of the OSD SoSE case studies
• Validation goal: Show that the SoSE cost model is a valid method conducting sensitivity analyses for two different SoS management strategies– Collaborative
– Acknowledged
• Not part of the validation goal: The estimation of actual effort associated with a specific SoS or a given set of SoSs– The calibration/validation of the SoSE model for this purpose is left for
future work
March 2009 USC CSSE Annual Research Review 35
Model Validity Rationale Model Validity Rationale (continued)(continued)
• Validity argument– COCOMO II and Academic COSYSMO are multiple regression models
that have been calibrated and validated with actual data from primarily DoD programs
– Academic COSYSMO calibration data contains 3 SoS data points
– Most other COSYSMO calibration data points interface to other systems, which implies that they are part of one or more SoSs
– SoSE model was developed using • Academic COSYSMO that includes ability to distribute effort across SE phases• Locally validated COSYSMO extension to adjust effort for reuse/oversight of
evolving system components• COCOMO II technique for using multiple effort multipliers to characterize
components with different characteristics and complexities
– SoSE model parameters • Based on ranges of size drivers determined through case studies and surveys• Uses nominal cost driver values unless reasons identified in SoSE or SE
survey data to indicate otherwise• Resulting in a relative comparison of the two management approaches
March 2009 USC CSSE Annual Research Review 36
Model Validity Rationale Model Validity Rationale (continued)(continued)
• Validity argument (continued)– The prediction accuracies (PRED factors) are
• COCOMO [Clark and Reifer, 2007]– PRED(30) = 75% (with no stratification of projects)– PRED(30) = 80% (with stratification of projects)
• COSYSMO [Valerdi, 2005]– PRED(30) = 75% (with stratification of projects)– PRED(30) = 85% (anecdotal evidence from local calibrations)
– The OSD SoSE cases studies show that SoS systems engineers perform the same types of activities as addressed by the SE cost model, COSYSMO
– The OSD SoSE case studies identify differences between SoSE and SE for a single system and most of these differences are with respect to parameters in the SE cost model, COSYSMO
– There exists a local (single organization) calibrated and validated method within COSYSMO to estimate effort for oversight of related/interfacing systems or reusable components [Wang et al, 2008]
March 2009 USC CSSE Annual Research Review 37
ReferencesReferences
Ackoff, R. (1971); “Towards a System of Systems Concepts”, Management Science, Vol 17, No. 11, Theory Series, pp. 661-671.ANSI/EIA (1999). ANSI/EIA-632-1988 Processes for Engineering a System.Berry, B. (1964); “Cities as Systems within Systems of Cities”, The Regional Science Association Papers, Volume 13..Black, L. and Repenning, N. (2001); “Why Firefighting is Never Enough: Preserving High-Quality Product Development”, System Dynamics
Review, Vol. 17, No: 1, pp. 33-62Blanchard, B. and Fabrycky, W. (1998). Systems Engineering and Analysis, Prentice Hall.Boehm, B., Abts, C., Brown, A. W., Chulani, S., Clark, B., Horowitz, E., Madachy, R., Reifer, D. J. and Steece, B. (2000). Software Cost
Estimation With COCOMO II, Prentice Hall.Boehm, B. and Lane J. (2006) "21st Century Processes for Acquiring 21st Century Software-Intensive Systems of Systems." CrossTalk - The
Journal of Defense Software Engineering, Vol. 19, No. 5, pp.4-9.Boehm, B., Valerdi, R., Lane, J., Brown, A., (2005) “COCOMO Suite Methodology and Evolution,” CrossTalk - The Journal of Defense
Software Engineering, Vol. 18, No. 4, pp. 20-25, April 2005.Cocks, D. (2006); “How Should We Use the Term “System of Systems” and Why Should We Care?”, Proceedings of the 16th Annual INCOSE
International Symposium.Cresswell, A. et al. (2002); "Modeling Intergovernmental Collaboration: A System Dynamics Approach", Proceedings of the 35th Annual
Hawaii International Conference on System Sciences.Dahmann, J. and Baldwin. K. (2008); “Understanding the Current State of US Defense Systems of Systems and the Implications for Systems
Engineering”, Montreal, Canada: Proceedings of the IEEE Systems Conference, 7-10 April.Department of Defense (DoD) (2006); Defense Acquisition Guidebook, Version 1.6, accessed at http://akss.dau.mil/dag/ on 2/2/2007.Department of Defense (DoD) (2008); Systems Engineering Guide for System of Systems, version 1.0.Dorner, D. (1996); The Logic of Failure, Metropolitan Books.Ferreira S. (2002); Measuring the Effects of Requirements Volatility on Software Development Projects. Ph.D. Dissertation, Arizona State
University.Finley, J. (2006); “Keynote Address”, Proceedings of the 2nd Annual System of Systems Engineering Conference Ford D. and Sterman J. (2003); "Iteration Management for Reduced Cycle Time in Concurrent Development Projects", Concurrent
Engineering Research and Application (CERA) Journal.Friedman, T. (2005), The World is Flat: A Brief History of the Twenty-First Century, Farrar, Straus and Giroux, New York.Greer, D., Black, L., Adams, R. (2005), "Improving Inter-Organizational Baseline Alignment in Large Space System Development Programs",
Proceedings of IEEE Aerospace Conference.Highsmith, J. (2000); Adaptive Software Development: A Collaborative Approach to Managing Complex Systems, Dorset House Publishing.
March 2009 USC CSSE Annual Research Review 38
References References (continued)(continued)
INCOSE (2006); Systems Engineering Handbook, Version 3, INCOSE-TP-2003-002-03.isee Systems (2007), "iThink", http://www.iseesystems.com/Softwares/Business/ithinkSoftware.aspx accessed on 2/10/2007.ISO/IEC (2002). ISO/IEC 15288:2002(E) Systems Engineering - System Life Cycle Processes.Ferreira S. (2002); Measuring the Effects of Requirements Volatility on Software Development Projects. Ph.D. Dissertation, Arizona
State University.Finley, J. (2006); “Keynote Address”, Proceedings of the 2nd Annual System of Systems Engineering Conference Ford D. and Sterman J. (2003); "Iteration Management for Reduced Cycle Time in Concurrent Development Projects", Concurrent
Engineering Research and Application (CERA) Journal.Friedman, T. (2005), The World is Flat: A Brief History of the Twenty-First Century, Farrar, Straus and Giroux, New York.Greer, D., Black, L., Adams, R. (2005), "Improving Inter-Organizational Baseline Alignment in Large Space System Development
Programs", Proceedings of IEEE Aerospace Conference.Highsmith, J. (2000); Adaptive Software Development: A Collaborative Approach to Managing Complex Systems, Dorset House
Publishing.INCOSE (2006); Systems Engineering Handbook, Version 3, INCOSE-TP-2003-002-03.isee Systems (2007), "iThink", http://www.iseesystems.com/Softwares/Business/ithinkSoftware.aspx accessed on 2/10/2007.ISO/IEC (2002). ISO/IEC 15288:2002(E) Systems Engineering - System Life Cycle Processes.Kreitman, K.(1996), "From 'The Magic Gig' to Reliable Organizations: A New Paradigm for the Control of Complex Systems",
Symposium on Complex Systems Engineering, http://cs.calstatela.edu/wiki/index.php/ Symposium_on_Complex_Systems_Engineering, accessed on 1/11/2007.
Krygiel, A. (1999); Behind the Wizard’s Curtain; CCRP Publication Series, July, 1999, p. 33Lane, J. and Boehm, B. (2007); Modern Tools to Support DoD Software Intensive System of Systems Cost Estimation: A DACS
State-of-the-Art Report, Data and Analysis Center for Software, DACS Report Number 347336.Lane, J. and Valerdi, R., (2007); "Synthesizing System-of-Systems Concepts for Use in Cost Estimation", Systems Engineering, Vol.
10, No. 4. Lu, S. (2003); Engineering as Collaborative Negotiation: A New Paradigm for Collaborative Engineering,
http://wisdom.usc.edu/ecn/about_ECN_what_is_ECN.htm accessed on 2/14/2007.Madachy, R., B. Boehm, and J. Lane (2007); “Assessing Hybrid Incremental Processes for SISOS Development”, Software Process:
Improvement and Practice, Vol. 12, Issue 5, pp. 461-473.Maier, M. (1998); “Architecting Principles for Systems-of-Systems”; Systems Engineering, Vol. 1, No. 4 (pp 267-284)
March 2009 USC CSSE Annual Research Review 39
References References (continued)(continued)
NAVSTAR Global Positioning System Joint Program Office, http://gps.losangeles.af.mil/ , accessed on 12/6/2006.
Northrop, L., et al. (2006); Ultra-Large-Scale Systems: The Software Challenge of the Future, Software Engineering Institute.
Pinney, B. (2001); Projects, Management, and Protean Times: Engineering Enterprise in the United States, 1870-1960, PhD Dissertation, Massachusetts Institute of Technology.
Pressman, J. and Wildavsky, A. (1973); Implementation: How Great Expectations in Washington are Dashed in Oakland; Or, Why It’s Amazing that Federal Programs Work at All, This Being a Saga of the Economic Development Administration as Told by Two Sympathetic Observers Who Seek to Build Morals on a Foundation of Ruined Hopes, University of California Press.
Rechtin, E. (1991); Systems Architecting: Creating & Building Complex Systems, Prentice Hall.
SEI (2001), Capability Maturity Model Integration (CMMI), CMU/SEI-2002-TR-001.
Sheard, S. (2006), "Foundations of Complexity Theory for Systems Engineering of Systems of Systems", Proceedings of the IEEE Conference on System of Systems Engineering.
United States Air Force (USAF) Scientific Advisory Board (SAB) (2005); Report on System-of-Systems Engineering for Air Force Capability Development; Public Release SAB-TR-05-04
Valerdi, R. (2005); Constructive Systems Engineering Cost Model. PhD. Dissertation, University of Southern California.
Valerdi, R. and Wheaton, M. (2005); "ANSI/EIA 632 as a Standardized WBS for COSYSMO", AIAA-2005-7373, Proceedings of the AIAA 5th Aviation, Technology, Integration, and Operations Conference, Arlington, Virginia.
Wang, G., Valerdi, R., Ankrum, A., Millar, C., and Roedler, G. (2008), "COSYSMO Reuse Extension", Proceedings of the 18th Annual International Symposium of INCOSE, The Netherlands.