Regular Paper Case Studies of Systems Engineering … Studies of Systems Engineering and Management in Systems Acquisition George Friedman1 and Andrew P. Sage*, 2 1Department of Industrial

Case Studies of SystemsEngineering andManagement in SystemsAcquisitionGeorge Friedman1 and Andrew P. Sage*, 2

1Department of Industrial and Systems Engineering, University of Southern California, Los Angeles, CA

2Department of Systems Engineering and Operations Research, George Mason University, Fairfax VA, 22030-4444

CASE STUDIES OF SYSTEMS ENGINEERING AND MANAGEMENT IN SYSTEMS ACQUISITION

Received 6 April 2003; Accepted 22 September 2003, after one or more revisionsDOI 10.1002/sys.10057

ABSTRACT

This paper discusses the role of case studies in systems engineering and systems manage-ment, especially case studies that involve systems acquisition. We first provide a briefoverview of case studies, including some of the analysis techniques useful for the conduct ofcase studies. Next, we discuss a two-dimensional framework for systems engineering andmanagement case studies. The framework is in the form of a 9 row by 3 column matrix. Wepresent a number of vignettes of case studies, at least one for each of the 27 cellular entriesin this matrix. The hope is that this will be a stimulus and precursor of additional systemsengineering and management case study efforts, both in terms of appropriate frameworks forthese and in the actual conduct of case study research. © 2003 Wiley Periodicals, Inc. Syst Eng7: 84–97, 2004

Key words: case studies; case framework; systems engineering taxonomy

1. INTRODUCTION

Systems engineering concepts—including methods,processes and systems management to enable trustwor-thy implementation of processes and methods to ensurequality products and services—are very important forsuccess today. The material to follow was brought to-gether with two major considerations:

Regular Paper

*Author to whom all correspondence should be addressed (e-mail:[email protected]).

Systems Engineering, Vol. 7, No. 1, 2004© 2003 Wiley Periodicals, Inc.

84

1. In the teaching of systems engineering, case stud-ies are potentially very instructive in that theyrelate aspects of the real world to the studentthrough exposure to realities in the world ofprofessional practice. Frequently, the bad exam-ples are even more valuable than the good ones,since they emphasize the penalties for not follow-ing the proper concepts and processes of systemsengineering.

2. In teaching systems engineering, there has pre-viously been little distinction between the dutiesand responsibilities of public sector, government,and private sector, industry, activities. There isalso a third sector, comprised of nonprofit organi-zations, which may often need to be distin-guished from for-profi t private sectororganizations.

Understanding of all three roles is needed. Many, ifnot most, of the textbooks available, with the exceptionof strictly government publications, stress the privatesector aspects of systems engineering. However, infuture curricula, the distinction should be made clearerand the government aspects deserve much more empha-sis, as does the need for and role of joint government-contractor effort. Of course, the private sector industrialaspects should not be ignored; government and non-profit sector professionals should be expected to be-come smart buyers of private sector contractor activitiesand appreciate their many problems and challenges.

It is usually important to define a concept, even thosethat seem to be familiar. For our purposes (Yin, 2003a),a case study is empirical inquiry that:

• Investigates a contemporary phenomenon withinits real-life context, especially when

• Boundaries between phenomenon and contextare not clearly evident, and in which

• Multiple sources of evidence are generally used.

Many other authors infer similar definitions. Case stud-ies have has a long history in many disciplines, notably,sociology, psychology, medicine, history and politicalscience, and business. The history of case studies showsperiods of intense use and also periods of considerabledisuse.

2. AN OVERVIEW OF CASE STUDYRESEARCH

Case studies have had a long history, and formal oneshave been conducted for approximately a century. In-formal case studies have, of course, existed for a much

longer period of time. In the social sciences, case stud-ies became less popular a half century ago as the socialscience disciplines became more “scientific” and quan-titative. Case studies were seen as suffering from fun-damental problems of external and internal validity, andsometimes even from construct validity and reliability(Yin, 2003a):

• Internal Validity—Were the findings actuallyjustified by the research, or were there problemsof researcher bias? Has the case study researcherdemonstrated a causal relationship between fac-tors by showing that other plausible factors couldnot equally well or perhaps better explain theobserved relationships?

• External Validity—Could the research findingsbe generalized?

• Construct Validity—Do the measures used inthe case study make the concepts involved opera-tional? In other words have we used multipleevidence sources, have we sufficiently estab-lished chains of evidence and have those provid-ing evidence to the case study been allowed toreview the case report before finalizing it.

• Reliability—To what extent would other re-searchers who are studying the same case inexactly the same way arrive at equivalent conclu-sions?

Each of these is particularly important in case studyresearch. For example, relative to researcher bias, it isparticularly important to avoid even the appearance ofresearcher bias. At a minimum, this suggests stronglythat the case study research team members should notall be stakeholders of the programs being reviewed.

A recent revival of interest in case study research hasoccurred, especially in evaluation research in manyareas such as enterprise management. This has led tothe recognition that case study research can fill impor-tant needs. In particular, it is often insufficient to knowthat X can cause Y. We also need to know the how andthe why X causes Y, and for what specific X and Y. Casestudy research can potentially answer these interroga-tives and thereby help us become more contextuallyaware. Modern case study research is able to deal withissues of internal and external validity if the case studyis well defined, developed, and deployed, with appro-priate formulation, analysis and assessment, and inter-pretation across these three phases of the case studyeffort. These are the three basic phases and steps in atwo-dimensional framework for systems engineering(Sage, 1992, 1995; Sage and Rouse, 1999), and it isfully appropriate and, we believe, desirable that a sys-tems engineering process be used to identify a suitable

CASE STUDIES OF SYSTEMS ENGINEERING AND MANAGEMENT IN SYSTEMS ACQUISITION 85

framework for case study research in systems engineer-ing.

A prototypical characteristic of case studies is thatthey support a holistic understanding and interpretationof the systems of action, or interrelated activities en-gaged in by the participants or actors in the case situ-ation subject to study. Selecting cases must beaccomplished in such a way as to maximize what canbe learned in the period of time available for the study.Case studies often are selective, focusing on one or twoissues that are believed most fundamental to under-standing the issue under consideration. Case studiesshould be multiperspective in nature, where the casestudy researcher considers not only perspectives of theactors, but also of the interaction between them. Whatis “most fundamental and important” in a given casedepends greatly upon the viewpoint or perspective ofthe case study participants and observers.

Yin (2003a) has summarized the major strengths andlimitations inherent in case study designs. First, casestudies are useful for addressing questions regardinghow and why phenomena behave the way they behave.These case studies more often lead to hypotheses aboutbehavior rather than being useful for validating generalclaims about behavior. Second, case study researchoften reveals a rich detail of information that highlightsthe critical contingencies that exist among the variablesin the case study. Finally, the case study method isespecially useful for exploration of topics when there isnot a strong theory to which one can appeal. Even whenthere is a strong normative theory, case studies can oftenlead to valuable insights, and potentially even to revi-sion of the normative theories.

Case studies can involve either single or multiplecase designs. Single case designs are generally used toconfirm or challenge a theory, or to represent a uniqueor extreme case of behavior. Single case designs requirevery careful investigation in order to avoid misrepresen-tation and to allow maximum access by the researcherto relevant evidence. Multiple case studies follow areplication logic in which each individual case studyconsists of a “whole” study in which relevant informa-tion is gathered from various sources and conclusionsare drawn based on this information.

Yin (2003a) has identified six sources of evidence incase studies.

1. Documentation, which could be letters, memo-randa, agendas, administrative documents, news-paper articles, or any information believedrelevant to the investigation. Documents arecommunications between actors in the study andoften serve to corroborate or refute evidence fromother sources. Documents are used to make infer-

ences about events. They can result in false clues,especially in the hands of inexperienced casestudy researchers, and this is one criticism of casestudy research, leading to the observation that acase study researcher should be a “critical ob-server” to avoid being misled.

2. Archival Records can be organizational docu-ments, official lists of names, survey data, andother such records. A case study researcher mustbe careful in evaluating the accuracy of recordsbefore using them. Even if the records are quan-titative, they might still not be accurate.

3. Interviews represent a most important source ofcase study information. There are several formsof interviews: open ended, focused, and struc-tured or survey.

4. Direct Observations are obtained by such activi-ties as field visits to a case study site, and otherefforts to obtain data that range from the formalto the casual. It is thereby possible to cover eventsand contexts in real time. The disadvantages arethat it may be a time consuming and costly activ-ity, discriminatory results may be obtained unlessthe observations are broadly based, and directobservation may cause events to proceed differ-ently than the otherwise would.

5. Participant Observations have many of thesame strengths and weaknesses as direct obser-vations. They may lead to insightful observationof interpersonal behavior but are subject to biasif the case study researcher manipulates events insome ways.

6. Physical Artifact Observations may providemuch insight into cultural features and technicaloperations. The weaknesses of this source ofevidence are selectivity and availability of obser-vations.

The current rebirth of interest in case study researchhighlights the central role of theory and the importanceof having a framework within which to address casestudy research. We address this latter subject in our nextsection of this paper. Theory should guide design andanalysis of case studies, and the results of a case studyshould assist in the development of theory. Therefore,identification of boundaries for a case study should bestrongly influenced by the theoretical perspective takenby the case study researcher to the purpose and subjectmatter for the study.

Yin (2003a) recommends three general analyticalstrategies for establishing a case study:

1. Relying on theoretical propositions. The first andgenerally more preferred strategy is to follow thetheoretical propositions that led to the case study

86 FRIEDMAN AND SAGE

in the first place, at least initially. These proposi-tions necessarily shape data collection and allowus to focus attention on certain data and to ignoreother data. They ideally help us to organize acomplete, or entire, case study and to identifyalternative potential explanations that should beexamined. The “how” and “why” are especiallyrelevant here.

2. Identify thoughts about contending explanations.This allows us to examine and accept or rejectalternative premises concerning case behavior.

3. Developing a case description. In this often-usedapproach, the researcher writes a narrative thatdescribes the case and its history.

There are at least five specific techniques for analyz-ing case studies: pattern matching, explanation build-ing, logic models, time-series analysis, and cross-casesynthesis. These approaches are not mutually exclusiveand use of multiple approaches will generally be veryuseful, whenever it is possible to do so.

1. Pattern Matching. Pattern matching contrastsand compares an empirically observed patternwith one that is predicted. This might be used, forexample, to compare our case study to one foundin the literature. Here a researcher develops aprediction about patterns expected to be seen incollected data. The associated analysis attemptsto determine the pattern that best corresponds toobserved data. The major difficulty with thisapproach, assuming, of course, that it is possibleto collect the relevant data, is to decide upon howclose is the “fit” of the data to a predicted pattern.Much researcher interpretation and judgmentmay be needed.

2. Explanation Building. Here, the researcheranalyses data by building up an assumed expla-nation about the case. The purpose of this is toanalyze the case study data by establishing anexplanation about the case. This is usually ac-complished by linking events and issues causally.This approach goes beyond pattern matching inthat specification of the nature of the links, usu-ally causal, between elements that make up thepattern is a hoped for result. The researcher thentests the evidence for the relationships.

3. Toulmin Logic Models. Toulmin argumentation(Toulmin, Rieke, and Janik, 1984) is based on thenotion that the statements that make up an expla-nation fall into six basic types: data or grounds,inference warrants, backing, modal qualifier, re-buttals or reservations, and resulting claims orconclusions. Toulmin logic, the structure of

which is suggested in Figure 1, and associatedanalysis involves breaking case study materialdown into these categories and then linking themtogether such as to make structure, function, andpurpose of cases and their explanation more ex-plicit. Janssen and Sage (2000) discuss the use ofToulmin logic in a knowledge management andconflict resolution support system that may wellhave case studies as suitable inputs.

4. Time Series Analysis is used in an attempt tomatch observable patterns in data to an underly-ing model proposed for explanatory purposes.There are three major situations in time seriesanalysis. The first and simplest situation is whenthere is a single endogenous or dependent vari-able that may or may not be affected by unusualevents. The second situation extends the first byinclusion of input or causal variables that mayhave a role in the prediction or modeling of theassumed single dependent variable. The thirdsituation provides for simultaneous modeling ofmultiple dependent time-series. Very interestingsituations occur when the data is such as to sug-gest chaotic behavior patterns and the potentialneed to involve complex adaptive system proper-ties as case explanations.

5. Cross-Case Analysis and Synthesis is particu-larly useful when case studies are themselvescomprised of more than a single case.

The works of Yin (2003a) and Stake (1995) concerningcase study research are especially recommended. Yin(2003b) provides several illustrative examples of casestudies in the social sciences. Tien (1999) provides auseful discussion of the evaluation of systems and Adel-

Figure 1. Toulmin logic structure.


man (1992) discusses evaluation in decision supportand expert systems..

The Harvard Business School is among the manybusiness schools that emphasize case studies in theireducational program (Barnes, Christensen, and Han-sen, 1994). Shapiro (1988) suggests that the total caseprocess in an academic study environment, based uponthe premise that management is primarily a skill-basedactivity rather than one based on a collection of toolsand techniques, is comprised of four steps:

1. Individual analysis and preparation of and for thecase;

2. Optional informal small group discussions;3. Classroom discussions; and4. End of class generalization relative to the learn-

ing accomplished.

There are two generic types of case studies, the first ofwhich is retrospective analysis, which can be furthercategorized as historical descriptions, research eventsstudies, and matched comparisons. The second typerepresents a combination of retrospective assessmentwith prospective approaches and use of such analysis

approaches as aggregate statistics, peer review, and thelaws of bibliometrics. Each approach is useful anddevelopment of newer case study techniques has notresulted in the obsolescence and retirement of the ear-lier approaches.

In this section, we have discussed the completenessand rigor of case studies descriptions. However, inapplying case studies to systems engineering researchand education, there is a symmetrical issue: that ofpotential overdescription of the elements and environ-ment associated with crucial events of the systemsengineering process. It will often be the minimum es-sential set of measurements and data that most clearlyilluminates the workings of newly investigated phe-nomena. Excessive descriptions and elaborations maytend to cloud the inner workings of the judgmentalprocess needed for effective case study research.

3. A FRAMEWORK FOR SYSTEMSENGINEERING CASE STUDY RESEARCH

To support the objectives identified in Section 1, weidentify a two dimensional framework for systems en-gineering case study research. Figure 2 represents the

Figure 2. A framework of key systems engineering concepts and responsibilities. [Color figure can be viewed in the online issue,which is available at www.interscience.wiley.com.]


initially suggested framework. It is constructed as a 9! 3 matrix with nine general SE concept areas as rows,A–I and three columns, 1–3, depicting the domain ofthe principal activity: primarily contractor, shared be-tween contractor and government, and primarily gov-ernment. This is intended to provide the generalstructure of a framework for systems engineering con-cepts and their illustration through related case studies.This two dimensional framework is central to the casestudy efforts suggested here.

Six of the nine concept domain areas in Figure 1represent phases in the systems engineering lifecycle:

A. Requirements Definition and Management B. Systems Architecting and Conceptual DesignC. Detailed System and Subsystem Design and Im-

plementationD. Systems and Interface Integratio E. Validation and VerificationF. System Deployment and Post Deployment

Three of the nine concept areas represent necessaryprocess and systems management support:

G. Life Cycle SupportH. Risk ManagementI. System and Program Management

While other concepts could be identified, we believethat these nine are the most relevant to systems engi-neering in that they cover the essential life cycle proc-esses in systems acquisition, and the systemsmanagement support of these. Most other concept areasthat were identified appear to be subsets of one of these.

As noted, the three columns of this two-dimensionalconcept framework represents the responsibility do-main and perspectives of government, and contractor,and shared responsibilities of government and contrac-tor. This could be expanded much further. For example,each of these groups will be concerned with beingresponsive to abstractions or contextual awareness in-terrogatives of why, how, what, who, when, and where.We might also disaggregate the contractor responsibili-ties into nonprofit contractor responsibilities and for-profit contractor responsibilities. Our belief, at thispoint, is that this additional complexity is not warrantedas the notion of “contractor” and “government” needsnatural modification when there are multiple contrac-tors and multiple government agencies involved. Thisleads naturally into consideration of “system family”(Sage, 2004) considerations needed to engineer a sys-tem of systems, a family of systems, a federation ofsystems, or a coalition of systems. These expressionsare in common contemporary use to describe a system

(family, federation, coalition) that is comprised of a setof systems each of which are fully formed and func-tional, and which possesses such properties as manage-rial independence, geographic and/or temporaldistribution, emergence, evolution, emergence, and ad-aptation.

Thus, while the three column structure of “Contrac-tor Responsibilities,” “Shared Responsibilities,” and“Government Responsibilities” is very appropriate; thespecific nature of these will vary greatly dependingupon whether the system being engineered is a mono-lithic (stand alone) system or a System of Systems (orFamily or Federation of Systems). In a great manycontemporary efforts that involve the engineering of asystem, we are really dealing today with engineering aSystem (or Family or Federation or Coalition) of Sys-tems. When the government (or a private developer, forthat matter) is acquiring a stand-alone system, it mayoften contract directly with a contractor that offers“maximum value added” to the government. But whenthe government is acquiring a System of Systems, thereare at least three rather different approaches that mightbe chosen (Carlock and Fenton, 2001):

a. Option (a) is that of letting a prime SOS engineer-ing contract to a single systems engineering (andintegration) firm. This prime contractor is thenresponsible for the total systems engineering ef-fort to bring about the SOS. There are a numberof pitfalls here. For example, the governmentmay feel, after contract award, that the primecontractor is selecting dismal quality firms assubcontractors for various systems that comprisethe SOS.

b. Option (b) is that where the government assignsseparate contracts for the engineering of each ofthe component systems in the SOS and then thegovernment acts as the integrator and SOS pro-gram manager.

c. Option (c) is that where the government assignsseparate contracts for the engineering of each ofthe component systems in the SOS and then alsocontracts with another firm that acts as an inde-pendent SOS integrator and SOS program man-ager.

Thus, delineation of responsibilities in the three col-umns may be expected to vary considerably betweenacquisition of a stand alone system and acquisition ofan SOS and the program policy and management op-tions (a, b, c) chosen. What is contractor responsibility,shared responsibility, and government responsibilitydepends on which of these three options is chosen. Whatthis suggests is the great importance of considering the


three basic responsibility domains in the systems engi-neering concept framework suggested here; and devel-oping familiarity with the roles of the two majorplayers, government, and contractor, as well as sharedroles, in the contemporary engineering of systems.

4. CASE STUDIES ILLUSTRATINGSYSTEMS ENGINEERING CONCEPTS

It would be highly desirable to write and present hereone or two case studies, with each illustrating the wealthof information and knowledge that would fill all of thecells in the matrix of Figure 2. While ultimately this ispossible, we present here, for conceptual purposes,simpler studies or vignettes illustrating cell-wise facetsof good and bad systems engineering practice. We canobtain a synthesis of lessons learned from each of thesecell wise representations and do present this here. Fromexperiences gained in doing this and in understandingthese, one is in a better position to attempt larger moreholistic efforts involving complete cases.

What follows is a list comprising at least one systemsengineering concept within each of the 27 elements ofthe case study concept framework matrix. These con-cepts are written in the spirit of being requirements forgood systems engineering; thus they are written as“shall statements.” Each of the concepts is associatedwith at least one illustration or vignette of a situation,in this instance drawn from a multitude of differentcases, which would in an actual application be charac-teristic of a specific case history. Essentially, then, thetrajectory of these composite situations across the rowsand columns of the matrix would form the skeleton ofa full case history. These are intended for augmentationand revision as experience with this effort is gained.Some of these situations are decades old, and the currentfocus on evolutionary acquisition and system familiesshould provide even more current and broad scopeillustrations. Perhaps this list might ultimately becomea set of useful Heuristics for a Body of Knowledge ofSystems Engineering Concepts and Practices.

4.1. Requirements Management (A)

4.1.1. Contractor (A1)SE concept: Requirements shall flow down in a

coherent and traceable manner from the top levelto all lower levels of the system being engineered.

Case study: The flow-down of requirements wasoften slowed by the lack of early lifecycle sys-tems modeling capabilities on the part of thecontractor, so there were many paragraphs withTBDs (“to be determined” elements instead ofquantitative specifications). As the TBDs stub-

bornly refused to be quantified, program man-agement “incentivized” their determination bycoupling pay raises with how many TBDs wereeliminated. The response by the engineers was to“temporarily” eliminate these TBDs by remov-ing the paragraphs in which they were imbedded,thereby creating “silent specs” with the hope tofill them in when the analysis was complete.Time passed, and the engineers were transferredto other programs without warning their succes-sors of the silent specs. Some of these specsshould have been passed on to a key subcontrac-tor but were not. The result of this was that thesystems-level test failed, the prime contractorlost schedule and over $100M in profit. Theprogram was eventually cancelled.

4.1.2. Shared (A2)SE concept: Customer and contractor shall share

with one another their knowledge of the state oftechnical maturity relative to the new, unprece-dented systems being engineered.

Case study: A government customer desired to pro-cure two new navigational systems: one stellarinertial and the other pure inertial. The mostcapable contractor in the pure inertial field de-cided to expand the portion of the market sectorunder their purview and substantially underbidthe proposal for the stellar inertial system. Thiscontractor won; so the stellar inertial contractorcounter attacked and won the pure inertial con-tract. Both contractors, unfamiliar with the new(to them) technologies to be used in the productto be engineered, did poorly both technically andfinancially. They suffered—as did the cus-tomer—from the procurement policy that theycould not be negative about the technical aspectsof contractors who had submitted reputable pro-posals in the past.

4.1.3. Government (A3)SE concept: The government shall integrate the

needs of its user organizations with the manage-ment activities of its developmental organiza-tions. Often, the user and developmentorganizations are separated by gulfs of cultureand language. The development organizationsponsors new technology and contracts out pro-grams; the user organization—the ultimate cli-ent—operates the new systems, trains thepersonnel, fights battles, and should be inti-mately involved in writing the requirements fornew programs.


Case study: An advanced state of the art navigationalsystem was specified for an attack aircraft. Sev-eral years after the program start, the operationalarm of the government added more requirementsin the crew station interface area. They claimedthat they had insufficient voice in the construc-tion of the original specification. Results: sub-stantial financial and schedule impact.

4.2. Systems Architecture (B)

4.2.1. Contractor (B1)SE concept: The systems baseline architecture of

complex programs shall be established early inevery program and shall involve all dimensionsof technical issues, as well as such enterprisearchitecture issues as customer needs and satis-faction, political pressures and continuity offunding. A properly executed systems architec-ture activity provides benefits of effectiveness farin excess of its costs. In dealing with systemfamilies, it is to be anticipated that the systemsarchitecture will evolve and emerge from thebaseline architecture as the system is engineered.

Case study: If the allocation of functions to equip-ment and crew is performed without deep under-standing of human/machine interactions andtheir respective strengths and limitations, theoverall systems performance will suffer. In aparticularly stressful case of a combat aircraftrequired to perform aggressive maneuvers whileattempting to acquire and classify a low contrasttarget, the task of aircraft controls was delegatedto the pilot, and the task of classification wasdelegated to an automatic pattern recognitionprogram. The effectiveness of the system wouldhave been substantially improved if the maneu-vers were under automatic control (supervised bythe pilot) and the task of pattern recognition weregiven to the far superior cognitive capability ofthe crew member.

4.2.2. Shared (B2)SE concept: The systems architecture should be

established early for the reasons stated in 4.2.1,and the best judgment of both government andcontractor shall be employed across all the keyissues, including the choice of employing newlydeveloped or legacy systems.

Case study result: The existence of commercial off-the-shelf (COTS) hardware and software appearsto offer a very attractive alternative to the expen-sive and lengthy developments required to pro-duce specially designed hardware and software.

However, unless the total system architectureimplications are well understood, COTS can beextremely disappointing regarding its incom-plete characterization, testing and maintenance,and logistic aspects. Frequently, many upgradegenerations of software occur during a singlegovernment development cycle, and the producerof the software—who view the government as asmall customer compared to their commercialmarkets—is unwilling to continue supportingthe old versions. In many cases, the governmentpendulum of employing excessive specificationsswinging to relying entirely on commercial bestpractices has injured the government and indus-try’s ability to manage the integrity of new de-signs.

Additional case study result: The effectiveness ofguided missiles could be greatly improved byintegrating the functions of guidance and fusing;that is, the information regarding the terminalkinematics gathered by the guidance system canbe transferred to the fusing system, substantiallyimproving the probability of kill. However, thesetwo systems are generally kept separate by tradi-tional procurement policies and divisions.

4.2.3. Government (B3)SE concept: A total systems architecture shall be

employed by the government for the reasonsstated in 4.2.1 in order to provide a sound basisof effectiveness across the broadest spectrum ofcontractors and operations.

Case study: Joint developments in aircraft and mis-siles have been undertaken across the services inorder to attain envisioned economies. In mostcases, these have been met with several difficul-ties in extended schedules, higher costs, compro-mises in operational effectiveness, and awkwardintegration within diverse logistical support sys-tems. In the early stages of program planning andsystems architecture, all these potential problemswere lost in the glare of the hoped-for savings. Insome cases, the joint programs were cancelled.

4.3. System/Subsystem Design (C)

4.3.1. Contractor (C1)SE concept: System design shall proceed in a logical

and orderly manner through a process of func-tional decomposition and design traceability thatoriginates with the system functional architec-ture and ultimately results in design specifica-tions for the system to be engineered.


Case study result: If the functional decomposition isexcessively rigorous, potential technical syner-gies may be missed. For example, if the decom-position is strictly treelike, then the merging ofradar and inertial navigation would never havethe opportunity to develop into synthetic arrayradars and inertial components for flight controland for navigation would not have the opportu-nity for integration.

4.3.2. Shared (C2)SE concept: Government customers and contractors

shall have a contractually feasible sharing ofsystems design responsibility.

Case study result: On a defense suppression missile,the government customer engineered the overallsystems architecture and performed sole sourceselection of the subsystems. The top level archi-tecture appeared logical, but second-order inter-face problems developed during systemsintegration and test. Had the total systems re-sponsibility been given to a single prime contrac-tor, these problems would have been veryunlikely. Result: eventual program cancellationwith criticism of the contractor.

Additional case study result: A contractor proposeda technology which was proprietary. Despite thefact that this technology substantially improvedsystem effectiveness, the government gave thiscontractor a negative evaluation because the in-tellectual property (IP) protection would limitcompetitive reprocurement later in the program.The result was that the program went ahead withan inferior technology and diminished effective-ness—injuring the government—while the con-tractor was discouraged from performinginternal research and development (IR&D),thereby injuring both contractor and governmentcapabilities.

4.3.3. Government (C3)SE concept: The customer shall share high level

measures of effectiveness with the contractor,thereby ensuring that the proposals selected forfunding are those which are most responsive toall stakeholders, especially the operational or-ganizations.

Case study result: At a meeting of the contractors’chief engineers and the government engineers,the concept of such overarching measures ofeffectiveness was suggested by a contractor. Thegovernment responded that their engineerswanted to reserve the right to alter their selectioncriteria after they had reviewed all the proposal

submissions. The result: the government lost anopportunity to communicate its fundamentalneeds to the contractors who really are interestedin serving the customers better, and when thefinal decisions are made, the contractor feels thatthe government acted arbitrarily and perhapseven capriciously. In short, there was a diminish-ing trust at both government and contractor ends.

4.4. Systems Integration and Interfaces (D)

4.4.1. Contractor (D1)SE concept: The contractor shall assure that systems

integration and interfaces at each of the subsys-tems and components levels supports total sys-tem functionality throughout its life cycle.

Case study result: In general, component and sub-system testing have proved to be insufficient inpredicting problems that are likely to occur in themany stages of systems integration. It is neces-sary to insure that systems integration issues aredealt with throughout the lifecycle, especially inarchitecting and preliminary system design

4.4.2. Shared (D2)SE concept: The contractor and government shall

assure that all systems are integrated withinthemselves as well as interfaced with other exist-ing operational equipment and systems.

Case study result: Joint programs across two or moreof the services have suffered from differences inoperational use and doctrine, as well as from verydifferent support and logistics systems. In theplanning and proposal stages, neither the govern-ment nor the contractor often, or at all, considersthe multitudes of issues associated with satisfy-ing diverse customers with the same—or smallvariations of the same—system.

4.4.3. Government (D3)SE concept: The government shall assure that all its

operational systems—in development, in opera-tion, or in planning—are compatible and mutu-ally supportive in a broad “system of systems”and “federation of systems” context.

Case study result: Most DOD agencies have exist-ing, evolving, and planned intelligence, com-mand and control systems for war fighting andantiterrorism. The difficulty of obtaining reliableintelligence is made far more difficult by theinteragency rivalries and delays in getting theinformation to the fighting forces on time.


4.5. Verification and Validation (E)

4.5.1. Contractor (E1)SE concept: Every requirement shall have a test and

every test shall have a requirement which re-quires validation and verification. The criteria fordetermining test success and failure shall be es-tablished early in the program, as shall verifica-tion and validation measures.

Case study result: Near the end of the developmentphase of a complex and unprecedented electroniccountermeasures program, an exhaustive test wasconducted. The contractor was elated by the re-sults, feeling that the grade earned for perform-ance was at least an A or A–. The governmentgave the contractor a D+, only then sharing withthe contractor the specific test criteria that hadbeen used to evaluate performance.

Another case study result: Although test planningshould occur almost simultaneously with re-quirements development, often the test planningis delayed by years and test people are frequentlynot even involved with the earlier life cyclephases of architecting and preliminary systemsdesign.

4.5.2. Shared (E2)SE concept: Government facilities are often the most

expensive and their use in the verification andvalidation (V&V) process shall be shared withthe contractors. Test criteria shall be shared early.

Case study result: The coordination of scarce gov-ernment testing facilities is crucial to the successof all programs. Often, program slippages occurwhich directly impact the scheduling of theseresources negatively affecting schedules andbudgets.

4.5.3. Government (E3)SE concept: The government shall be the final word

on the confidence levels derived from its testingduring development, operational test and evalu-ation, and actual deployment and operational use.These require judgments at the highest levels,because not all operational conditions can beadequately tested and actual operations are fre-quently far different from what the planners in-itially imagined and intended.

Case study result: Most anti-terrorism and thirdworld military technology which are presentlyfielded were developed to solve military goals ofthe cold war, where the Soviet Union was thedominant adversary. Today, the focus has shiftedto combat environments such as the Persian Gulf

and antiterrorism—with substantially differentgoals, scenarios and testing challenges.

4.6. Deployment and Post-Deployment (F)

4.6.1. Contractor (F1)SE concept: As systems undergo operational test and

evaluation, the contractor shall maintain the ap-propriate engineering and testing organizationalcapabilities to support gathering, analyzing, andrecommending possible changes in the systemdesign or support through reengineering.

Case study result: Once the system is delivered tothe customer, contractors sometimes are temptedto transfer their most capable engineering andtest personnel to new programs and essentiallyforget the needs of properly supporting the sys-tem throughout its deployment lifecycle andplanning for reengineering.

4.6.2. Shared (F2)SE concept: Both government and contractor shall

work cooperatively to conduct an effective opera-tional test and evaluation (Opeval) and to be opento feeding information back to the original man-agement organizations to consider designchanges or subsequent modifications throughreengineering.

Case study result: Coordination is often difficult andthe program review process is not as intenselyfocused as those during the development phase.The Opeval environment is sometimes not a re-alistic representation of actual test conditions. Inmany cases, the real Opeval does not occur untilactual combat situations emerge.

4.6.3. Government (F3)SE concept: The government shall assure that a

properly funded Opeval occurs, one which drawsupon both contractor and government resources,and that all data gathered in the tests be evaluatedfor potential recommendation for system modi-fication, redesign, or reengineering. Preplannedproduct improvement policies shall be encour-aged.

Case study result: In many cases, as the deploymentbecomes fully operational and deficiencies arediscovered, the choice of whether to performdesign modifications or to start a new develop-ment is not as illuminated as other debates in theprocurement world. For decades, there has beena rhythm in development that every generationwill have a next generation and this precept hasoften blinded both the development and user


communities to the possibilities of extending thelife of existing and proven systems in a “pre-planned product improvement” manner.

4.7. Life Cycle Support (G)

4.7.1. Contractor (G1)SE concept: All design activities shall be performed

from the viewpoint of the entire life cycle ratherthan the early prototype development phase.

Case study result: Many preliminary design teamstravel from program to program writing propos-als and—if the company is a winner—some-times working on those programs for as much asa year, until they depart for the next program. Inthis flurry of activity, the greatest emphasis isgiven to the “dramatic” vector of key perform-ance parameters, while the many other factorsrequired for an effective life cycle are ignored orpostponed.

Another case study result: The manufacturing worldhas often complained that producibility has notreceived proper attention from the design com-munity. Yet there are many instances where adesign that optimizes certain aspects of produci-bility—such as minimum time to assemble—isvery poorly designed for maintainability. Fur-thermore, it is rare that enough attention is everpaid to training and the writing of readable anduseful user manuals.

4.7.2. Shared (G2)SE concept: A balanced blend of all methods, meas-

urements, technologies, and processes shall beemployed in support of an effective life cycle.

Case study result: Over the years, feeling that theiroperational readiness and capability have suf-fered because of the overemphasis on perform-ance, operational commanders have exertedconsiderable pressures to increase the priority ofreliability, availability, and maintainability(RAM) and other aspects of the integrated logis-tics support (ILS) family. Additional manage-ment initiatives have entered the scene, adding tothe complexity of program development withoften conflicting requirements, such as the“maximization of reliability, availability, andmaintainability.”

4.7.3. Government (G3)SE concept: Funding support, throughout a pro-

gram’s life cycle, shall be maintained in balance,and procurement of early development programsshall recognize the importance of total life cycle

cost control through such approaches as earnedvalue management.

Case study result: The record shows many programswhich suffer disruption in funding, thereby add-ing substantially to the total cost.

Additional case study result: Despite the preachingof the importance of total life cycle cost, it is veryrare that a development contract is awarded basedon the total life cycle cost rather than the cost ofthe specific development contract.

4.8. Risk Management (H)

4.8.1. Contractor (H1)SE concept: At every level of detail, risk shall be

identified, prioritized, and mitigated, since theearly management of risk requires orders of mag-nitude fewer resources and causes orders of mag-nitude less program disruption than allowing thepotential risk become an actual crisis. Risk isassociated with all three major programmaticdimensions: technical, schedule, and cost.

Case study result: Most programs in the past havehad no formal risk management processes; insome cases risk was partially handled via a “pro-gram manager’s reserve.” When the reserve waspositive, the manager could afford to be reason-able. As it diminished and became negative, theinstance of retorts such as “I only want can-doteam players working for me, not the-sky-is-fall-ing worry-warts” and “you’re paid to fix prob-lems, not bring them to me; if your job had noproblems then I could have filled it with someonewith half your salary” become more and morefrequent. As risk management becomes moreformalized, the resulting qualitative matrices ofrisk likelihood vs. risk consequence are devel-oped to help prioritize. While better than nothing,they are not as valuable as quantitative assess-ments of probability and impact, employing de-cision and risk management theory. Even in thehighest tech companies, there is a fear that thesequantitative methods are onerous and undulymathematical—despite the fact that they feel thatthey are up to the most advanced engineeringchallenges the world has ever seen.

4.8.2. Shared (H2)SE concept: It is to the government’s benefit as well

as the contractor’s to identify and mitigate riskearly. The government shall insure that risk man-agement is part of the contract systems engineer-ing management plan (SEMP), is properly


funded, and is represented in every program re-view.

Case study result: The government has recently in-creased the emphasis on risk management, butthe approach is still largely qualitative. Somegovernment organizations view risk manage-ment as being exclusively in the domain of pro-gram management rather than shared betweenprogram management and systems engineering.Typically, the analysis required to perform ra-tional decisions regarding risk mitigation comesfrom engineering organizations, while the deci-sions to initiate (or terminate) risk mitigationactions come from program management.

4.8.3. Government (H3)SE concept: Risk management at all program levels

shall be an essential and inherent part of allsystems management program planning and lifecycle activities.

Case study result: A method for risk mitigation—es-pecially for those who are risk averse—is toemploy insurance, since the utility functions of arisk neutral insurance company and a risk averseclient are both benefited. However, the U.S. Gov-ernment, with its multi-$trillions of resourcesshould be far more risk neutral than any insurancecompany; yet it covers the cost of insuring manyof its contracted activities. An example is that ofspace launches in which several billions of dol-lars were spent on insurance during the pastdecade and the projection is that at least thatamount will be spent during the next decade.Over half of this amount went to the profit andoverhead structure of the insurance companies—who are this business to earn the profits they feelthey deserve, considering the risks they take.These funds could have been saved if the govern-ment had undertaken appropriate self-insurancemeasures.

4.9. System and Program Management (I)

4.9.1. Contractor (I1)SE concept: Each and every systems engineering

program—even small ones—shall have a Sys-tems Engineering Management Plan (SEMP)which allows the systems engineering process tobe tailored to that program. Programs differ bymany orders of magnitude in size, complexity,unprecedented technology, and difficulty of in-terface; the SEMP permits the right amount ofeffort for each circumstance.

Case study result: A systems engineer wrote aSEMP—because the customer directed it as partof the request for proposals—but as soon as itwas written and approved, it was filed and neverread. When corporate reviewers showed up toprovide program oversight, they suffered papercuts when opening documents with their fresh,unread pages.

Additional SE concept: Develop a professionalcadre of capable systems engineers, includingprofessional development and promotion struc-tures and plans.

Additional case study result: In many companies,systems engineering is performed by an unorgan-ized, disparate, uneven, disgruntled group of be-leaguered semiprofessionals drifting likemercenaries from one battlefield to the next,never knowing when the next war will come, butfearing that—no matter how hard they work inthe short term—they will be unprepared for it. Itwould not be too great a conjecture to estimatethat a large percentage of the vast majority ofunpaid overtime contributed by industry to thenational defense posture is from this set of peo-ple.

4.9.2. Shared (I2)SE concept: The role of systems engineering in

program development and management shall berecognized and supported.

Case study result: A major program started with awell-funded systems engineering program. Afew years into the program, the requirementschanged and these changes necessitated majorstructural revisions and therefore a massive hit onthe budget. All activities were reviewed for pos-sible budgetary reductions. The systems engi-neering budget was hit the hardest becauseneither the contractor nor the government engi-neers could clearly define the program risks as-sociated with reducing the SE activity. One of theactivities cut was the recording of the traceabilityof the design rationale from the top requirementsto the subsystem specs. Years later, a new pro-gram office asked for this rationale and the cor-porate representative had to convene a specialteam to reconstruct the entire process.

4.9.3. Government (I3)SE concept: The government shall establish security

levels for programs to protect crucial technolo-gies and special operational capabilities.

Case study result: The government establishes secu-rity levels in order to streamline procurement


management processes and to minimize the num-ber of people involved with oversight. In somecases, although secrets were kept inside the com-partments, competence was kept out. In manycases, early in the development, the so called“ilities people” were also kept out, thereby mak-ing life cycle support difficult if not impossible.For those programs which were sufficiently suc-cessful to enter limited production, the belatedentrance of the –"ilities" folk had a doubly nega-tive impact:

1. The neglected disciplines finally had to bedealt with—much more expensively than ifthey had been considered properly from thebeginning.

2. The experts who were rejected earlier de-lighted in making the adjustments even morepainful than they otherwise might have been,as a “proper and fitting revenge” for being leftout in the first place.

5. SUMMARY

In this paper, we have provided a summary backgroundin case study research appropriate to the developmentof definitive case studies in systems engineering andsystems management. We identified a systems engi-neering framework that appears suitable to orchestratesuccessful case studies. The framework itself is robustin the sense that issues, such as ethical and legal issues,which affect systems engineering and managementacross the levels of product, processes, or policy arecapable of being identified within the framework. Fi-nally, we provided a number of pragmatic illustrationsof case study results, with at least one illustration appli-cable to each of the cells in the 27-cell two-dimensionalframework. Each of these is associated with at least oneimportant systems engineering and management con-cept. It would be of interest also to demonstrate that the27 vignettes meet the reliability and validity require-ments of case study research described in the firstsection of the paper such that these represent scientifi-cally justified research findings. If this were a singlecase study effort, this demonstration would, of course,be vital. Some suggestions for extension of the frame-

work are provided in the course of our discussions.Others might include identification of a frameworkdevelopment process and measures of effectiveness foran instantiated framework, although these might wellbest be considered in terms of specific case studies. Itis hoped that the methodological issues discussed in thispaper will be of value for systems engineering andmanagement case study research including, but not atall limited to a set of concurrent case studies now inprogress at the Center for Systems Engineering of theAir University and the Air Force Institute of Technol-ogy.

REFERENCES

L. Adelman, Evaluating decision support and expert systems,Wiley, Hoboken, NJ, 1992.

L.B. Barnes, C.R. Christensen, and A. Hansen, Teaching andthe case method, 3rd edition, Harvard Business SchoolPress, Cambridge, MA, 1994.

P.G. Carlock and R.E. Fenton, Systems of Systems (SoS)enterprise systems engineering for information-intensiveorganizations, Syst Eng 4(4) (2001), 242–261.

T. Janssen and A.P. Sage, A support system for multipleperspectives knowledge management and conflict resolu-tion, Int J Technol Management 19(3/4/5) (2000), 472–490.

A.P. Sage, Systems engineering, Wiley, New York, 1992.A.P. Sage, Systems management for information technology

and software engineering, Wiley, New York, 1995.A.P. Sage, “System families,” McGraw Hill Yearbook of

Science and Technology, McGraw-Hill, New York, 2004,pp. 344–347.

A.P. Sage and W.B. Rouse (Eds.), Handbook of systemsengineering and management, Wiley, New York, 1999.

B.P. Shapiro, An introduction to cases Note 9-584-097, Har-vard Business School, Cambridge, MA, 1988.

R.E. Stake, The art of case study research, Sage Publications,Thousand Oaks, CA, 1995.

J.M. Tien, “Evaluation of systems,” Handbook of systemsengineering and management, A.P. Sage and W. B. Rouse(Editors), Wiley, New York , 1999, pp. 811–824.

S. Toulmin, R. Rieke, and A. Janik, An introduction to rea-soning, Collier Macmillan, New York, 1984.

R.K Yin, Case study research: Design and methods, 3rdedition, Sage Publications, Thousand Oaks, CA, 2003a.

R.K. Yin, Applications of case study research, 2nd edition,Sage Publications, Thousand Oaks, CA, 2003.


George Friedman is presently Adjunct Professor of Engineering, Industrial and Systems EngineeringDepartment, School of Engineering, University of Southern California, Los Angeles, and is director ofresearch of the Space Studies Institute in Princeton. Previously, he retired as the Corporate Vice Presidentof Engineering and Technology for the Northrop Corporation, was the Vice President of Publications forthe Aerospace and Electronics Systems Society of IEEE, chairman of the Planetary Defense Committeeof AIAA, and served as a consultant to the Air Force Scientific Advisory Board and the NATO IndustrialAdvisory Board. He is a founder of INCOSE, was its third President and serves on the editorial board ofSystems Engineering. He is an elected Fellow of IEEE, INCOSE, and the Institute for the Advancementof Engineering (IAE). His paper on Constraint Theory was awarded the IEEE Baker Prize for the bestpaper published by the IEEE from all its societies in 1969. He received the B.S. in Mechanical Engineeringfrom the University of California Berkeley and the M.S. and Ph.D. in Engineering from UCLA.

Andrew P. Sage received the BSEE degree from the Citadel, the SMEE degree from MIT, and the Ph.D.from Purdue, the latter in 1960. He received honorary Doctor of Engineering degrees from the Universityof Waterloo in 1987 and from Dalhousie University in 1997. He has been a faculty member at severaluniversities, and in 1984 he became First American Bank Professor of Information Technology andEngineering at George Mason University and the first Dean of the School of Information Technology andEngineering. In May 1996, he was elected as Founding Dean Emeritus of the School and also wasappointed a University Professor. He is an elected Fellow of the Institute of Electrical and ElectronicsEngineers, the American Association for the Advancement of Science, and the International Council onSystems Engineering. He is editor of the John Wiley textbook series on Systems Engineering andManagement, the INCOSE Wiley journal Systems Engineering, and is coeditor of Information, Knowl-edge, and Systems Management. He edited the IEEE Transactions on Systems, Man, and Cybernetics fromJanuary 1972 through December 1998, and also served a two-year period as President of the IEEE SMCSociety. In 1994 he received the Donald G. Fink Prize from the IEEE, and a Superior Public Service Awardfor his service on the CNA Corporation Board of Trustees from the US Secretary of the Navy. In 2000,he received the Simon Ramo Medal from the IEEE in recognition of his contributions to systemsengineering and an IEEE Third Millennium Medal. In 2002, he received an Eta Kappa Nu EminentMembership Award and the INCOSE Pioneer Award. His interests include systems engineering andmanagement efforts in a variety of application areas including systems integration and architecting,reengineering, and industrial ecology and sustainable development.
