Ultra Large Scale Database

8/9/2019 Ultra Large Scale Database

1/150

Ultra-Large-Scale

SystemsThe Software Challengeof the Future


2/150

Ultra-Large-Scale Systems Study Lead

Linda Northrop

Ultra-Large-Scale Systems Study Report

Author Team

Peter Feiler

Richard P. Gabriel

John Goodenough

Rick Linger

Tom Longstaff

Rick KazmanMark Klein

Linda Northrop

Douglas Schmidt

Kevin Sullivan

Kurt Wallnau

Chief Editor

Bill Pollak

Information Designer

Daniel Pipitone


3/150

Ultra-Large-Scale SystemsThe Sotware Challenge o the Future

June 2006

Unlimited distribution subject to the copyright.

Pittsburgh, PA 15213-3890


4/150

Ths report was prepared or the

SEI Administrative Agent

ESC/XPK

5 Eglin Street

Hanscom AFB, MA 01731-2100

The ideas and ndings in this report should not be construed as an ocial DoD

position. It is published in the interest o scientic and technical inormation

exchange.

This work is sponsored by the U. S. Army. The Sotware Engineering Institute is

a ederally unded research and development center sponsored by the U. S.

Department o Deense.

Copyright 2006 Carnegie Mellon University.NO WARRANTY

THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE

ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN AS-IS

BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF

ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER

INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR

PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS

OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON

UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITHRESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT

INFRINGEMENT.

Use o any trademarks in this report is not intended in any way to inringe on the

rights o the trademark holder.

Internal use. Permission to reproduce this document and to prepare derivative

works rom this document or internal use is granted, provided the copyright and

No Warranty statements are included with all reproductions and derivative

works.

External use. Requests or permission to reproduce this document or prepare

derivative works o this document or external and commercial use should be

addressed to the SEI Licensing Agent.

This work was created in the perormance o Federal Government Contract

Number FA8721-05-C-0003 with Carnegie Mellon University or the operation

o the Sotware Engineering Institute, a ederally unded research and develop-

ment center. The Government o the United States has a royalty-ree government-

purpose license to use, duplicate, or disclose the work, in whole or in part and in

any manner, and to have or permit others to do so, or government purposes

pursuant to the copyright license under the clause at 252. 227-7013.

For inormation about purchasing paper copies o SEI reports, please visit the

publications portion o our Web site:

http://www. sei.cmu.edu/publications/pubweb. html


5/150

Introducton

Table o Contents

Acknowledgments ................................................................................vii

Executive Summary ..............................................................................ix

Part I Section 1: Introduction ................................................................. 1

1.1 Genesis o the ULS Systems Research Study .............................1

1.2 The DoDs Goal o Inormation Dominance ..............................2

1.3 The Missing Key to Inormation Dominance .............................2

1.4 The Engineering Perspective on Sotware Development ............3

1.5 The Need or a New Perspective .................................................4

1.6 From Engineering to Complex Systems .....................................5

1.6.1 From Buildings to Cities ..............................................................5

1.6.2 From Systems to Ecosystems ......................................................6

1.6.3 Beyond the Internet ......................................................................7

1.7 The Results o This Study............................................................8

Section 2: Characteristics o ULS Systems...................................11

2.1 Decentralized Control ................................................................13

2.2 Inherently Conficting, Unknowable, and Diverse

Requirements ............................................................................. 14

2.3 Continuous Evolution and Deployment ....................................15

2.4 Heterogeneous, Inconsistent, and Changing Elements .............16

2.5 Erosion o the People/System Boundary ...................................17

2.6 Normal Failures ......................................................................... 19

2.7 New Paradigms or Acquisition and Policy ..............................20

2.8 Summary ....................................................................................20

Section 3: Challenges in ULS Systems ........................................ 21

3.1 Design and Evolution ................................................................22

3.2 Orchestration and Control .........................................................25

3.3 Monitoring and Assessment ......................................................26

Charact

erstcs

Challenges


6/150

Table o Contents

Section 4: Overview o Research Areas ....................................... 29

4.1 Human Interaction .....................................................................31

4.2 Computational Emergence ........................................................324.3 Design ........................................................................................33

4.4 Computational Engineering .......................................................35

4.5 Adaptive System Inrastructure .................................................36

4.6 Adaptable and Predictable System Quality ...............................38

4.7 Policy, Acquisition, and Management ....................................... 40

Section 5: Summary and Recommendations ............................. 41

5.1 Toward a Roadmap or a ULS Systems Research Program .....42

5.1.1 DoD Missions and Capabilities .................................................43

5.1.2 Research Tracks Associated with Missions and Capabilities ...48

5.1.3 Research Areas by Funding Types ............................................. 50

5.1.4 Research Risk/Reward ............................................................... 52

5.2 Recommendations .....................................................................54

5.3 Study Conclusion .......................................................................55

Part II Section 6: Detailed Description o Research Areas .................57

6.1 Human Interaction .....................................................................57

6.1.1 Context-Aware Assistive Computing ........................................ 59

6.1.2 Understanding Users and Their Contexts ..................................60

6.1.3 Modeling Users and User Communities ...................................61

6.1.4 Fostering Non-Competitive Social Collaboration .....................62

6.1.5 Longevity ...................................................................................63

6.1.6 Further Reading .........................................................................64

6.2 Computational Emergence ........................................................65

6.2.1 Algorithmic Mechanism Design ...............................................66

6.2.2 Metaheuristics in Sotware Engineering ...................................68

6.2.3 Digital Evolution........................................................................71

6.2.4 Further Reading .........................................................................73

6.3 Design ........................................................................................74

6.3.1 Design o All Levels ..................................................................766.3.2 Design Spaces and Design Rules ..............................................77

6.3.3 Harnessing Economics to Promote Good Design .....................79

6.3.4 Design Representation and Analysis .........................................80

6.3.5 Assimilation ...............................................................................81

Re

searchAreas

Recommendatons


7/150

6.3.6 Determining and Managing Requirements ...............................85

6.3.7 Further Reading .........................................................................87

6.4 Computational Engineering .......................................................89

6.4.1 Expressive Representation Languages ......................................90

6.4.2 Scaled-Up Specication, Verication, and Certication ..........93

6.4.3 Computational Engineering or Analysis and Design ...............95

6.4.4 Further Reading .........................................................................97

6.5 Adaptive System Inrastructure .................................................97

6.5.1 Decentralized Production Management ....................................98

6.5.2 View-Based Evolution ............................................................. 100

6.5.3 Evolutionary Conguration and Deployment .........................102

6.5.4 In Situ Control and Adaptation ................................................104

6.5.5 Further Reading .......................................................................106

6.6 Adaptable and Predictable System Quality .............................107

6.6.1 Robustness, Adaptation, and Quality Attributes .....................108

6.6.2 Scale and Composition o Quality Attributes .........................109

6.6.3 Understanding People-Centric Quality Attributes ..................111

6.6.4 Enorcing Quality Requirements .............................................112

6.6.5 Security, Trust, and Resiliency ................................................ 113

6.6.6 Engineering Management at Ultra-Large Scales ....................114

6.6.7 Further Reading .......................................................................115

6.7 Policy, Acquisition, and Management .....................................118

6.7.1 Policy Denition or ULS Systems .........................................118

6.7.2 Fast Acquisition or ULS Systems ..........................................120

6.7.3 Management o ULS Systems .................................................122

6.7.4 Further Reading .......................................................................123

Glossary..............................................................................................125


8/150

Table 1: Relationship Between Research Areas and Challenges ..... 31

Table 2: Research Topics Needed or Specic DoD

Missions and Related Capabilities ......................................49

Table 3: Research Topics Categorized by DoDResearch Funding Type .......................................................51

Table 4: Research Areas and Range o Risk/Reward ....................... 53

Lst o Tables


9/150

Acknowledgments

The principal team o authors who wrote this report consists o

Peter Feiler, John Goodenough, Rick Linger, Tom Longsta, Rick Kazman,

Mark Klein, Linda Northrop, and Kurt Wallnau rom the Carnegie Mellon

Sotware Engineering Institute (SEI), along with Richard P. Gabriel,

Sun Microsystems, Inc.; Douglas Schmidt, Vanderbilt University; and

Kevin Sullivan, University o Virginia.

The team thanks the many individuals who contributed directly to its work.

The Ultra-Large-Scale (ULS) Systems Study Team consisted o

Gregory Abowd, Georgia Institute o Technology; Carliss Baldwin,

Harvard Business School; Robert Balzer, Teknowledge Corporation;

Gregor Kiczales, University o British Columbia; John Lehoczky,

Carnegie Mellon University; Ali Mili, New Jersey Institute o Technology;

Peter Neumann, SRI International; Mark Pleszkoch, SEI; Mary Shaw,

Carnegie Mellon University; Daniel Siewiorek, Carnegie Mellon University;and Jack Whalen, Palo Alto Research Center (PARC). All o these

individuals made substantive contributions to the oundational ideas that

the author team developed in this report.

Invited speakers at an early meeting were Assistant Secretary o the Army

Claude Bolton (on video), Oce o the Assistant Secretary o the U. S.

Army (Acquisition, Logistics, & Technology) (ASA ALT); Peter Freeman,

National Science Foundation; David Emery, DSI; and Bruce Krogh,

Carnegie Mellon University. They provided valuable inspiration to thestudy principals.

This report was reviewed in drat orm by individuals chosen or their

diverse perspectives and technical expertise. The purpose o this indepen-

dent review was to elicit candid and critical comments that would help make

the published report as technically sound as possible. The author team is

grateul to those who careully and thoughtully reviewed its interim drats:

John Bay, Air Force Research Lab; Brian Barry, Bederra Corporation;

Barry Boehm, University o Southern Caliornia; Larry Druel, SouthCarolina Research Authority (SCRA); Peter Freeman, National Science

Foundation; Ron Goldman, Sun Microsystems Laboratories; Watts S.

Humphrey, SEI; Bruce Krogh, Carnegie Mellon University; Jim Linnehan,

ASA ALT; Martin Rinard, Massachusetts Institute o Technology;

Dennis Smith, SEI; and Guy Steele, Sun Microsystems, Inc.


10/150

Although these reviewers provided many constructive comments and

suggestions, they were not asked to endorse the nal conclusions or

recommendations presented in this report, nor did they see the nal drat

o the report beore its release. Responsibility or the nal content o this

report rests entirely with the author team.

The author team is also grateul or support provided by SEI sta:

David Carney, Suzanne Couturiaux, Pamela Curtis, David Gregg,

Laura Huber, Bob Krut, Melissa Neely, Ray Obenza, Daniel Pipitone,

Bill Pollak, Hal Stevens, Pennie Walters, Sharon West, Barbara White,

David White, David Zubrow, and the SEI Inormation Technology sta.

Acknowledgments


11/150

Executve Summary

The U. S. Department o Deense (DoD) has a goal o inormation

dominanceto achieve and exploit superior collection, usion, analysis, and

use o inormation to meet mission objectives. This goal depends on

increasingly complex systems characterized by thousands oplatorms,

sensors, decision nodes, weapons, and warghters connected through

heterogeneous wired and wireless networks. These systems will push ar

beyond the size o todays systems andsystems o systems by every measure:number o lines o code; number o people employing the system or

dierent purposes; amount o data stored, accessed, manipulated, and

rened; number o connections and interdependencies among sotware

components; and number o hardware elements. They will be ultra-large-

scale (ULS) systems.

The sheer scale o ULS systems will change everything. ULS systems will

necessarily be decentralized in a variety o ways, developed and used by a

wide variety o stakeholders with conficting needs, evolving continuously,and constructed rom heterogeneous parts. People will not just be users

o a ULS system; they will be elements o the system. Sotware and

hardware ailures will be the norm rather than the exception. The acquisition

o a ULS system will be simultaneous with its operation and will require

new methods or control. These characteristics are beginning to emerge in

todays DoD systems o systems; in ULS systems they will dominate.

Consequently, ULS systems will place unprecedented demands on sotware

acquisition, production, deployment, management, documentation, usage,

and evolution practices.

Fundamental gaps in our current understanding o sotware and sotware

development at the scale o ULS systems present proound impediments to

the technically and economically eective achievement o the DoD goal

o deterrence and dominance based on inormation superiority. These gaps

are strategic, not tactical. They are unlikely to be addressed adequately by

incremental research within established categories. Rather, we require a

broad new conception o both the nature o such systems and new ideas or

how to develop them. We will need to look at them dierently, not just as

systems or systems o systems, but associo-technical ecosystems. We will

ace undamental challenges in the design and evolution, orchestration and

control, and monitoring and assessment o ULS systems. These challenges

require breakthrough research.

Bold-italicormattingoawordorphraseindicatesthatitsdenitionappearsintheGlossary.


12/150

We propose a ULS systems research agenda or an interdisciplinary portolio

o research in at least the ollowing areas:

Human Interaction: involves anthropologists, sociologists, and social

scientists conducting detailed socio-technical analyses o user interactions

in the eld, with the goal o understanding how to construct and evolve

such socio-technical systems eectively.

Computational Emergence: explores the use o methods and tools based

on economics andgame theory (e.g., mechanism design) to ensure

globally optimal ULS system behavior and explores metaheuristics and

digital evolution to augment the cognitive limits o human designers.

Design: broadens the traditional technology-centric denition o design

to include people and organizations; social, cognitive, and economic

considerations; and design structures such as design rules and government

policies.

Computational Engineering: ocuses on evolving the expressiveness o

representations to accommodate the semantic diversity o many languages

and ocuses on providing automated support or computing the evolving

behavior o components and their compositions.

Adaptive System Inrastructure: investigates integrated development

environments and runtime platorms that will support the decentralized

nature o ULS systems as well as technologies, methods, and theories

that will enable ULS systems to be developed in their deployment

environments.

Adaptable and Predictable System Quality: ocuses on how to maintain

quality in a ULS system in the ace o continuous change, ongoing

ailures, and attacks and ocuses on how to identiy, predict, and control

new indicators osystem health (akin to the U. S. gross domestic product)

that are needed because o the scale o ULS systems.

Policy, Acquisition, and Management: ocuses on transorming

acquisition policies and processes to accommodate the rapid and

continuous evolution o ULS systems by treating suppliers and supply

chains as intrinsic and essential components o a ULS system.

Executve Summary


13/150

The proposed research does not supplant current, important sotware research

but rather signicantly expands its horizons. Moreover, because we are

ocused on systems o the uture, we have purposely avoided couching our

descriptions in terms o todays technology. The envisioned outcome o the

proposed research is a spectrum o technologies and methods or developing

these systems o the uture, with national-security, economic, and societal

benets that extend ar beyond ULS systems themselves.

Though our research agenda does not prescribe a single, denitive roadmap,

we oer three structures that suggest ways to cluster and prioritize groups o

research areas mapping the research areas and topics to (1) specic DoD

missions and required capabilities, (2) DoD research unding types required

to support them, and (3) estimates o the relative starting points o the

research. These structures can then be used to dene one or more roadmaps

that could lead to one or more ULS systems research programs or projects.

As a rst step, we recommend the unding and establishment o a ULS

System Research Startup Initiative, which over the course o the next two

years would, among other things,

work with others to conduct new basic research in key areas;

oster the growth o a community o inormed stakeholders andresearchers; and

ormulate and issue an initial Broad Agency Announcement (BAA) to

attract researchers with proven expertise in the diverse set o disciplines

(e.g., sotware engineering, economics, human actors, cognitive psychol-

ogy, sociology, systems engineering, and business policy) that are collec-

tively required to meet the challenge o ULS systems.

The United States needs a program that will und the sotware research

required to sustain ongoing transormations in national deense and achieve

the DoD goal o inormation dominance. The key challenge is the decision

to move orward. The ULS System Research Agenda presented in this report

provides the starting point or the path ahead.


14/150


15/150

Introduction

.

Geness o the

ULS Systems

Research Study

The oce o the Assistant Secretary o the U. S. Army (Acquisition,

Logistics, & Technology) (ASA ALT) unded the Sotware Engineering

Institute (SEI) to lead a 12-month investigation o ultra-large-scale (ULS)

systems sotware. ASA ALT posed this question to the SEI: Given the

issues with todays sotware engineering, how can we build the systems o

the uture that are likely to have billions o lines o code?

The intended outcome o the study was a proposed research agenda or

ULS systems; a proposal or a program that would und, coordinate, and

conduct needed research; and the creation o a collaborative research

network that would work toward solving the ULS system problem or the

U. S. Department o Deense (DoD).

Although a billion lines o code was the initial challenge, increased code

size brings with it increased scale in many other dimensions, posing

challenges that strain current sotware oundations. To understand thechallenges and the research needed to meet them, the study brought together

sotware experts and experts rom outside the sotware engineering eld

rom a variety o institutions and organizations. This multi-disciplinary

team sought solutions both within and beyond traditional sotware and

systems engineering disciplines. This report describes and justies the

ULS system research agenda that resulted rom the year-long study.

To appreciate the need or the study and the value o its output, it is

important to rst understand current DoD objectives and to analyze theundamental shortalls in todays sotware concepts, tools, and methods

or reaching those objectives.

Part I


16/150

Introduction

Introducton Characterstcs Challenges Research Areas Recommendatons

.

The DoDs Goal

o Inormaton

Domnance

The DoD has a goal o inormation dominanceto achieve and exploit

superior collection, usion, analysis, and use o inormation to meet mission

objectives. As articulated in the recent Quadrennial Deense Review Report,2

achievement o this goal depends on increasingly complex systems

characterized by thousands oplatorms, sensors, decision nodes, weapons,

and warghters connected through heterogeneous wired and wireless

networks. These systems will be ULS systems. The Global Inormation Grid

(GIG), the Army Future Combat Systems (FCS), and FORCEnet are

examples o emerging DoDsystems o systems intended to organize and

coordinate this large human, application, and technology space to

provide DoD planners with the right inormation at the right place and

the right time across a range o support systems and battleeld systems;

optimally and adaptively manage inormation resources to provide usable

target-quality inormation to warghters engaged in tactical operations;

and

conduct eective inormation operations even in the ace o attack, while

also denying inormation critical to adversaries.

This technology base must be capable o orchestrating the human, comput-

ing, and communications environment to aggregate, lter, and prioritize the

delivery o this inormation to work eectively in the context o transient

and enduring resource constraints and ailures. An essential property o

inormation dominance is continuous adaptation. Adaptation is needed to

compensate or changes in the mission requirements (such as rerouting strike

packages to engage time-critical targets and modiying target/weapon

pairings to avoid ratricide) and operating environments (such as dynamic

network topologies, publish/subscribe membership changes, and intermittent

connectivity).

.3

The Mssng

Key to Inormaton

Domnance

Although systems comprise ar more than sotware, it is sotware that

undamentally makes possible the achievement o the DoD goal o inorma-

tion dominance and the envisioned improvement in human and organiza-

tional perormance. At the same time, though, sotware also presents the

greatest impediment to DoD goals.

We have seen transormations in all acets o society that have been

catalyzed by advances in sotware technology. The revolution in business

practices eected by Google, or example, is directly traceable to advances

in sotware technologyspecically, new algorithms that enable a scalable

architecture or inormation searches.

Quadrennial Deense Review Report, February006,http://www.deenselink.mil/qdr/report/Report006003.pd


17/150

Yet, rom the perspective o the underlying science and engineering

knowledge base, sotware is the least well understood and the most

problematic element o large-scale systems. Sotware and sotware project

ailures are among the dominant causes osystem cost and schedule

overruns; o ailures osystems to satisy the requirements o those who

procure and use them; and, increasingly, o costly and dangeroussystem

ailures. Despite the careul application o modern sotware engineering

techniques, sotware ailure is ar more prevalent than hardware ailure

as a cause o major system outages. While some problems are caused by

poor practice, the root cause o most system problems is our inadequate

sotware knowledge base.

.4The Engneerng

Perspectve on

Sotware

Development

The problems presented by sotware have long been recognized. By the late1960s, it was clear that the sotware problem was real, signicant, and

growing rapidly. The NATO Conerence held in 1968 was a watershed event. 3

At that conerence, the community agreed to characterize sotware develop-

ment as an engineeringproblem and to establish as a goal that the computer

science research and development community solve the major open problems

by working to establish a new theoretical and practical discipline osotware

engineering.

In the nearly 40 years since 1968, tremendous progress has been made in theeld o sotware engineering. The net eect has made it possible to construct

increasingly complex systems. At the same time, as our sotware engineering

capabilities have grown, so have our aspirations, and our aspirations continue

to largely exceed our capabilities. That is, with respect to current large

sotware-intensive systems, our aspiration to establish sotware development

as an engineering discipline is, to a signicant extent, still an aspiration. As we

struggle to develop todays systems, we simultaneously aspire to develop the

ar more ambitious systems that we envision or the uture.

The Presidents Inormation Technology Advisory Committee (PITAC) clearly

stated that our current sotware science and technology base is inadequate to

meet current and uture needs;4 blue-ribbon panels have identied major

problems in sotware development as challenge problems in computer science;

and reports such as that by the Standish Group5 document an unimpressive

record in large sotware projects in the private sectoreven in the relatively

well-understood domain o business applications. In the Standish Groups

most recent report, only 34% o all projects were deemed to be successul.

3 Naur,P.&Randell,B.(eds.).Sotware Engineering: Report o a Conerence Sponsored by the NATO Science

Committee,Garmisch,Germany,7-Oct.968,Brussels,ScienticAairsDivision,NATO(969).

ThePresidentsInormationTechnologyAdvisoryCommittee. Inormation Technology: Investing in Our Future,nal

report.February999.

StandishGroup.The Chaos Report, 003.


18/150


19/150

as microeconomics, biology, city planning, and anthropologyelds

concerned with people as well as with coherence in the context o scale and

complexity.

.6

From Engneerng to

Complex Systems

Alan Kay7 amously said that the right perspective is worth 80 IQ points.8 For

40 years, we have embraced the traditional engineering perspective. The basic

premise underlying the research agenda presented in this document is that

beyond certain complexity thresholds, a traditional centralized engineering

perspective is no longer adequate nor can it be the primary means by which

ultra-complex systems are made real. Electrical and water systems are

engineered, but cities are notalthough their orms are regulated by both

natural and imposed constraints. Firms are engineered, but the overall

structure o the economy is notalthough it is regulated.Ecosystems exhibithigh degrees o complexity and organization, but not through engineering.

The protocols on which the Internet is based were engineered, but the Web as

a whole was not engineeredalthough its orm is constrained by both natural

and articial regulations. In this report, we take the position that the advances

needed or ULS systems require a change in perspective, rom the satisaction

o requirements through traditional, rational, top-down engineering to their

satisaction by the regulation o complex, decentralized systems.

.6.

From Buldngs

to Ctes

One way to understand the dierence in scale between traditional and ULS

systems is to think about buildings, inrastructure systems, and cities.

Designing and building most o todays large systems can be compared to

designing and constructing a single, large building or a single inrastructure

system (such as or power or water distribution). In contrast, ULS systems will

operate at levels o complexity more similar to cities. At rst it might seem that

designing and building a city is simply a matter o designing and building a

large number o buildings. However, cities are not conceived or built by

individual organizations, but rather by the actions o many individuals acting

locally over time. The orm o a city is not dened in advance by speciying

requirements; rather, a city emerges and changes over time through the loosely

coordinated and regulated actions o many individuals. The actors that enable

cities to be successul, then, include both extensive inrastructures not present

in individual buildings as well as mechanisms that regulate local actions to

maintain coherence without central control. These mechanisms include

government organizations and policies, city planning, streets and transportation

systems, communication and emergency services, and distribution o ood and

consumer goods, to name a ew. Moreover, it is not easible to design and builda city in one attempt. People, companies, communities, and organizations

decide to build parts o cities or their own purposes. Cities grow and thrive

7 AlanKayis,amongotherthings,thewinnerotheTuringAward,thehighestscienticawardintheeld

ocomputerscience.

8 PredictingTheFuture,reproducedromStanord Engineering,Volume,Number,Autumn989,pp.-6.


20/150

Introduction


based on cultural and economic necessities, and, although some aspects

o a city are designed and constructed in a local context, most elements that

make up the essence o a city arise rom more global policies and mecha-

nisms, such as zoning laws, building codes, and economic incentives

designed to encourage certain sorts o growth and construction.

Closer examination o the two design and construction problemsbuildings

versus citiesreveals that, although it is necessary, skill in designing and

constructing buildings does not help much in designing and constructing a

city. To point out just one o the major dierences: every day in every city,

construction is going on, repairs are taking place, modications are being

made, and yet the cities continue to unction. Like cities, ULS systems will

not simply be bigger systems: they will be interdependent webs o sotware-

intensive systems, people, policies, cultures, and economics.

.6.

From Systems to

Ecosystems

Another way to understand the needed

shit in perspective is in terms o the

concept o ecosystemsand o what we

call socio-technical ecosystems in

particular. Like a biological ecosystem, a

ULS system comprises a dynamic

community o interdependent andcompeting organisms (in this case, people,

computing devices, and organizations) in

a complex and changing environment. The concept o an ecosystem connotes

complexity, decentralized control, hard-to-predict eects o certain kinds

o disruptions, diculty o monitoring and assessment, and the risks in

monocultures, as well as competition with niches, robustness, survivability,

adaptability, stability, and health.

In a ULS system, there will be competition or resources, such as bandwidth,

storage capacity, sensors, and weapons. The system will enorce rules

intended to encourage eective use o these resources to achieve mission

objectives. There may be variations in service depending on how dierent

commanders, planners, and automated subsystems attempt to apply the

available resources to missions with dierent levels o importance and

urgency. With appropriate incentives and rules enorced by the system,

these resources will be optimized so that they are appropriately available.

In addition, there may be an overall measure o the quality o service

being provided to dierent parts o the system or or dierent purposes

(e.g., quality-o-service requirements will change or some parts o the

system during a mission). This measure can be used to determine i the

incentives are working as intended. As system behavior changes in response

to the incentives, the incentives may also need to be changed to ensure

that key mission goals are accomplished.


21/150

We traditionally view system development and acquisition as a technology-

centric, rational, largely top-down, centrally controlled process o implemen-

tation and deployment. By contrast, a dening element o ULS systems is

that they will include people, organizations, and technologies at all levels,

rom those responsible or overall policy implemented within the system

to those producing the system to those actually using it. There will be

organizations and participants responsible or setting acquisition, production,

and operationalpolicies governing the overall system, and there will be

organizations, technologies, and people responsible orproducingULS

systems. For example, the DoD, Congress, and the Oce o the Secretary

o Deense (OSD), in their roles as policy makers, will play essential roles

in an overall system. Contractors and warghters will participate at another

level o system organization. There are rich kinds o interdependencies

among players, systems, and activities across all o these levels.

The concept o an ecosystem is not novel. Nor is the analogy perect.

For example, unlike biological ecosystems, whose dynamics involve nutrient

and energy fows, ULS ecosystem dynamics involve exchanges o economic,

security, and other orms o value. Yet even with its imperections, the

analogy helps illuminate the nature o the problem acing the ULS system

developer: an inadequate understanding o how to create and maintain

systems that have characteristics similar to those ound in ecosystems.

The challenge acing the ULS systems research community is to help ll

these major gaps in knowledge.

.6.3

Beyond the Internet

The Web oreshadows the characteristics o ULS systems. Its scale is much

larger than that o any o todays systems o systems. Its development,

oversight, and operational control are decentralized. Its stakeholders have

diverse, conficting, complex, and changing requirements. The services it

provides undergo continuous evolution. The actions o the people making use

o the Web infuence what services are provided, and the services provided

infuence the actions o people. It has been designed to avoid the worst

problems deriving rom the heterogeneity o its elements and to be insensi-

tive to connection ailures.

But the Web was not designed with the DoDs needs in mind. Security was

not given much attention in its original design, and its use or purposes or

which it was not initially intended, such as e-commerce, has revealed

exploitable vulnerabilities. The elements o the Web are loosely intercon-

nected in the sense that the ailure o most elements does not have a

signicant eect on many users. The bandwidth available to the Web is much

greater than what is likely to be available or the DoDs ULS systems. And

although the Web is an important element o peoples work lives, it is not as

critical as a ULS DoD system would be.


22/150

Introduction


The existence o the Web suggests what ULS systems might be like, but

despite the Webs success and ubiquity, ULS systems or DoD purposes are

likely to push the boundaries o what we have learned (and are learning)

rom the existence o the Web.

.

The Results o

Ths Study

The DoDs goal o inormation

dominance, the sotware that

will be required to achieve that

goal, and the concomitant need

or a new perspective in

sotware development are all

undamental to the research

agenda that resulted rom ourstudy. Our agenda is meant to

catalyze a unded ULS system

research program and inspire

researchers rom a diverse set o disciplines to address ULS system sotware

challenges. We expect researchers to be motivated to investigate the

identied topics and to propose other research in those areas. Prosecuting a

research agenda o this magnitude is like planting many seeds. Some will

bear ruit; others will be weeded out and perhaps substituted with heartier

types. Similarly, over time we see this research agenda changing to matchbreakthroughs and results. We also recognize a complementary need to

establish a transition component o the research program that will ensure that

successul results get packaged and transitioned to the DoD as quickly as

possible. The rest o this report is divided into two parts as ollows:

Part I descrbes and justes our ULS system research agenda.

Section 2: Characteristics o ULS Systems examines the consequences o

scale implied by ULS systems. It characterizes ULS systems as complex

systems that are signicantly beyond the reach o the traditional engineering

paradigm o sotware development and provides the basis or understanding

the technical challenges that are inherent in these systems.

Section 3: Challenges in ULS Systems describes and analyzes the major

challenges posed by ULS systems.

Section 4: Overview o Research Areaspresents seven major research areas

and underlying topics that hold promise or addressing ULS system

challenges, thereby lling the most important gaps in sotware knowledge

and capability at the scale needed or ULS systems.


23/150

Section 5: Summary and Recommendations shows how the proposed

research topics are related to needed DoD capabilities and outlines a path

orward or a substantive, long-term, unded ULS System Research

Program.

Part II provdes more detaled normaton or techncally

orented readers.

Section 6: Detailed Description o Research Areaspresents each research

area and research topic in detail.

We also provide a Glossary at the end o the report. Words and phrases that

are dened in the Glossary are ormatted in bold-italic typeace when they

appear within the text.


24/150


25/150

Characteristics oULS Systems

There are characteristics o ULS systems that will arise because o their

scale. These characteristics undermine current, widely used, sotware

engineering approaches and provide the basis or the technical challenges

associated with ULS systems.

The primary characteristic o ULS systems is ultra-large size on any imagin-

able dimensionnumber o lines o code; number o people employing the

system or dierent purposes; amount o data stored, accessed, manipulated,

and rened; number o connections and interdependencies among sotware

components; number o hardware elements; etc. But to understand the nature

o ULS systems, we must go beyond just the concept o size; we must un-

derstand the eects o scale and the demands that ULS systems are likely to

place on technologies and processes. Issues that are not signicant at smaller

scales become signicant at ultra-large scales. The problems introduced by

scale require new solution approaches and new concepts o system design,

development, operation, and evolution. In short,scale changes everything.

ULS systems will have some characteristics in common with todays systems

o systems (SoSs). Mark Maier9 has developed a list o characteristics that

distinguish large monolithic systems rom systems o systems:

operational independence o the elements: Component systems are

independently useul.

managerial independence o the elements: Component systems are

acquired and operated independently; they maintain their existenceindependent o the SoS.

evolutionary development: The SoS is not created ully ormed but

comes into existence gradually.

9 Maier,MarkW.Architecting Principles or Systems-o-Systems.

http://www.inoed.com/Open/PAPERS/systems.htm,996.


26/150

Characteristics o ULS Systems


emergent behavior: Behaviors o the SoS are not localized to any

component system. The principal purposes o the SoS are ullled by

these behaviors.

geographic distribution: Components are so geographically distributedthat their interactions are limited primarily to inormation exchange

rather than exchanges o mass or energy.

Maier goes on to dene dierent classes o systems o systems based on the

amount o management control that is possible. The class he calls virtual

systems o systems is closest to ULS systems:

Virtual systems lack a central management authority.

Indeed, they lack a centrally agreed upon purpose orthe system-o-systems. Large scale behavior emerges,

and may be desirable, but the supersystem must rely

upon relatively invisible mechanisms to maintain it.

Maier then cites the Web and national economies as examples

o virtual SoSs.

Maiers denitions are useul or classiying systems but are not so useul

or understanding the underlying technical problems o ULS systems.

The characteristics o ULS systems that will arise because o their scale

are much more revealing. These characteristics are as ollows:

decentralization: The scale o ULS systems means that they will

necessarily be decentralized in a variety o waysdecentralized data,

development, evolution, and operational control.

inherently conicting, unknowable, and diverse requirements: ULS

systems will be developed and used by a wide variety o stakeholders

with unavoidably dierent, conficting, complex, and changing needs.

continuous evolution and deployment: There will be an increasing need

to integrate new capabilities into a ULS system while it is operating. New

and dierent capabilities will be deployed, and unused capabilities will

be dropped; the system will be evolving not in phases, but continuously.

heterogeneous, inconsistent, and changing elements: A ULS system

will not be constructed rom uniorm parts: there will be some mists,

especially as the system is extended and repaired.


27/150

erosion o the people/system boundary: People will not just be users o

a ULS system; they will be elements o the system, aecting its overall

emergent behavior.

normal ailures: Sotware and hardware ailures will be the norm ratherthan the exception.

new paradigms or acquisition and policy: The acquisition o a ULS

system will be simultaneous with the operation o the system and require

new methods or control.

Although these characteristics are not all independent and are already

evident in some o todays largest systems, each implies a change in the

undamental assumptions that underlie todays sotware engineering

approaches. Each contributes to the complexity o bringing ULS systems

into existence, validating their behavior, and evolving their capabilities.

Each derives rom the consequences o scale changing everything. In the

remainder o this section, we discuss each characteristic and the assumptions

that the characteristic undermines. Understanding how these assumptions

are undermined helps in understanding why ULS systems present new

challenges and underscores the need or new research.

.

Decentralzed

Control

The scale o ULS systems will allow only limited possibilities or centralizedor hierarchical control o data, development, evolution, and operation. Even

the limited amount o hierarchical control that is possible today or very

large systems will be challenged at the scale o ULS systems and likely

require dierent control models.

One o todays assumptions undermined by this characteristic is the ollowing:

All conicts must be resolved and must be resolved uniormly. Todays

systems are predicated on the idea that conficts must be addressed and

resolved. We assume that there is a confict-resolution process and an organi-

zation that makes decisions to be ollowed by other elements o the system.

The scale o ULS systems will make it impossible to resolve all conficts and

to resolve conficts centrally. In an ecosystem, there is no central authority

or resolving conficts; other mechanisms serve this purpose. An ecosystem

characterization o ULS systems suggests that mechanisms will be in place

to resolve conficts locally among those who have an immediate interest in

the issue. Moreover, i the same confict arises elsewhere in the system, it

might be resolved dierently. Todays designers o large systems generally

do not consider the idea that they should tolerate conficts or resolve the

same confict dierently at dierent times and places. They also do not

typically include procedures or determining which conficts are o greatest

importance to globalsystem viability; we have little understanding o what

organizational processes work best or this purpose.


28/150



.

Inherently Confctng,

Unknowable, and

Dverse Requrements

The scale and complexity o problems to be solved by ULS systems mean

that, in many cases, the requirements to be satised by the systems will not

be adequately known until the systems are in use. Even then, as the system is

put into operation, perceptions o the problem it is solving will change. Each

attempt at a solution will give a deeper understanding o what the problem

really is, leading to yet another attempt at a solution. Some problems ad-

dressed by ULS systems are likely to be so complex that there can be no ully

satisactory solution; requirements will never really converge.

ULS systems, by their very size and nature, will be able to serve a wide

spectrum o purposes. The more a ULS system does, the more diversity and

confict it is likely to engender. Moreover, integrating solutions across this

spectrum will likely require knowledge rom several domains, and making

use o relevant cross-domain knowledge is dicult today.

Among todays assumptions undermined by this characteristic

are the ollowing:

Requirements can be known in advance and change slowly as

experience with a system grows. We know that requirements or systems

today are nevercompletely understood in advance o building and using

a system. Even so, most systems today are developed on the assumption

that key requirements are suciently well understood that i a system is

built to meet these requirements, it will be useul. But ULS systems are

likely to encounter so-calledwicked problems10 in which requirements

are neither knowable in advance (because there is no agreement about

what the problem is) nor stable (because each solution changes the

problem, and no solution is considered to have solved the problem11).

In ULS systems, even more so than today, system development, opera-

tion, and usage will have to be based on a premise o continual change

and (re)negotiation o user needs.

Tradeo decisions will be stable. Because o the large number o ULS

system users, each having dierent goals, dierent tradeos in system

behavior may be appropriate or dierent groups o users, and even these

tradeos will change over time. For example, one group may place a

high value on ast response to certain types o queries, at a cost o having

an increased error rate; another group may value accuracy more than

responsiveness. It must be possible to congure the system as it appears

to specic groups o users so that both requirements can be met simul-

taneously. Although we sometimes give attention to the idea that certain

tradeo decisions will have to be revised over the lie o the system, we

0Rittel,H.&Webber,M.DilemmasinaGeneralTheoryoPlanning,pp-69,Policy Sciences,Vol.,Elsevier

ScienticPublishingCompany,Inc.,Amsterdam,973;alsohttp://en.wikipedia.org/wiki/Wicked_problems

Wickedproblemshaveincomplete,contradictory,andchangingrequirements;andsolutionstothemareotendicult

torecognizeassuchbecauseocomplexinterdependencies.http://en.wikipedia.org/wiki/Wicked_problems


29/150

do not generally build systems today with the idea that the system can be

readily recongured to support dierent tradeos or dierent users.

.3

Contnuous

Evoluton and

Deployment

Another consequence o size is that ULS systems will be in service or

a long time. Their size will make it impractical to replace or retire them.

Instead, like very large systems today, they will continuously evolve to meet

new and modied requirements and to incorporate new technologies. But

we envision a dierent type o evolution than is typical o todays very large

systems. By evolution, we mean change that is guided and constrained by

rules and policies that allow local needs to be satised in local ways without

destroying the integrity and value o the overall system. The evolution o a

ULS system will be supported by dierent subgroups o stakeholders, each

subgroup seeking to achieve a solution that ts its own needs, but under the

guidance o general economic, technical, and political rules that limit the

impact (both positive and negative) o any given change.

One o todays assumptions undermined by this characteristic

is the ollowing:

System improvements are introduced at discrete intervals (build-use-

build). As with cities, ULS systems must continue to unction despite

ongoing, simultaneous construction, repairs, improvements, and demolitions.

For cities, we have considerable experience in determining

how to make improvements without catastrophically interering with the

needs o the inhabitants. For most systems today, we do not typically

allow dierent subgroups to make changes simultaneously. In ULS systems,

however, the introduction oconcurrentchanges will generally be neces-

sary because waiting to sequence all changes will unacceptably delay the

introduction o needed capabilities and problem xes. A key issue will

be to know which changes can be made concurrently and which must be

coordinated.


30/150



.4

Heterogeneous,

Inconsstent, and

Changng Elements

The size o a ULS system also means that its elements (i.e., its hardware, its

sotware, its procedures and rules, its people, etc.) will be heterogeneous,

inconsistent, and changing.

Heterogeneous: Sotware elements will be heterogeneous in part because

they will come rom a variety o sources. Parts o the system will be written

in dierent languages, tuned or dierent hardware/sotware platorms,

and designed according to dierent philosophies and methodologies. Many

sotware elements will originate in legacy systems, written long beore the

rst ULS system comes into existence. Some will be dynamically supplied

and assimilated. Many will be services that will be provided over the Internet.

A key aspect o ULS system design, construction, and evolution will entail

integrating and compensating or such heterogeneous elements and engineer-

ing perspectives.

Heterogeneity can also be a benet. In heterogeneous systems, not all ele-

ments are equally vulnerable to ailures or attacks.

Inconsistent: With sotware originating in dierent places and being created

and modied by dispersed teams with dierent schedules, processes, goals,

and stakeholders, a ULS system will necessarily be composed o dierent

versions o the same sotware elements, possibly with inconsistencies in their

design, implementation, and usage. The system will be used in unanticipated

ways, giving rise to conditions that were not considered when some o its

constituent parts were designed. Changes in usage patterns will stress the

system, leading to clashes o assumptions about how long operations take,

how much storage is available, or how much data can be processed by a

particular algorithm. Dierent stakeholders will have dierent expectations

o how the system is to perorm.

Changing: Parts o the system will always be changing. The operatingenvironments will be changing as ailed hardware is replaced, hardware and

sotware are upgraded (and sometimes downgraded), and congurations o

components are modied. Many changes in ULS systems will be dynamic to

adapt to evolving mission requirements and operating conditions. As a result,

the exact composition o a ULS system cannot be known when it is being

designed and may vary rom moment to moment.


31/150


are the ollowing:

The eect o a change can be predicted sufciently well. When replac-

ing an element, we assume that we understand suciently well which

characteristics o the element are essential to the adequate unctioning o

the system and which are inessential. Because o this knowledge, we can

make replacements that are not exactly identical (e.g., dierent vendor,

dierent code, dierent perormance). The scale o a ULS system

increases the likelihood that some seemingly inessential dierence

between the original and the replaced units will, at least occasionally,

have signicant consequences. The system must be designed to limit the

eect o such unexpected consequences, at least when they are consid-

ered harmul.

Confguration inormation is accurate and can be tightly controlled.

When considering the eect o changes on a system, todays change

processes or smaller systems make the (mostly reasonable) assumption

that the conguration o the system is both known and knowable. In ULS

systems, conguration inormation will never be completely accurate,

and yet, despite the inaccuracies, changes must be installed eectively.

Components and users are airly homogeneous. In architecting and

implementing todays systems, we begin by believing that we understand

the capabilities o users and how they will use the system. In ULS

systems, actual usage will drit rom what was anticipated not just

because o normal human tendencies, but because o the scale and variety

o people involved with the system. Today we oten start with the premise

that we can control the range o hardware and sotware components that

are employed over the lie o the system. The scale o ULS systems will

make it more dicult to ensure that each element o the system is reason-

ably up to date in its sotware versions, hardware capabilities, etc. The

system design will have to take into account these kinds o disparities,

which are normally not considered today.

.5

Eroson o the

People/System

Boundary

People will not just be users o a ULS system: they will be part o its overall

behavior. In addition, the boundary between the system and user/developer

roles will blur. Just as people who maintain and modiy a city sometimes also

live within the city, in a ULS system, sometimes a person will act in the role

o a traditional user, sometimes in a supporting role as a maintainer osystem

health, and sometimes as a change agent adding and repairing the systemsunctions.

Considering people to be part o the ULS system means that as the systems

conguration and computational capabilities change, processes and proce-

dures must be in place to help people understand how to accomplish their


32/150



current objectives and become aware o new possibilities. Equally, as the

pattern o usage by people changes (or example, i a large number o people

attempt some action more or less simultaneously), the system must have

procedures in place to adapt to the changed demand. This kind o interplay

will be needed because the size o the system and the number o changes

will require the provision o methods or adapting quickly to changes in

expectations and capabilities, whether these changes are on the people side

or the hardware/sotware side. Increasing reliance on machine learningor

adaptation could be exploited by adversaries. Thereore a ULS system must

also be skeptical as it learns how to adapt to new situations.

The law o large numbers and the size o ULS systems potentially allow or

analyses leading to types o improvement and adaptation that are not easible

in smaller systems. For example, with a suciently large number o interac-

tions, the system can begin to gather data on observed regularities in peoples

behavior and begin to build statistically reliable models o what types o

services will be required. These opportunities arise rom considering people

as integral elements o the overall systems behavior. Although the behavior

o individuals can be dicult to predict in isolation, the behavior o large

groups is more amenable to analysis.


are the ollowing:

People are just users o the system. For some aspects o a ULS system,

it will be impossible to understand or analyze the eects o the system

without giving ull consideration to the behavior o people as elements

o it. The inclusion o human behavior in the analysis o overall system

unction is not new, but, in ULS systems, it will become more imperative

than or todays systems.

The collective behavior o people is not o interest. When designing

and analyzing ULS systems, their scale means that the collective behavior

o groups o users and developers will be a signicant actor in how the

system is used, viewed, and accepted.

Social interactions are not relevant. Todays emphasis in designing

inormation systems is on the technologyhow to make the system

suciently ast, reliable, unctional, etc. It is not oten the case that a

socio-technical perspective is taken. In ULS systems, desirable emergent

behaviors will partly be a unction o how groups o people make use othe technology and how the technology supports group needs. Failure to

take into account the conventions that govern peoples behaviors, ailure

to take into account the benets and problems associated with the actions

o masses o people, and ailure to treat people as a part o the system

would be a mistake.


33/150

.6

Normal Falures

Because the physical underpinnings o a ULS system will be vast, hardware

ailures will no longer be an exception. And because sotware components

will be stressed beyond their designed-or capabilities, sotware may also

behave in undesirable ways. Moreover, even when deects are low, requency

o use may result in a continuing, low-level occurrence o ailure events

somewhere in the system. For example, i a le transer protocol ails once in

a million uses but the transer protocol is used a million times a day, a ailure

will occur, on average, once a day. We can expect that, with ULS systems,

unusual situations and boundary conditions will occur oten enough that

something will always be ailing.

Because o the scale o ULS systems, they must be designed to cope with

ailures (o various kinds) as a continuous problem. This realization may

sound daunting or pessimistic, but we are amiliar with this eect in our

everyday lives. In a large city, there will be an occasional re, crime, or

accident oten enough that it is worth having re, police, and ambulance ser-

vices. But the occurrence o res, crimes, or accidents in any given building

is suciently rare that we dont have these services or each building. We

know that these problems are inevitable, so we can make them a part o the

community inrastructure. Similarly, in ULS systems, the types o ailures

that occur inrequently in smaller scale systems will occur requently enough

that the ULS system will have to support special capabilities or dealing with

them on a regular basis. This means not just designing systems to cope with

the consequences o such ailures, but designing them to contain the eects

o ailures and, to the extent possible, to give warnings beore the ailures

occur or have a widespread impact.

Among todays assumptions undermined by this characteristic are the

ollowing:

Failures will occur inrequently. The scale o ULS systems can increasethe number o ailures per unit o timeas system usage and size

increase, certain types o ailure will be normal. ULS systems must be

designed to limit how much o the ULS systems behavior is aected by

ailure.

Deects can be removed. Much o sotware engineering is devoted to

the prevention and detection o deects; yet in very large systems, it is

impossible to be sure that all deects have been removed, and, in practice,

more oten than not, systems are shipped and used despite the presence

o deects. ULS system design must pay more attention to ault tolerance

than is typically paid (except or high-assurance systems). In addition,

ULS systems, because o their scale, will almost certainly introduce new

types o aults that will require new ault-tolerance strategies.


34/150



.

New Paradgms

or Acquston and

Polcy

Because o its size, those responsible or making a ULS system possible

(managers, acquirers, developers, suppliers, legislators, etc.) will be unable to

comprehensively dene and control uncertain and ever-changing stakeholder

requirements. Any requirement or centralized, global control over changes

cannot possibly take all these dierent purposes into account, manage them

eciently, or allow or rapid changes in response to immediate needs. A

successul ULS system must have the ability to develop organically. Our

vision is not unbridled anarchy: as in cities, there will be rules, regulations,

zoning laws, governing ocials, and enorcement personnel to keep things

going, allowing citizens to adapt to meet local needs while maintaining a

viable overall structure.

The size o ULS systems will present a challenge to managers. I the actual

needs o system stakeholders can never be ully anticipated, the entire

contracting, design, and construction processno matter how closely man-

aged or quality controlwill all too oten result in systems that ail to meet

users expectations and needs.

One o todays assumptions undermined by this characteristic

is the ollowing:

A prime contractor is responsible or system development, operation,

and evolution. The centralized control implied by the usual prime-contractor

model is incompatible with ULS systems. For example, there is no prime

contractor in charge o the development, operation, and evolution o the

Internet, although there are organizations that have responsibility or some

portions o Internet capability. Despite this decentralized approach, the

Internet has evolved successully. O course, there is a certain amount o

centralization with respect to the development and evolution o interace

standards, but much o the development, operation, and evolution o Internet

services and core capabilities does not ollow a prime-contractor model.

.

Summary

Although todays systems have some o the characteristics identied in

this section, what will distinguish ULS systems is that they will have all

o these characteristics. Consequently, as we have noted above, many key

assumptions we make today will be undermined: these systems will surpass

the thresholds at which todays approaches will work even nominally.

To understand the challenges posed by ULS systems, we will need to

look at them dierently, not just as systems or systems o systems, but as

socio-technical ecosystems:socio-technicalbecause they are composed o

people and technology interacting in complex ways, andecosystem because

characterizing them in this way will give us the richest understanding o the

technical and management problems that will be encountered when building,

operating, and developing them.


35/150

3

Challenges InULS Systems

Section 2 described how the characteristics o ULS systems challenge the

undamental assumptions o todays sotware developers and acquirers. It is

clear that todays approaches to dening, developing, deploying, operating,

acquiring, and evolving sotware-intensive systems will not suce. The

success o ULS systems and the achievement o the missions that they are

intended to support depend on the development o new capabilities. I we

characterize ULS systems as cities or socio-technical ecosystems, we nd

that current knowledge and practices are geared toward creating individual

buildings orspecies. What we lack is a scientic understanding o, and

adequate methods and technologies or, eectively developing sotware-

intensive systems on the scale o whole cities orecosystems. These gaps in

knowledge and capability are strategic, not tactical. They are unlikely to be

addressed adequately by incremental research within established categories.

Rather, we need to develop a broad new conception o both the nature o

such systems and new ideas or how to develop them. Understanding the

demands that ULS systems will make is key to dening the research that is

needed or new solutions.

The challenges that we will ace in developing ULS systems are organized in

three broad areas:

1. Design and Evolution

2. Orchestration and Control

3. Monitoring and Assessment


36/150

3 Challenges in ULS Systems


3.

Desgn and

Evoluton

How do we systematically address the socio-

technical ecosystem characteristics o ULS

systems? How do we design the ecosystem

inrastructure, which includes the services

provided by and to the participants in the

system, the rules (both ormal and social)

guiding their behavior, the acquisition

practices, the supply-chain inrastructure,

economic issues, etc.? How do we design the organizational processes

responsible or producing and continually updating the designs o ULS

system components and integration schemes?

The scale o complexity and uncertainty in ULS system design will be so

great as to resist treatment by traditional development methods, which are

characterized by centralized control (true even o decentralized methods

such as open-source development) and by the testing o a small number

o hypotheses about what constitutes a good solution. The challenge will

be to nd new ways to harness and coordinate the design capabilities and

motivations not just o individual companies, prime contractors, and supply

chains, but o whole industries, within which competition or value will

drive much richer and more economical exploration o complex design

spaces. Developing and evolving architectures around which industries will

organize presents challenges that we are not equipped to understand today.

Maintaining the conceptual integrity o system designs in the context o

such decentralized design activities spread across the economy will present

challenges to current knowledge, tools, and methods. The design o the

industrial ecosystems, including incentive structures and sources o value

(including procurement practices) that drive them, requires new thinking and

integrative research across numerous disciplines. We need a new science to

support the design o all levels o the systems that will eventually produce

the ULS systems we can envision, but not implement eectively, today.

ULS systems will be deeply embedded in the real world. These systems

will comprise not only inormation technology (IT) components, but

also machines o many kinds, individuals and teams, diverse sensors,

inormation streams and stores (including verbal and non-verbal human

communications), and so orth. We have traditionally viewed sotware

as programming the computer components o such systems. We ace a

challenge in understanding and designing sotware in a new way: as the

programming oallo the inormation-processing mechanisms andbehaviors o complex ULS systems.


37/150

The characterization o systems as inormation-processing mechanisms

is not new; it dates back at least to Wieners concept ocybernetics.2

However, recent developments in areas such as computer science and

distributed cognition put us in a position to reconsider how inormation

is processed in truly complex systems and to design their underlying

inormation processes to make them sentient, adaptive, and eective in

perorming complex missions.

Finally, adding new capabilities to our systems today is a laborious process

o redesign and reengineering. ULS systems will require internal inrastruc-

ture and mechanisms to acilitate the development and introduction o such

improvements whether they are initiated by system designers, implementers,

and operators or the users and computational elements interior to the ULS

system. New theories o ULS system evolution will help to provide the rules

and mechanisms needed to acilitate eective evolution o these systems, but

the challenges in accomplishing this exceed our current change-management

research and practice.

Listed below is a sample o specic challenges in ULS system design

and evolution. Each one can be mapped to the characteristics described in

Section 2. For example, economics and industry structure deals with how to

align the structure o systems with industry elements and economic orces to

discover and meet key ULS requirements. This challenge relates to the de-

centralization andnew paradigms or acquisition and policy characteristics;

the issue is how to engage industrial partners in developing and evolving

ULS systems when a dierent acquisition and management control model is

needed. It relates to the characteristics oinherently conficting, unknowable,

and diverse requirements andheterogeneous, inconsistent, and changing ele-

ments because these actors aect how the contractual mechanisms will have

to work. The normal ailures characteristic aects the acceptance criteria or

contractor work.

economics and industry structure: How do we align design architec-

tures13 and industry structures to harness economic orces in the service

o discovering and meeting key requirements?

social activity or constructing computational environments:

How do we model interaction with a social context in a way that oers

guidance or how to design and support ULS systems?

NorbertWiener(89-96):WienersworkbeoreandduringWorldWarIIledtothepublishingo Cybernetics, or

Control and Communication in the Animal and Machinein98.Init,hedescribedanewwayolookingathowthe

worldunctionedbasedonhisresearchonthewayinwhichinormationistransmittedandprocessed.Hesawaworld

thatocusedoninormation,notenergyandondigitalornumericprocesses,notmachineoranalogprocesses.

3Bydesign architecturewemeanasetodecisionsthatpartitionsthetaskoproducingthecompletedesignora

systemintoasetolargelyseparablesubtasks.


38/150



legal issues: How will we resolve the legal issues that would today

prevent a ULS system rom achieving its ull potential? These issues

include licensing, intellectual property, and liability concerns that

arise due to the size and complexity o a ULS system that is developed

under multiple authorities. How will legal policies (e.g., regarding the

certication o security- and saety-critical components) adapt (i at all)

to the characteristics o ULS systems (e.g., to sel-recongurability as a

pervasive technical characteristic)?

enorcement mechanisms and processes: How do we create enorce-

ment mechanisms or the set o (legal, design, and process) rules that

support and maintain the integrity o the system? What structures are

required to negotiate exceptions to the rules so that the ULS system can

be adaptable without aecting its long-term sustainment?

defnition o common services supporting the ULS system: How

do we dene an inrastructure (a set o technological, legal, and social

services) that will be common to many elements o the ULS system?

rules and regulations: How will whole industries come together to

agree on rules and regulations to ensure overall coherence and quality

while still being suciently fexible to permit stakeholders to explore and

compete within rich design spaces?

agility: How can the groups responsible or ULS development, mainte-

nance, and evolution be kept suciently agile to respond eectively to

changes in requirements, system conguration, system environment, etc.?

handling o change: How can the processes or developing, maintaining,

and evolving a ULS system be adapted to handle in situ design change

and evolution rather than relying on static requirements preceding design

and implementation?

integration: How can we minimize the eort needed to integrate

components built independently by dierent teams, with dierent goals,

and at dierent times to create the current system?

user-controlled evolution: How do we provide components and

composition rules that give users the ability to create new, unplanned

capabilities?

computer-supported evolution: How do we provide automated

methods to evolve ULS systems?

adaptable structure: How do we create designs that are eective even as

requirements and the ULS environment change continually?

emergent quality: How do we organize processes or producing ULS

systems so that they converge on high-quality designs?


39/150

3.

Orchestraton and

Control

By orchestration we mean the set o activities needed to make the ele-

ments o a ULS system work in reasonable harmony to ensure continuous

satisaction o the mission objectives. Orchestration involves management

and administration but at a scale well beyond that o traditional, centralized,

relatively ne-grained controls. Orchestration requires a combination o

up-ront design, overall policy promulgation and enorcement, and real-time

adjustment o operating parameters.

Orchestrating a ULS system requires supporting interdependencies and

controlling the consequences o local actions with respect to their eect on

the emergent whole, even though each part o a system might be acting to

maximize its local utility. In a city, or example, when dierent groups want

more services than can be provided, there are procedures or deciding what

gets provided and to whom. The city governance authorities do not control the

actions o individual citizens, but they do speciy general rules o behavior

that are intended to minimize unnecessary confict, disruption, or uneconomic

use o city resources. Similarly, the policies and inherent capabilities that

constrain the interactions o the participants in a ULS ecosystem create a

ramework or the long-term viability and adaptability o a ULS system in a

world o changing missions, deployments, and required unctionality.

Orchestration is needed at all levels o ULS systems. At one level, the

activities o otherwise autonomous companies developing key technologies

or ULS systems will need to be orchestrated. At another level, the behaviors

o a ULS system in operation will need to allocate resources in real time to

satisy real-time mission objectives. At the highest level, orchestration could

both aect and be driven by doctrine, policy, appropriations, and procure-

ment practices.

This style o system management is undamentally dierent rom the way

we manage systems today. To succeed in developing and operating ULSsystems, we thus need new knowledge, technologies, and methods in the

ollowing areas:

online modifcation: How can necessary adjustments to a system be

made while the system is running, with minimal disturbance to user

services; how can the changes be propagated throughout the system when

necessary?

maintenance o quality o service: How can the overall quality o

service be maintained while enabling the fexibility to provide dierent

levels o service to dierent groups?

creation and execution o policies and rules: What policies and rules

lead to eective solutions despite divergent viewpoints o stakeholders?

How are such rules and policies created? How are they executed?


40/150



adaptation to users and contexts: How can the needs o users and

stakeholders be discovered and understood; how can those needs be

translated into execution-time modications and adaptations? How can

the contextboth the users context and the physical contextbe sensed,

captured, and translated into adaptations?

enabling o user-controlled orchestration: How do we provide com-

ponents and composition rules that give users the ability to adapt and

customize portions o the system in the eld?

3.3

Montorng and

Assessment

The eectiveness o ULS system design, evolution, and orchestration

has to be evaluated. There must be an ability to monitor and assess ULS

system state, behavior, and overall health and well being. The monitoring

and assessment o complex systems, and subsequent adjustment o systemparameters, is not a new idea. In a city, or example, sensors collect inorma-

tion about trac conditions, and this inormation is then distributed to those

to whom it is relevant, allowing people to select alternate routes around

trac jams, or example. In complex telecommunication, transportation,

and electrical distribution networks, continuous measurements are made o

system conguration, resource usage and demand, system and component

ailure status, etc. Measurements are taken by monitoring embedded sensors.

Assessment activities then determine what the measurements mean, such

as by simulating the uture health o the system or determining the need orcontrol or orchestration actions.

The criteria or success or overall health are dierent or ULS systems than

or smaller systems designed to accomplish a task that does not change as

the system is used. For example, consider criteria or the success o a city.

Dierent criteria are used by the governing bodies, dierent groups o

citizens, etc. I the power ails in one part o the city, the rest o the city may

not be overly inconvenienced, at least as long as the power ailure does not

last too long, impair some critical acility, or happen too oten. Because a

city provides distributed services to dierent groups o people, its success in

delivering services will depend in part on the expectations and needs o each

group and in part on the qualities o the delivered service. Understanding

such criteria is a critical aspect o the monitoring and assessment o a ULS

system.

The primary approach today to system monitoring and assessment is through

the use o metrics. We characterize the quality or unctionality o a system

by a set o metrics captured at critical probe points dened or the systems

constituent components and networks. This is ar rom adequate or ULS

systems. Our current measurement science is analogous to a small set o tests,

each like a specic X-ray or MRI scan or a broken bone, whereas we need

ensemble indicators o a ULS systems overall health, tness, and well being.


41/150

The scale, decentralization, distribution, and heterogeneity o ULS systems

will present challenges to eective monitoring and assessment. Among other

things, it is likely that some ULS system indicators should be statistical,

composite measures o a systems overall stateakin to the gross domestic

product or to measuring climate change by estimates such as the net loss or

gain o ice mass. To maintain a ULS system at a set o reasonable expecta-

tions, to infuence the direction o its evolution, and to assess and predict its

overall quality and eective unctionality, we will need to augment current

measurement approaches with additional new theories and practices. Finally,

because ULS systems are socio-technical systems with people as partici-

pants, ULS system indicators must refect the conditions not only o the

technological but also the human, organizational, economic, and business

elements o the system.

Examples o challenges associated with monitoring and assessment are the

ollowing:

defning the indicators: What system-wide, end-t

Documents

Ultra Large Scale Database