2013 Grand Challenges in M&S - Expanding Our Horizon

8/9/2019 2013 Grand Challenges in M&S - Expanding Our Horizon

1/6

Grand Challenges in Modeling and Simulation:Expanding Our Horizons

Simon J. E. TaylorICT Innovation Group

Department of Information Systems

and ComputingBrunel University, Uxbridge, UK

Margaret L. LoperInformation & Communications Lab

Georgia Tech Research InstituteAtlanta, GA, USA

Osman BalciDepartment of Computer Science

Virginia Tech

Blacksburg, VA, USA

David M. NicolInformation Trust Institute

University of Illinois at Urbana-Champaign, IL, USA

Wentong CaiSchool of Computer Engineering

Nanyang Technological University

Singapore

George RileySchool of Electrical and Computer

EngineeringGeorgia Tech

Atlanta, GA, USA

ABSTRACTThere continues to be many advances in the theory and practice ofModeling and Simulation (M&S). However, some of these can beconsidered as Grand Challenges; issues whose solutions requiresignificant focused effort across a community, sometimes withground-breaking collaborations with new disciplines. In 2002, thefirst M&S Grand Challenges Workshop was held in Dagstuhl,Germany, in an attempt to focus efforts on key areas. In 2012, anew initiative was launched to continue these Grand Challengeefforts. Panel members of this third Grand Challenge presenttheir views on M&S Grand Challenges. Themes presented in this

panel include M&S Methodology; Agent-based M&S; M&S inSystems Engineering; Cyber Systems Modeling; and NetworkSimulation.

Categories and Subject DescriptorsI.6.0 [Simulation and Modeling]: General

General TermsAlgorithms and Theory

KeywordsModeling and Simulation Methodology; Agent-based Modelingand Simulation; Modeling and Simulation Life Cycle; CyberSystems Modeling; Network Simulation.

1. INTRODUCTIONAfter several decades of progress, Modeling and Simulation(M&S) continues to produce advancements in theory and practice.There continues to be innovation in existing application areas ofM&S and new application areas continue to be identified. Someof these can be considered as Grand Challenges; issues whosesolutions require significant focused effort across a community,

sometimes with ground-breaking collaborations with newdisciplines. In 2002, the first M&S Grand Challenges Workshop

was held in Dagstuhl, Germany, in an attempt to focus efforts onkey areas (www.dagstuhl.de/02351). In 2012, a new initiativewas launched to continue these Grand Challenge efforts. The firstevent in this new phase of activities was the M&S Grand

Challenge Panel held at the 2012 Winter Simulation Conference[1]. This discussed issues including interaction of models fromdifferent paradigms, parallel and distributed simulation,ubiquitous computing, supercomputing, grid computing, cloudcomputing, big data and complex adaptive systems, modelabstraction, embedded simulation for real-time decision support,simulation on-demand, simulation-based acquisition, simulationinteroperability, high speed optimization, web simulation science,spatial simulation, and ubiquitous simulation. The second eventwas another Grand Challenge Panel that took place at theSymposium on Theory of Modeling and Simulation (TMS13)during SpringSim 2013 in San Diego [2]. This addressed a rangeof topics across big simulation applications (data, models,systems), coordinated modeling, human behavior, composability,sustainable funding, cloud-based M&S, engineering replicability

into computational models, democratization of M&S, multi-domain design, hardware platforms and education.

Panel members of this third Grand Challenge present their viewson M&S Grand Challenges. Themes presented in this panelinclude M&S Methodology; Agent-based M&S; M&S in SystemsEngineering; Cyber Systems Modeling; and Network Simulation.

2. OSMAN BALCI: Developing a New

Modeling and Simulation Methodology

2.1 OverviewM&S stands to be the only applicable technique in many cases for

bringing solutions to many complex problems. However, M&S iscurrently applied in an ad hoc manner without an underlying

holistic methodology. Seventeen types of M&S are used in dozensof different disciplines to show how diverse M&S is. However,despite the diversity, much of the underpinnings of the currentM&S methodologies date back to 1960s. Current M&Smethodologies do not sufficiently meet the requirements forsolving complex multifaceted problems. Development of aholistic M&S methodology for solving such complex problems

poses to be a grand challenge. This position statement justifies theneed for a new M&S methodology and identifies a set ofrequirements for its development.

Permission to make digital or hard copies of all or part of this work for

personal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and thatcopies bear this notice and the full citation on the first page. To copyotherwise, or republish, to post on servers or to redistribute to lists,requires prior specific permission and/or a fee.SIGSIM-PADS13, May 1922, 2013, Montral, Qubec, Canada.Copyright 2013 ACM 978-1-4503-1920-1/13/05...$15.00.

409


2/6

2.2 INTRODUCTIONAModelis a representation and abstraction of anything such as areal system, a proposed system, a futuristic system design, anentity, a phenomenon, or an idea. Modeling is the act ofdeveloping a model. Simulation is the act of executing,experimenting with or exercising a model or a set of models for aspecific objective (intended use) such as analysis (problemsolving), training, acquisition, entertainment, research, oreducation. Simulation cannot be conducted without a model and

modeling is an integral part of simulation. Therefore, we refer toboth modeling and simulation activities as M&S.

Many areas or types of M&S exist. Taxonomy of M&S areas ispresented in Table 1. Use of the 17 M&S areas listed in Table 1spans dozens of different disciplines for many objectives /intended uses. Each M&S area possesses its own characteristicsand methodologies, is applicable for solving certain problems, andhas its own community of users. Some M&S areas have their ownsocieties, conferences, books, journals, and software tools.

For a description of the M&S areas, the reader is referred to theACM SIGSIM M&S Knowledge Repository at http://www.acm-sigsim-mskr.org/MSAreas/msAreas.htm

2.3 THE NEED FOR A NEW M&S

METHODOLOGYEmployment of M&S as a technique to solve complex problemsincludes the M&S of many diverse systems (problem domains),each with its own unique characteristics. In the current state of theart, the M&S methodology that is effective for one problemdomain typically does not satisfy the needs of M&S for another

problem domain. However, many problem domains dictate thecreation of a simulation model that represents many diversesystems in an integrated manner.

Two paradigms have been primarily used for discrete M&Sdevelopment: procedural and object-oriented. Under the

procedural paradigm, discrete simulation models have beendeveloped using the following conceptual frameworks (a.k.a.world views, simulation strategies): activity scanning, event

scheduling, three-phase approach, and process interaction [3].These four conceptual frameworks were created in early 1960sand some of them are still being used. It is time for new ideas!

The object-oriented paradigm originated in SIMULA simulationprogramming language in 1967. Smalltalk, C++, Objective C,Java, and C# followed as the contemporary object-oriented

programming languages that are commonly used for softwaredevelopment today.

There is a need for a holistic M&S methodology to meet the grandchallenges we face today. The development of the holistic M&Smethodology poses to be a grand challenge because of therequirements stated below.

2.4 REQUIREMENTS FOR A NEW M&S

METHODOLOGYThe Merriam-Webster dictionary defines methodology as a bodyof methods, rules, and postulates employed by a discipline. Thenew M&S methodology should present identifiable techniques,approaches, and strategies, and provide effective guidance toM&S engineers, analysts, and managers. We provide somerequirements below under which the new M&S methodologyshould be developed.

Table 1. M&S Areas (Types)

A. Based on Model Representation

1. Discrete M&S

2. Continuous M&S

3. Monte Carlo M&S

4. System Dynamics M&S

5. Gaming-based M&S

6. Agent-based M&S

7. Artificial Intelligence-based M&S

8. Virtual Reality-based M&S

B. Based on Model Execution

9. Distributed / Parallel M&S

10. Web-based M&S

C. Based on Model Composition

11. Live Exercises

12. Live Experimentations

13. Live Demonstrations

14. Live Trials

D. Based on What is in the Loop

15. Hardware-in-the-loop M&S

16. Human-in-the-loop M&S17. Software-in-the-loop M&S

Top 10 requirements for the new M&S methodology:

(1) The new M&S methodology must be structured based on acomprehensive and effective M&S life cycle.

An M&S life cycle [4]:

Represents a framework for organization of theprocesses, work products, quality assurance activities,and project management activities required to develop,use, maintain, and reuse an M&S application from birthto retirement.

Specifies the workproductsto be created under thedesignatedprocessestogether with the integratedverification and validation (V&V) and quality assurance(QA) activities.

Is critically needed forprojectmanagement tomodularize and structure an M&S applicationdevelopment and to provide guidance to an M&Sdeveloper (engineer), manager, organization, andcommunity of interest.

Identifies areas of expertise in which to employqualifiedpeople.

Is required to show the V&V and QA activities asintegrated within the M&S development activities basedon the principle dictating that V&V and QA must gohand in hand with the M&S development.

Enables to view M&S engineering from the four Ps(Perspectives): Process, Product, People, and Project.

The author has developed such an M&S life cycle [4] based on hisexperience with DoD-related complex M&S development

projects. The new M&S methodology can be created based on thatlife cycle representation.

410


3/6

(2) The new M&S methodology must be applicable for providingeffective and integrated M&S-based solutions to complex

problems.

An effective M&S-based solution is the one that is sufficientlycredible, accepted, and used by the decision makers. The newmethodology should assist in reducing the M&S builders risk andM&S users risk.

M&S Builders Risk is the probability that the M&S

application is rejected although it is sufficiently credible andacceptable. The consequences of this risk will result in higher costand prolonged project duration.

M&S Users Risk is the probability that the M&S applicationis accepted in spite of the fact that it is not sufficiently credible.The consequences of this risk can be catastrophic since incorrectdecisions will be made based on the M&S results.

(3) The new M&S methodology must be a holistic methodologyapplicable for M&S of many diverse systems in an

integrative manner.

Many problem domains (universes of discourse) contain diversesystems embedded within each other forming a system of systems.Each system possesses its own characteristics, e.g., discrete,continuous, real-time, or distributed. Each system can require atype (area) of M&S listed in Table 1. The new methodology mustaccommodate as many M&S areas (types) as possible.

(4) The new M&S methodology must provide a unifying

conceptual framework throughout the entire M&Sdevelopment life cycle.

Using different conceptual frameworks from one phase of theM&S development life cycle to another increases the complexityof development and probability of inducing errors. The newmethodology must employ a conceptual framework that can beused for each life cycle phase.

(5) The new M&S methodology must enable network-centricM&S application development.

Two main reasons exist for creating an M&S application asnetwork-centric. (i) To be able to train geographically-dispersedpeople using M&S, the M&S application must be accessible overa network such as Internet, local area network, virtual privatenetwork, or Secret Internet Protocol Router Network (SIPRNET).(ii) M&S application execution can be improved by distributingthe execution of its components on different server computers atdifferent network nodes. The new methodology must enable theconstruction of a network-centric M&S application.

(6) The new M&S methodology must enable reuse andcomponent-based M&S application development using alibrary of reusable components.

Undoubtedly, the reuse provides significant economical andtechnical benefits that cannot be underestimated [6]. The new

methodology must enable the creation and use of a library ofreusable components specifically created for a problem domain(universe of discourse) of interest.

(7) The new M&S methodology must facilitate verification,

validation, and quality assurance throughout the entire M&Slife cycle.

V&V aims to assess the transformational accuracy (verification)and behavioral/representational accuracy (validation) of an M&Sapplication. QA aims to assess the other M&S qualitycharacteristics such as interoperability, fidelity, credibility, and

acceptability. V&V or QA is not a stage, but continuouslyconducted activities hand-in-hand with the development activitiesthroughout the entire M&S life cycle. The new methodology mustfacilitate the effective application of V&V and QA principles [5].

(8) The new M&S methodology must support integration with

real-life hardware and software systems for performing LiveExercises.

Live exercises, experimentations, demonstrations, and trials are

very much needed for solving some complex problems such asemergency response management training, military training, andtechnology assessment. It is critically important that the M&Sapplication be integrated with real-life systems.

(9) The new M&S methodology must support real-time decisionmaking.

An M&S application can be embedded within a real-time decisionsupport system to enable real-time decision making during, forexample, an emergency situation.

(10)The new M&S methodology must enable the development ofM&S applications for Analysis as well as for Training

objectives.

Analysis(problem solving) is conducted under such objectives as

comparison of different operating policies, evaluation of a givenemergency response management plan, prediction, and sensitivityanalysis. Trainingrefers to simulation-based training of, e.g., firstresponders and decision makers. The same methodology must beable to be used for developing an M&S application for multipleintended uses (objectives).

2.5 CONCLUDING REMARKSThe U.S. Government is the largest sponsor and consumer ofM&S applications in the world. Billions of dollars are spentannually by the U.S. Government for developing, using, andmaintaining M&S applications. However, the M&Smethodologies in use today date back to 1960s. It is criticallyimportant to conduct research to develop new methodologies toaddress the ever-increasing complexity of M&S development.

Reuse has been very difficult or in some cases impossible in theM&S discipline. However, the issue of reuse is extremelyimportant and should be effectively addressed [6].

Component-based M&S development is an unsolved problem forstand-alone M&S applications due to differences in programminglanguages, operating systems, and hardware. However, network-centric M&S development promises new advances in component-

based M&S engineering and should be fully investigated.

3. WENTONG CAI: Agent-based Modeling

and Simulation for Complex Adaptive SystemsTraditionally M&S involves the development of a dynamic,stochastic, and discrete model of the physical system of interest.Once the model is shown to exhibit the expected behaviour for agiven set of known inputs, it can be used to predict the system

behaviour for a set of unknown scenarios or inputs. For manysystems which are well understood, or whose behaviour can becharacterized by a chronological sequence of events, thisapproach to modelling has been very successful. There ishowever a large set of systems in which a global or macro scaleunderstanding is simply not possible to conceive. Simulating andmodelling of these systems in a traditional manner is often toochallenging as system level trends and properties are difficult tocharacterize.

411


4/6

Fp

aa

p

csi

t

s

i

b

ipa

ta

a

o

e

cf

b

ac

at

a

ta

ac

t

e

I

c

o

bf

t the heart ofor studying ahenomena, age

pproach of desnd letting the serspective this

omplex adaptiystem is the reter-connected

nderstanding itramatic increasomains [8].ere are still fu

ehaviour Mo

imulation is detith respect todividual entiti

odelling decisiased agent mo

odel that incors required. Creosts a great cha specific doma

ith the currento obtain behavnalytics techniq

erification an

gent-based M&ndesirable conhe central argu

f the system ehis unpredictabffect relationshi

he most effecti

ompare the simrom the systeme applicable for

s there is usuomplexity and

lso typically leere are data a

ata is not a strai

calability. In ciewed as an in

nd thread of coepending on taditional viewgents that are i

pplications ageapabilities andhether agente scale and c

xecuting these

nteroperability

onstructed byowever, these

perate at diffifferent modell

etween differeor example, t

gent-based mod attemptingt-based model

ribing rules anstem level prois an attracti

e systems, wult of a hugeindividual c

is clear to seein its applicati

hile agent-basedamental and c

elling. Theermined by theits ability to ces in a typic

on is the leveldel is sufficien

orates social,ting a complex

llenge. This usn and extracti

wealth of dataour rules induues.

Validation.S to understandequence when

ment for agent-

erges in somle nature of the

ps a very challe

ve approach to

ulation resultsbeing modelled.agent-based M

lly no existingarrangement of

d to an overlyailable, evolvightforward task

omputational teependently exe

ntrol. The comeir model. Iof large num

nteracting with

ts themselves ctherefore sig

odels involve homplexity of t

odels extremel

and Reusabilit

sub-models creodels may no

rent levels ofng paradigms.

t models diffice crowd mod

el is the princito understandoffer an attrac

parameters oferties emerge.e method for

ere the overalumber of decismponents [7]

hy agent-baseon in a wide varM&S offers sallenging issue

effectivenessintegrity of the

apture the truel environment.

of details: whett or a complex

sychological, aagent behavioually requires stg the essence

availability, thetively from th

The major motcomplex adaptiascertaining vased model is t

unknown andsystem makesging issue.

establishing m

f the model wi However, this&S for complex

system to cothe huge numb

arameterized mg model parameither.

rms an autonocuting process,

plexity of agentn agent-based

bers of simpleeach other. H

an have significificant processieavyweight or lie studied syst

y compute inten

. A large-scal

ated using thework with eacabstraction a

This makes t

lt. In a typicall needs to pa

le of emergennatural compltive and intuiti

individual entitiFrom a modellireasoning abo

behaviour ofions made by t. Given t

M&S has seeety of applicatich great promi.

of agent-bassimulation mocharacteristics

An importa

er a simple rul agent behavio

d cultural factmodel itself aldying theories

rom the theori

re is a possibili data using d

ivation for usive systems haslidity of modehat the behavio

unexpected wtracing cause a

del validity is

th the output dapproach mayadaptive syste

pare with. Ters of agents w

odel. So, eventers to match t

ous agent canith its own st

s can vary wideM&S there is(or lightweigwever, for ma

ant computationg requiremenghtweight agenems often mak

sive.

ed model can

existing modeother: they md created usie communicati

crowd simulatioss along vario

e.exe

esg

ut

aeis

an

se

edelofnt

e-ur

rssoofs.

tyta

gans.ur

y.d

to

taots

eill

ife

ete

lyat)y

alts.ts,es

e

ls.aygn

n,us

informunderstindivid

differeand acabstrac

The gemultipl

differeartificideveloof commentio

basedorchestresearc

4.

Sim

CyclMany

in scieeducati

set ofin neecomple

will into creneededdesign

Systemusers'designoperati

shownprojectthe un

system

We caperspedevelosystem

througdeactivto a si

ation about theand the eventuals can deter

t crowd modetion, and maytion.

nerality of agene disciplines.

t communitieal intelligence,ed solutions a

putational modned challenges

&S for complrated effort ish communities.

ARGARE

lation in th

f the most imp

nce - energy,on. The Natio

rand challenge of engineerinx systems, and

vitably use Mte solutions toto focus on hod and managed

s engineering ieeds into thethrough an ite

onal system [1

in Figure 1, reps development.erstanding of t

s.

Figure 1.

also think abtive. A lifement activitie

s follow a ge

developmentation. The systple life cycle

actions of inds in the simine how to res

s may use diffrespond to e

-based M&S hAs a conseque

(such as coand M&S) h

d implementedelling. Obviouand to push the

ex adaptive sysrequired acros

LOPER:

e Systems

ortant problems

infectious disal Academy of

in broad realmg solutions [9].finding solutio

S as a problem- these problew complex engover their life c

an interdiscipliefinition of a

rative process]. The system

resents the sequ. The V procehe complexity

ystems Engine

out systems encycle is a cats into distinct,neral life cycl

and testing to dems engineerinodel [11], as sh

ividuals and allated worldond to these e

erent models oents at differe

s seen its applicce researchers

putational socave addressedsystems to enaly, to deal witresearch forwa

tems, a well-ors multiple dis

odeling

ngineerin

facing the worl

ase, climate cngineering has

s of human conAll of these ps to these gran

solving techniqs, systems en

ineering projectycles.

nary approach tystem, its archhat results in

engineering

ence of steps ins was designedassociated with

ering V Proces

ineering fromgorization ofcontrollable p

e from concep

eployment, opeV process ca

own in Figure 2

so needs too that thevents. But,

perceptionnt level of

ation acrossworking in

ial science,challenges,le this typethe above-

rd in agent-

ganized andiplines and

Life

d are rooted

hange, andidentified a

ern that areroblems are

challenges

ue. In orderineering iss should be

o translatingitecture andan effectiveV process,

a product orto simplifydeveloping

s

a life cyclehe systemshases. Allt definition

rations, andbe mapped.

412


5/6

spp

oa

a

e

p

p

Fcee

ef

pa

pb

ai

p

pba

ra

l

Figu

&S has becoystems engineerogram managlanning, result

anufacturing,pportunities tond when appr

chieved: cost suality, and cost

o solve the gr

se systems enge as an M&Shat types of

ngineering life

evelopment, enroduction ande do have this&S tools and s

hases for solvi

or example,onstructive simxamine alternatvaluate cost of

ngineering anidelity, physicseveloping protanufacturing

esign, and inveo opportunity

roduce detailend put them

erform? Forringing high aifferent levelso that early en

esign and devectually interactput on system

final system orohibitive to m

grand challensed in each ofhat types of m

hase. Once thietter M&S toolcross different

. DAVID

odels ofodel validati

epresents a systnd difficult as

ead to disastroolumbia were

e 2. Systems E

e an importaing life cycle tement, designprediction, su

and logisticsuse M&S in thopriately appli

avings, acceleravoidance [12].

nd challenge p

neering to desicommunity doM&S are bescycle phase (i.

gineering anddeployment, an understanding,olutions across

g these grand c

e commonlylations (e.g., w

ive solutions, lodifferent conce

manufacturin-based simulatiotypes. Onceevelopment p

stments have befor changes).

designs and pin an operati

xample, thisd low fidelityf resolution, inugh in the syst

lop more effectwith the systerequirements (

nce it has beeke).

ge for M&S isthe systems e

odels and simu

s is better unders and techniquehases of the sy

ICOL: U

yber Systen - determiniem of interest -ect of M&S.

s results. Prodamaged durin

ngineering Lif

t tool acrossinclude, requirand engineeri

plement to tes

support. Te systems engid, four major

ated schedule, i

oblems of our t

gn and manage't have a good

used/needed., concept defi

manufacturid operations an we can desigthe systems eng

allenges.

se low-fidelityar-games) in cook at operationpts. Later in t

g developmentons for detailewe are in thease we have

en made in a spWhat if we h

ototypes earlieonal setting t

ight lead tosimulations toa quick and easems engineerin

ive solutions, b early-on, anhere now user

produced and

then to understgineering lifeations are most

stood, we wills that can be retems engineeri

certainty a

sg whether ais widely recoFailure to prop

tective tiles onliftoff by a sm

Cycle

ll phases of tements definitiog, efficient t

t and evaluatio

here are maeering life cycbenefits can

mproved prod

imes, we need

the solutions, bunderstandingin each systeition, technolo

g developmed support). On better-integratineering life cyc

, aggregate-levcept definitionl effectivenesse life cycle (i.

) we use hig design and fengineering acommitted to

ecific solution (ad the ability

in the life cyc see how th

the challengeether that are

y way. If we cprocess, we c

y letting the us have substantiinteract with t

changes are co

and how M&Sycle phases, aeffective in ea

e able to develsed or integrat

g life cycle.

nd Trust i

model correctnized as a crucierly validate m

the space shutall piece of foa

en,stn,

ye,e

ct

to

utofsy

t,ceedle

el,toore.,

h-orda

soto

leey

ofatanan

erale

t-

isd

ch

ped

lyalay

lem

insulatiAs parmodel

effectsimulatof dam

numbeflawed,other s

The faend sy

As oucompusystem

cyber-smodelsmany

contambe closin the

The firEverydata ormeani

engineOur whow esimply

in the

The se

validatwhich

paramenetwor

drop pinterestmeasur

A dettrustw

Our wtraffic

differe(e.g.,differe

A third

used hdiffereuse. T

predict

model.Uncer

seeks tpropagparticuThe p

quantifmodelsdevelomodels

of unce

on from a fuelt of a team thavalidation (in r

of the foamsed, but the resage would ens

of sub-model and values foub-models and

ulty sub-modeltem results.

r nations defing and coms grows in criti

imulation mod? The problem

sources for untinate a systemed lest the cybeeat of a cyber-

st gap relates tmodel dependsparameters useg. Data trustw

ering accuracyork in simulatiasy it is to intlacking good i

odels, e.g., refl

cond gap is tr

on needs to bethe abstractionsters. For exak switches wit

olicies if (a) kn is lacking, aes of interest (

ailed model urthy than a mo

rk in analyzinhas shown that

t switching pCP) are veryt switches.

gap is trust in

ave different lt levels of truhe final questiions of the mo

This is clostainty Quantifi

o explain howate through clar those of cyth forward her

ication relies o. The key noting methods o

to parameters

rtainty quantifi

tank, and the sht reviewed NAesponse to the

impact on thlts suggested tue. Post-disast

s involved in tr model paramtheir paramete

and parameter

ensive capabilimunication, sical importance.

els? How cas are deep and

ustworthiness todel. There

r-equivalent ofependent confli

trust in mode

on parametersd have the samorthiness depen

and on levelsg wireless co

roduce uncertaiformation abo

ective coefficie

st in model fo

quantified conof the modelmple, why incltheir variation

owledge of swid (b) the im

e.g., application

sing uncertainre abstract mod

abstractions innetwork utilizat

olicies, but pruch affected b

model compos

vels of trustwt relative to th

on asks how trdel of sub-mod

ely related tocation in conti

ncertainty andmposition.er-systems - ne

e is difficult b

n derivatives tion though isquantifying se

nd data may le

ation for cyber

uttle burned upA developed s

disaster), I lear

e tiles had acat only an acceer analysis rev

he simulationters were unkrs had accepta

s contaminated

ties rely incrulation of c

How trustwor

we developcan be subtle, a

o creep in, prore serious gapshe Columbia dict.

ling parameter

nd processes dlevel of certai

ds on data pro

of confidence imunications hnty in detailedt the paramete

ts.

mulation. P

sideration of tatch the availde detailed su in scheduling

ches used in tact of queuin throughput) is

parameters mel using trusted

switching andion is largely u

tocols involvipacket drop

tion. Data an

rthiness, sub-e data and parastworthy is thels that make

a well-developnuous system

error in continDiscrete-systeed a corresponcause classical

at are lackingmodel sensitivsitivity of metrd to a correspo

simulations.

on re-entry.tandards oned that the

tually beenptable levelealed that a

ere deeplyown, whilele validity.

the end-to-

asingly onyber-centricthy are our

trustworthynd there are

pagate, andthat need tosaster occur

s and data.ata. Not allty, trust, orenance, on

n that data.s shown usmodels by

s called for

rt of model

e degree toable data or-models of

and packet-

e system ofpolicy on

significant?

ay be lessparameters.

backgroundaffected by

g feedbackolicies, and

parameters

odels havemeters theyend-to-end

p the entire

ed field ofmodels that

ous modelsmodels -

ding theory.uncertainty

in discreteity, and soics in cyberding theory

413


6/6

6. GEORGE RILEY: Grand Challenges in

Network SimulationThe use of discrete event simulation methods to attempt to predict

performance of telecommunications networks has been animportant part of nearly all research in computer networks. Innearly all cases it is simply too expensive and time consuming toattempt to create an actual network to experiment with and studynetwork behavior under controlled conditions. However, the use

of network simulation tools is also fraught with pitfalls anddifficulties such that it becomes difficult to draw meaningfulconclusions from many simulation studies. There are severaldifficult problems that need solutions in order to continue thewidespread use of network simulation tools.

Performance and Scale. Virtually all network simulation toolsquickly run in to performance issues when modeling any non-trivial network. As the size of the network topology grows andthe link capacity increases, the total number of simulation events

per simulated second quickly leads to excessive running times,sometimes several hours or even days per simulated second.Clearly, meaningful research is difficult to achieve when waitinglong time periods between experimental runs. Further, processormemory also grows quickly as the topology size grows, oftenleading to experiment failure due to memory exhausting.Distributed simulation methods on supercomputers or networks ofworkstations can ease this problem somewhat at the expense ofextra simulator overhead for message exchanges and timemanagement.

Accuracy of simulation parameters. The behaviors of thevarious models in the simulation environment are intended toreflect accurately the behavior of the same element in a real-worldnetwork. However, in many cases it is simply not known theactual internal behavior of the element in question in a realnetwork (queuing discipline, queue size, queue size units, as anexample). Thus when trying to understand the effect of proposedchanges on a real network the observed metrics from the simulatormight not match the actual metrics on the real network due toincorrect assumptions about how the existing network is

configured.

Accuracy of simulation models. Clearly, models in a networksimulation environment are intended to behave identically to thesame network elements in real systems. However, models forcomplex protocols such as TCP are difficult to create and difficultto compare to actual networks. The number of different TCPvariations and behavior in deployed systems is surprisingly large,and it is nearly impossible to determine a priori which of the manyvariants are in use in a network. Further, models for queuingmethods also have a number of configuration parameters (see theRED queuing method for an example) for which the modeleroften does not know the correct or accurate settings. In short, the

behavior of nearly all network element models in networksimulators is intended to mimic some real world system for which

the modeler has incomplete or inaccurate information.

Modeling physical layer performance in wireless simulations.Network simulation has become nearly ubiquitous in the arena ofwireless networks. It is well known that the performance of suchnetworks is a function of many variables such as transmission

power, antenna gain and orientation, network density, routingprotocol in use, node mobility, and of course the network trafficdemand by the applications. Of these, the physical layercharacteristics are among the most important, and unfortunately

accurate modeling of the PHY layer is extremely challenging andnot very well understood. Many network simulation designershave created models for PHY layer behavior based on well-knowntheory, only to find that real-world experiments lead to vastlydifferent results.

The field of network M&S has been active for decades, withbigger and better tools being developed. Progress has been made,but the use of network simulation to accurately predict networkperformance still leads to inaccurate conclusions in many cases.The above list of challenges is by no means complete, but is agood starting point for researchers wishing to create bettersimulation tools.

7. CONCLUSIONSThe panel members of this third M&S Grand Challenges activity

present their views on why multifaceted M&S Methodology,Agent-based M&S, M&S in Systems Engineering, Cyber SystemsModeling, and Network Simulation pose serious methodologicaland technical challenges that are considered as grand challenges.

8. REFERENCES[1] Taylor, S.J.E., Fishwick, P.A., Fujimoto, R., Page, E.H.,

Uhrmacher, A.M., Wainer, G. 2012. Panel on Grand Challengesfor Modeling and Simulation. InProceedings of the WinterSimulation Conference 2012. ACM Press, NY.

[2] Taylor, S.J.E., Khan, A., Morse, K.L., Tolk, A., Yilmaz, L,Zander, J. 2013. Grand Challenges on the Theory of Modelingand Simulation. InProceedings of the 2013 Symposium on theTheory of Modeling and Simulation. SCS, Vista, CA. To appear.

[3] Balci, O. 1988. The Implementation of Four ConceptualFrameworks for Simulation Modeling in High-Level Languages.InProceedings of the 1988 Winter Simulation Conference. ACMPress, NY. 287-295.

[4] Balci, O. 2012. A life cycle for modeling and simulation.Simulation: Transactions of the Society for Modeling andSimulation International.88, 7, 870883.

[5] Balci, O. 2010. Golden rules of verification, validation, testing,

and certification of modeling and simulation applications. SCSM&S Magazine. Oct. 2010 Issue 4, The Society for Modelingand Simulation International (SCS), Vista, CA.

[6] Balci, O., Arthur, J. D., and Ormsby, W. F. 2011. Achievingreusability and composability with a simulation conceptualmodel.Journal of Simulation5, 3, 157-165.

[7] Holland, J. 1999.Emergence, From Chaos to Order. BasicBooks.

[8] Siebers, P. O., Macal, C. M., Garnett, J., Buxton, D., and PiddM. 2010. Discrete-event Simulation is Dead, Long Live Agent-

based Simulation!Journal of Simulation, 4, 3, 204-210.

[9] National Academy of Engineering. 2008. Introduction to theGrand Challenges for Engineering. Website accessed 15February 2013: http://www.engineeringchallenges.org.

[10] Committee on Pre-Milestone A Systems Engineering. 2009. Pre-Milestone A and Early-Phase Systems Engineering: ARetrospective Review and Benefits for Future Air ForceAcquisition. The National Academies Press.

[11] Stevens, R., Brook, P., Jackson, K., and Arnold, S. 1998.Systems Engineering: Coping with Complexity. Prentice Hall.

[12] Haskins, Cecilia (ed.). 2007. Systems Engineering Handbook: AGuide for System Life Cycle Processes and Activities, INCOSE-TP-2003-002-03.1, version 3.1.

414

Documents

2013 Grand Challenges in M&S - Expanding Our Horizon