Agent-based modelling of complex systemsprac.im.pwr.wroc.pl/~szwabin/assets/abm/lec/10.pdf · make it so simple that there are obviously no deficiencies, and the other way is to make

Agent-based modelling of complex systems

Janusz Szwabi skiń

Lecture #10: Verification, validation and replication

Outlook

● Correctness of a model● Model verification● Model validation● Model replication● Recommendations for model replicators and authors● ODD protocol

Motivation

Essentially, all models are wrong, but some are useful.George Box (1976)

The word model sounds more scientific than the word fable or tale, but I think we are talking about the same thing.

Ariel Rubinstein (2006)

There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult. It demands the same skill, devotion, insight, and even inspiration as the discovery of the simple physical laws which underlie the complex phenomena of nature.

Tony Hoare (1981)

Correctness of a model

● a model useful for answering real-world questions should provide outputs that address relevant issues

● outputs should be accurate● outputs must be useful for the model user● model accuracy can be evaluated through three different

processes:● verification ● validation● replication

● demonstrating that one particular set of results from a model corresponds to the real world is not sufficient

● multiple runs are needed● methodologies of testing the accuracy often rely upon

statistical methods

Verification of a model

● as models grow larger, it becomes more difficult to simply look at their code to determine whether they are actually carrying out their intended functions

● verification addresses this issue● goal: elimination of “bugs” from the code● not as simple as it sounds, especially if the model designer and

implementer are different people● general guideline for enabling verification:

● begin with a simple model● expand complexity of the model only as necessary

Verification of a model

● a simple model is easier to verify than a complex one● if the additional parts added to the model are incremental in

nature, they will be easier to verify as well● important note: even if all components of a model are

verified, the model itself may not be (additional complications may arise from interactions between different parts of the model)

Verification - communication

● often the model implementer and the model author are the same person

● sometimes a team of people builds a model:● domain experts describe the conceptual model● other team members implement the model

● communication between them is critical to ensure that the implemented model correctly reflects the conceptual one

● for the verification of the model it is useful if each group gains some familiarity with the domain of the others


● imagine we are approached by a group of political scientists who want to develop a simple model of voting behavior

● based on their observations of polls and the election results they think that people’s social interactions largely determine voting in elections:● people have some initial political preferences and they

express them when initially polled● in the interval between when they are polled and when they

actually cast their vote, they talk to colleagues, neighbors etc this may change their preferences→

● this may happen several times during the period before election

● they ask us to build a model of this phenomenon


● it would be useful if the political scientists knew e.g.:● the difference between Moore and von Neumann

neighborhood● what a small-world network looked like

● this knowledge helps to make informed decisions about how their conceptual model should be implemented

● it would also help if the implementer had a basic idea of how voting mechanisms are conceptualized within the discipline of political science

● communication of conceptual models is prone to human errors and misunderstanding

Verification – describing conceptual models

● as we begin to implement the voting model, we might realize that there are some mechanisms and agent properties that we did not understand completely

● we decide to write a document describing how we plan to implement the model verification if the conceptual models →are the same

● flowchart:● a way to describe conceptual models in more formal terms ● it depicts the flow of decisions that occur during the operation

of a piece of software code


Initializeagents torandomvotes

Count finalvotes

Color agentsbased on vote

Agents count thevotes of neighborsand change their

vote to the majority

Has the system

stabilized?

N

Y


● flowcharts may easily be rewritten in pseudocode:

Voters have votes = {0,1}

For each voter:Set vote either 0 or 1, chosen with equal probability

Loop until election:For each voter:

If majority of neighbors’ votes = 1 and vote = 0 then set vote = 1Else if majority of neighbors’ votes = 0 and vote = 1 then set vote = 0If vote = 1 then set color blueelse if vote = 0 then set color green

Display count of voters with vote = 1Display count of voters with vote = 0

End loop


● goal of pseudocode – serve as a midway point between natural language and formal programming language

● it can be read by anyone, regardless of his/her programming skills

● at the same time, it contains algorithmic structure (easier implementation)

Verification testing

● after designing the conceptual model with political science experts, we can start coding

● we start simple incrementally verifying the alignment between our conceptual model and the code

● consider the following setup procedure:


import matplotlib.pyplot as plt

import networkx as nximport random as rd

n = 100p = 0.1k = 2

def init(): g = nx.erdos_renyi_graph(n, p) for nd in g.nodes(): g.node[nd]['state'] = rd.choice(range(k)) return g


● we can create a small test that examines whether the code created the correct number of voters:

def check_init(g): ones = 0 zeros = 0 for n in g.nodes(): if g.node[n]['state'] == 1: ones = ones + 1 else: zeros = zeros + 1 if abs(ones-zeros) > 0.1*len(g): print('Warning! Difference in initial voters greater than 10%') else: print('Init procedure working fine!')

check_init(init())


● we can inspect the result of the setup by eye:color = {1 : 'red', 0 : 'blue'}def draw(g): positions = nx.spring_layout(g) nx.draw(g, with_labels = False, pos = positions, node_color = [color[g.node[n]['state']] for n in g.nodes()], vmin = 0, vmax = k - 1) plt.show()

G = init()draw(G)


● not all errors are visually aparent● writing verification tests is important● it is often good practice to write tests even before writing the

code● in our example: rather than use 10% of the population as a rule

of thumb, we could use statistical tests to check if the difference between 2 populations is greater than e.g. three standard deviations

● unit testing – writing small tests that check whether individual units are working correctly in the code


● unit tests allows us to modify the code without the fear that our changes will undetectably disrupt the previous version of the model

● “in-model” unit testing (“online” unit testing) – the tests are performed while the model is running

● “offline” unit testing:● a separate set of unit tests that are not a part of the model● tests run the model with particular inputs and check if the

outputs are correct● further reading (regarding unit testing of software): R. Patton,

“Software testing” (2nd ed.), 2005, Sams

Beyond verification

● despite the best efforts to verify that the implemented model corresponds to the conceptual one, the model will sometimes produce results different from the authors’ and implementers’ expectations

● it may become clear over time that the surprising results are consequences of individual-level decisions encoded in the model

Beyond verification

● coming back to our voter model example:

Beyond verification

● the results may confuse our political science colleagues:● they expected the model to coalesce with voters forming

static blocks with smooth edges● we got jagged edges and a model that never reaches

equilibrium – it cycles through a set of states● after further examination of the model we find that:

● the voters change their vote if the vote count of the neighbors is tied

● voters on the edge of the concentrated blocks keep cycling back and forth between the two states

● we found the reason for the jagged edges and the cycling of the model

Beyond verification

● after changing the behavior in case of tied votes, the model reaches an equilibrium and the edges between blocks are smooth

Beyond verification – sensitivity analysis and robustness

● it is important to check how sensitive our model is to the particular set of initial conditions

● types of sensitivity analysis:● altering input values of the model (e.g. the percentage of

green agents in the voter model)● varying model parameters● changing the environments (e.g. the topology of an

underlying interaction network)● definition of sensitivity depends a little bit on the context● sensitivity in the quantitative results does not necessarily mean

sensitivity in the qualitative results


● in the voter model:● if our primary result is the forming of solid islands of voters,

this qualitative result remains true if we perturb the initial distribution of the voters by 10% in either direction the →model is insensitive to small changes in the initial distribution (the result is robust)

● if our primary result are the sizes of the islands, it is sensitive to changes in the model


Verification benefits

● correspondence between the conceptual model and the implemented one

● if the two models correspond exactly, then the model authors understand the models’ low-level rules

● helps to understand the mechanisms that underlie the phenomenon being explored

● without going through verification process the authors cannot be confident in the conclusions drawn from the model

● helps to understand the “why” question (e.g. cycling of the voter model)

Verification issues

● difficult to achieve:● it is challenging to determine whether a surprising result is:

● the product of a bug in the code● the product of a miscommunication between the model author

and implementer● an unexpected outcome of low-level rule

● isolating and fixing the problem may be hard● even if we are sure the results is unexpected but accurate, it could

be difficult to discover the cause● verification is not binary:

● a model is not either verified or unverified● it exists rather along a continuum of verification● it is always possible to write more tests or to carry more sensitivity

analyses● no objective criteria when to stop

Validation of a model

● a process of ensuring that there is a correspondence between the implemented model and reality

● complex, multilevel and relative by nature● models are simplifications of reality impossible for a model to →

exhibit all of the same characteristics and patterns that exist in reality

● when creating a model we want to incorporate the aspects of reality that are pertinent to our questions

● it is important to keep the conceptual model questions in mind and validate aspects of the model that relate to these questions


● different levels at which validation may occur:● microvalidation – making sure that the behaviors and

mechanisms encoded into agents in the model match up with their real world analogs

● macrovalidation – ensuring that aggregate, emergent properties of the model correspond to aggregate properties in the real world

● the level of details in the validation process:● face validation – showing that the mechanisms and properties of

the model look like mechanisms and properties in the real world● empirical validation – ensuring that the model generates data

that correspond to similar patterns of data in the real world

Example – flocking model

● based on the original Boids model (Reynolds, 1987)● bird flocks can arise without being led in any way by special leader

birds● each bird is following exactly the same set of rules, from which flocks

emerge: ● alignment – bird turns so that it is moving in the same direction as

nearby birds● separation – bird turn to avoid hitting another bird● cohesion – bird moves toward other nearby bird● separation rule overrides the other two

● each bird moves forward at the same constant speed● rules are robust and can be adapted to swarming of insects, schooling

of fish, and patterns of flocking such as the “V” flocking of geese (Stonedahl and Wilensky, 2010)

Example – flocking model


● real birds have locations and headings, and the are aware of the birds around them based on their vision our model is face valid in regard to these →properties (sufficient to describe flocking behavior)

● real birds can be young or old, hungry or well fed, sick or healthy our →model is invalid in this regard● we may add other properties to the model to check if they change the

resulting behavior of the system; if not we have evidence that the property is irrelevant for the asked question

● real birds move in 3D, our agents in 2D● it is difficult to determine in advance whether 2D is sufficient or not● however, the notion of a flock carries over to 2D and it is possible to

imagine how a flock in 2D would look like● we provisionally assume that it is fine to model flocks in 2D● we can also build a 3D version of the model and compare the results


● our model shows ● emergence of flocks● interaction between flocks (one flock loosing a few birds to the other)● similar phenomena are observed in the real world

● the model is successfully macrovalidated● other test are possible, e.g. calculating the average size of the flock● in other traditional modeling forms (e.g. equation-based modeling) only

macrovalidation is carried out aggregate results of the model are compared →to aggregate observation results

● ABM produces data at all levels of aggregation validation at any level is →possible


● there will be rarely just one correct interpretation of our questions of interest leading to a single correct ABM

● often there are many equally good reality-to-model mappings● to use the model to make claims about the world, there has to be a

defensible connection between the two● face validation helps to ensure that someone just looking at the model

(without detailed analysis) can easily be convinced that the model contains elements that correspond to those existing in the real world

● in the flocking model:● agents appear to behave like birds● there are no behaviors that seem unreasonable for a bird to enact

● face validity can exist at both levels:● microvalidity – determining if agents correspond to real birds● macrovalidity – determining if the emergent flocks correspond to the

appearance of real flocks


● empirical validations sets a higher bar● data produced by the model must correspond to empirical data derived

from the real-world phenomenon● comparison based on the measures and numerical data● usually involves running statistical tests on the comparable data sets to

determine how similar they are● challenge: inputs and outputs in the real world are often poorly defined● it can be hard to isolate and measure parameters from a real-world

phenomenon● it is possible to show that the flock model generates many numerical

similarities to real flocks:● angular momentum, size and velocity of the flocks


● model calibration – finding parameters and initial conditions that cause the model to match up with real-world data● involves using empirical data to adjust parameters and mechanisms of a

model so that it can be used to examine a particular situation● pattern-oriented modeling – a particular use of empirical validation

(Grimm et al, 2005)● by matching patterns of empirical data at many levels the model author is

able to create a more valid model ● if patterns of behavior were matched at every single level and for every

type of behavior, the model would be a perfect replica of reality


● Important note! Do not get too caught up in validation!!!● the goal of a model building is to answer a question and explain a result, not

to simulate all of the aspects of a phenomenon

To stimulate, not to simulate.M. Resnick (1994)


● ABM are usually stochastic in nature● the same results are rarely produced twice given the same initial parameters● a problem when attempting to validate a model● in many cases reality will only present itself with a few instances of a

phenomenon we would like to model only few data points for our set of →initial conditions

● running the model repeatedly may produce thousands of different results● sometime only one of them may match the results of a real system● this is where statistical testing comes in

● our model is creating a distribution of the results● standard statistical tools (t-test, Kolmogorov-Smirnov test) help us to

determine if the distribution is consistent with the real one


● ABM may produce so many different end results that any comparisons to real phenomena become very difficult

● the output of such models may be divided into 2 classes:● invariant results – occur no matter how many times we run our model● variant results – change depending on how the model evolves

● for example, in a model of urban growth:● the western section of the city will always be developed● in 50 percent of the runs, the norther section may develop and the

southern not, and vice versa● such a partitioning of the result space helps to cut down the amount of data

needed for validation


● path dependent process – the history of the process greatly affects its final state

● usual cause for variant results● path dependence often identifies leverage points in the system:

● there is an area of ecological sensitivity south of the city● ABM shows that the south and north are equally likely to be developed● it would make sense for urban planners to offer incentives to develop north

of the city in the early stages of city growth● the model can tell us how to manipulate the world

Artificial Anasazi model

● an ABM of the Kayenta Anasazi (now called Kayenta Ancestral Pueblo) – ● a group of people that was living in the Long House Valley in the Black Mesa

region of modern-day Arizona● people lived in this area 1800 BCE● they disappeared suddenly around 1300 CE● extensive archeological record from about 200 CE until their disappearance

● Artificial Anasazi project – an attempt to discover why this group suddenly vanished from Black Mesa

● a model particularly renowned for the level of empirical validation● validation was carried out by comparing the results of the model to the

historical data● it is now believed that the model explains the disappearance of the real

group of people – a combination of environmental and social causes


● archeological estimates of the population of the Anasazi group are very rough● they are based on counts of rooms and buildings, estimates of how long

these buildings were used and how many people were living in a building● for each site excavated the starting year of occupation, the end day of

occupation and the maximum number of households is estimated● many of these estimates are based on wood pieces found in the remains of

the buildings and comparing these pieces with the tree ring data base to determine when a piece of wood was cut

● no evidence is found of occupation of the valley after 1300


● J. S. Dean, G. J. Gumerman, J. M. Epstein, R. L. Axtell, A. C. Swedlund, M. T. Parker and S. McCarroll (2000), Understanding Anasazi Culture Change Through Agent Based Modeling in Dynamics in Human and Primate Societies: Agent Based Modeling of Social and Spatial Processes, T. Kohler and G. Gumerman (eds.), Santa Fe Institute. New York & London: Oxford University Press.

● the model basically simulates households who perform agriculture and try to feed themselves

● no social hierarchy, no exchange of resource, no kinship networks, etc● the only interaction of households is indirectly via the occupation of a site for

agriculture● agents have a unique spot for agriculture on the landscape, and live on a

nearby spot● the climate variability results in some years being more agriculturally

productive than others


● households can store excess corn for up to three years● if a household does not get enough food it will look for another spot● if no suitable spot is available, they leave the valley● no agents enter from outside the valley● households also have their own demography

● they produce offspring (other households from their children)● they die of old age

● each cell represents a 100 meter by 100 meter space● each cell is within one of the different zones of land: General Valley Floor,

North Valley Floor, Midvalley Floor, Arable Uplands, Uplands Non-Arable, Kinbiko Canyon or Dunes

● these zones have agricultural productivity that is determined by the Palmer Drought Severity Index (PDSI)


● process overview (every year):● calculate the harvest for each household● if agent derives not sufficient food from harvest and storage or the age is

beyond maximum age of household the agent is removed from the system● calculate the estimated harvest for next year based on corn in stock and

actual harvest from current year● agents who expect not to derive the required amount of food next year will

move to a new farm location● find a farm plot● find a plot to settle nearby● if a household is older than the minimum fission age, there is a probability

pf, that a new household is generated. The new household will derive an endowment of a fraction of the corn stock

● update water sources based on input data● the household’s age increases by one year


● the model reproduces the archeological data quite well● model analysis shows that the abandonment of the valley around 1300

cannot be explained solely by environmental variations● the best fit (shown in the plot) was achieved by tweaking a number of

parameters (the fertility probability, the variance of harvest among plots with the same soil, the maximum age of a household, etc.)


Replication of a model

● one of the foundational components of the scientific method (B. Latour and S. Woolgar, Laboratory Life: The Social Construction of Scientific Facts, Sage Publications, Bevery Hills 1979)

● in order for an experiment to be considered acceptable by the scientific community, the scientists who originally performed the experiment must publish the details of how the experiment was conducted

● this description can help subsequent teams of scientists carry out the experiments themselvesto confirm the results

● this process confirms the fact that the experiment was not dependent on any local conditions


● replication is as important within the realm of computational models as it is within the realm of physical experiments

● replicating a physical experiment helps to prove that the original results are not due to mistakes or oversights in execution

● replicating a computational model serves this same purpose● additionally it increases our confidence in the model

verification since a new implementation of the conceptual model has yielded the same results as the original

● replication can aid in model validation as it requires the model replicators to try to understand the modeling choices that were made and how the original modelers saw the match between the model and the real world

Ethnocentrism model

● proposed by Axelrod and Hammond (2002)● this model suggests that “ethnocentric” behavior can evolve

under a wide variety of conditions, even when there are no native “ethnocentrics” and no way to differentiate between agent types

● agents compete for limited space via Prisoner Dilemma’s type interactions

● “ethnocentric” agents treat agents within their group more beneficially than those outside their group

● the model includes a mechanism for inheritance (genetic or cultural) of strategies

What is a Prisoner’s dillema?

● a standard example of a game analyzed in game theory● it shows why two completely "rational" individuals might not cooperate, even

if it is in their best interests to do so● it was originally framed by Merrill Flood and Melvin Dresher in 1950● Albert W. Tucker formalized the game with prison sentence rewards and

named it "prisoner's dilemma", presenting it as follows: Two members of a criminal gang are arrested and imprisoned. Eachprisoner is in solitary confinement with no means of communicating with the other. The prosecutors lack sufficient evidence to convict thepair on the principal charge. They hope to get both sentenced to a yearin prison on a lesser charge. Simultaneously, the prosecutors offer eachprisoner a bargain. Each prisoner is given the opportunity either to:betray the other by testifying that the other committed the crime, or tocooperate with the other by remaining silent. The offer may be describedby the following payoff matrix:



● it is implied that the prisoners will have no opportunity to reward or punish their partner other than the prison sentences they get, and that their decision will not affect their reputation in the future

● all purely rational self-interested prisoners would betray the other, because betraying a partner offers a greater reward than cooperating with them

● so the only possible outcome for two purely rational prisoners is for them to betray each other

● the interesting part of this result is that pursuing individual reward logically leads both of the prisoners to betray, when they would get a better reward if they both kept silent

● in reality, humans display a systemic bias towards cooperative behavior in this and similar games

Ethnocentrism model

● each agent has three traits: ● color● whether they cooperate with same colored agents● whether they cooperate with different colored agents

● an “ethnocentric” agent is one which cooperates with same colored agents, but does not cooperate with different colored agents

● an “altruist” cooperates with all agents● an “egoist” cooperates with no one● a “cosmopolitan” cooperates with agents of a different color

but not of their own color

Ethnocentrism model

● at each time step, the following events occur:● new agents appear in random locations with random traits (up to a

given number)● agents start with a chance of reproducing● each pair of adjacent agents interact in a one-move Prisoner’s

Dilemma in which each chooses whether or not to help the other:● they either gain, or lose some of their potential to reproduce

● in random order, each agent is given a chance to reproduce● offspring have the same traits as their parents, with a chance of each

trait mutating● agents are only allowed to reproduce if there is an empty space next

to them● agents has a chance of dying, making room for future offspring and

immigrants

Ethnocentrism model


● the model by Axelrod and Hammond suggests that “ethnocentric” behavior can evolve under a wide variety of conditions, even when there are no native “ethnocentrics” and no way to differentiate between agent types

● Wilensky and Rand tried to replicate it in 2007● in the process of replication differences were discovered

between the replicated model and the original one● the replicated model had to be modified to produce the original

results● the replication effort ultimately succeeded but the process that

was required to change the model was complicated and involved some unforseen problems

● Wilensky and Rand extracted some principles for model authors and replicators based on their experience

Replication of a model – dimensions and standards

● an original model and an associated replicated model can differ across at least dimensions:● time● hardware● languages● toolkits● algorithms● authors

● more than one of these dimensions are usually varied in the course of a single model replication


● time: ● a model can be replicated by the same person on the same

hardware and in the same language and environment, but rewritten at a different time

● the least likely change to produce significantly different results● different results would indicate that the published specification is

inadequate, since even the original researcher could not re-create the model from the conceptual model

● the only dimension that will always be varied● hardware:

● given the prevalence of hardware independent languages, changes in this dimension should not produce different results

● if the results are different, technical investigations are warranted and could point for example to the model being susceptible to small changes in the order of behavior execution


● languages: ● often the syntax and semantics of a language have a significant

impact on how a researcher translates the conceptual model into an actual implementation

● change in this dimension can highlight differences between the conceptual model and its implementation

● even minor details like floating point arithmetics or different implementation of protocols can cause significant differences in replicated models

● models widely accepted are robust to such changes● toolkits:

● even if the language remains the same the replicator may want to use a different toolkit (e.g. pycx vs pyabm in Python, Repast vs MASON in Java)

● differences in results can illuminate issues not only with the conceptual model but also with the toolkits themselves


● algorithms: ● a model could be replicated using different algorithms● e.g. there are many ways to update a large number of objects● a replicated model may simply carry out the steps of a model in an

order different from that of the original model● all of these differences can potentially create disparities in the

results● on the other hand, it is possible that the results will be the same

despite the differences in the algorithms● authors:

● strong test of a model’s replicability● if another researcher can reproduce the results, we have reasonable

evidence that the model is accurately described and the results are robust to changes in the dimensions of replication that have been altered


● a successful replication – the replicators are able to establish that the replicated model creates outputs sufficiently similar to the outputs of the original

● exact the same results are often not required● it is more important to show that similar changes generate similar effects● replication standard – criterion by which the replication’s success is judged● determinig the standard is left up to the researcher conducting the replication● different standards exists● according to Axtell, Axelrod, Epstein and Cohen (1996):

● numerical identity – difficult to establish (the same numerical results)● make use of specified random seeds to minimize the risk of failure

● distributional equivalence – similarity according to statistical measures● relational alignment – qualitatively similar relationships between input

and output variables


● once a replication standard has been chosen:● only data that is central to the conceptual model should be

measured and tested during replication● choose appropriate functions on a subset of output variables to be

the measures of replication● choose how often the results will be compared (at the end of the

run, at the end of every time step etc)

Benefits of replication

● successful replication shows that the original results were not exceptional or statistical abnormalities

● the knowledge and data from the original model is available to the model replicator

● supports model verification and validation● development of shared understanding – creation of a set of terms,

idioms and best practices that can be utilized by model authors to communicate about their models

● helps to develop a language for describing the modeling process

Recommendations for model replicators

● think about the replication standard – what criteria will be used to determine whether a replication has been achieved

● check the description of the conceptual model and decide whether to communicate with the authors and/or implementers of the original model

● it is beneficial to delay contacting the original authors until after a first attempt to recreate the original model has been made

● by first constructing the model a replicator provides a valuable service by noting which parts of the conceptual model are not sufficiently described in the original paper

● become familiar with the toolkit in which the original model was written

● obtain the source code of the original model (if possible)● again, wait with the inspecting the source code to avoid groupthink

Recommendations for model replicators

● details to be included in published replications:

Recommendations for model authors

● provide a detailed description of the conceptual model● details of the specification may not appear to be important for the

original author, but are important for a model replicator● publish at least the pseudocode of the model● carry out sensitivity analysis of the model● examine your model through the lens of a potential replicator

Recommendations for model authors

ODD protocol● V. Grimm, U. Berger, D. L. DeAngelis, J. G. Polhill, J. Giske, S. F. Railsback, The

ODD protocol: A review and first update, Ecological Modelling 221 (2010) 2760–2768

● ‘ODD’ (Overview, Design concepts, and Details) protocol was published in 2006

● primary objectives:● to standardize the published descriptions of individual-based and agent-

based models (ABMs)● to make model descriptions more understandable and complete, thereby

making ABMs less subject to criticism for being irreproducible● two emerging and unanticipated benefits:

● ODD improves the rigorous formulation of models● it makes the theoretical foundations of large models more visible

● originally designed for ABMs, it can help with documenting any large, complex model, alleviating some general objections against such models.

ODD protocol

ODD protocol● purpose:

● every model has to start from a clear question, problem, or hypothesis● ODD starts with a concise summary of the overall objective(s) for which the

model was developed● avoid to describe anything about how the model works here

● entities, state variables, and scales:● an entity is a distinct or separate object or actor that behaves as a unit and

may interact with other entities or be affected by external environmental factors

● its current state is characterized by its state variables or attributes● a state variable or attribute is a variable that distinguishes an entity from

other entities of the same type or category, or traces how the entity changes over time (weight, sex, age, hormone level, social rank, spatial coordinates),

● entities of an ABM are thus characterized by a set, or vector of attributes, which can contain both numerical variables and references to behavioral strategies

ODD protocol● entities, state variables, and scales (continued):

● if state variables have units, they should be provided● state variables should be low level or elementary in the sense that they

cannot be calculated from other state variables● most ABMs include the following types of entities:

● agents/individuals● spatial units (e.g., grid cells)● environment - while spatial units often represent environmental

conditions that vary over space, this entity refers to the overall environment, or forces that drive the behavior and dynamics of all agents or grid cells (e.g. temperature)

● collectives - groups of agents can have their own behaviors, so that it can make sense to distinguish them as entities

ODD protocol● process overview and scheduling:

● who (i.e., what entity) does what, and in what order?● when are state variables updated? ● how is time modeled, as discrete steps or as a continuum over which both

continuous processes and discrete events can occur? ● except for very simple schedules, one should use pseudo-code to describe

the schedule in every detail● in this ODD element only the self-explanatory names of the model’s

processes should be listed: ‘update habitat’, ‘move’, ‘grow’, ‘buy’, ‘update plots’

● these names are then the titles of the submodels that are described in the last ODD element ‘Submodels’

● processes are performed either by one of the model’s entities (for example: ‘move’) or by a higher-level controller that does things such as updating plots or writing output to files

ODD protocol● design concepts

● do not describe the model per se (not needed to replicate a model)● may be crucial for interpreting the output of a model● they are not described well via traditional model description techniques

such as equations and flow charts● included in ODD as a kind of checklist to make sure that important model

design decisions are made consciously● unused concepts can be omitted in the ODD description● there might be important concepts underlying the design of an ABM that

are not included in the ODD protocol● if authors feel that it is important to introduce a certain new concept to

understand the design of their model, they should give it a short name, clearly announce it as a design concept not included in the ODD protocol, and present it at the end of the “Design concepts” element

ODD protocol● design concepts – basic principles:

● general concepts, theories, hypotheses, or modeling approaches that underlay the model’s design

● relationship between these basic principles● scope of the principles (local or at the system level)

● design concepts – emergence:● key results or outputs of the model that are modeled as emerging from the

adaptive traits or behaviors of individuals● specify results that are expected to vary in complex and perhaps

unpredictable ways when particular characteristics of individuals or their environment change

● design concepts – adaptation:● specify adaptive traits of the individuals● rules for making decisions or changing behavior in response to changes in

themselves or their environment

ODD protocol● design concepts – objectives:

● measures that are increased by adaptive traits● criteria used by agents for ranking alternatives while making decisions● some synonyms: ‘fitness’ for organisms or ‘utility’ for economic reward

● design concepts – learning:● specify how agents may change their adaptive traits over time as a

consequence of their experience● design concepts – prediction

● fundamental to successful decision-making● specify how agents predict the future conditions if their adaptive traits or

learning procedures are based on estimating future consequences of decisions

● if appropriate, what internal models are agents assumed to use to estimate future conditions or consequences of their decisions?

ODD protocol● design concepts – sensing:

● internal and environmental state variables perceived by an agent● range of sensing (local, through networks, global)● the structure of the network imposed or emergent?

● design concepts – interaction:● kinds of interactions among agents (direct or indirect)● interaction network

● design concepts – stochasticity:● processes that are assumed random or partly random● purpose of using stochasticity (e.g. to reproduce variability in processes for

which it is unimportant to model the actual causes of the variability)● design concepts – collectives:

● aggregations (emergent and introduced by a modeler) that affect and are affected by the individuals (e.g. social groups, fish schools and bird flocks, and human networks)

● representation of collectives

ODD protocol● design concepts – observation:

● data collected from the ABM for testing, understanding, and analyzing it● how and when are they collected?

● initialization:● initial state of the model world, i.e., at time t = 0 of a simulation run● How many entities of what type are there initially, and what are the exact

values of their state variables (or how were they set stochastically)?● Is initialization always the same, or is it allowed to vary among simulations?● initial values chosen arbitrarily or based on data● references to those data should be provided

● input data:● input from external sources (e.g. a time series of environmental variables

read from a file)● provide the input data if possible● external models used to generate data

ODD protocol● input data (continued):

● if no external data is used, add the following statement to the description of that element: “the model does not use input data to repre-

● sent time-varying processes.” ● these are not the parameter values or initial values of state variables!

● submodels:● subprocesses listed in ‘Process overview and scheduling’● model parameters, their dimensions, and reference values● design of the submodels

ODD protocol - benefits● ODD promotes rigorous model formulation

● before ODD a common format for describing ABMs was missing● a natural and logical way to compose a model

● once people develop experience using ODD to describe models, they start formulating new ABMs in the ODD format

● increased readers’ understanding of model descriptions● ODD facilitates reviews and comparisons of ABMs

● comparing ABMs that deal with a certain kind of problem or system is quite a task

● if the models are described in the ODD format, a review of their purpose, scales, structure, and process formulation is greatly simplified: one just puts the corresponding parts together in a table and scans for similarities and differences

● ODD may promote more holistic approaches to modeling and theory● as complex models become the rule in bridging the biological and similarly

the social sciences, stringent protocols are needed to ensure communication between disciplines

ODD protocol - complaints● ODD can be redundant

● redundancy is the price for the hierarchical structure of ABM● ODD is overdone for simple models

● some ABMs are extremely simple● describing them in ODD could use considerably more space than a

complete description without ODD● however, the benefits of using ODD are just as applicable to simple models,

and it helps the reader understand what was left out to keep the model simple

ODD protocol - complaints● ODD separates units of object-oriented implementations

● in OOP model entities and their behaviors form one unit: objects with properties (state variables) and methods (processes)

● ODD requires the properties and methods to be presented separately● OOP is currently the natural platform for implementing ABMs, but it is not

the only programming paradigm nor is it the last● ODD was designed to be independent of software platforms● the principle of encapsulation in OOP (easy maintenance of the source

code) does not necessarily apply for creating readable accounts of that code for humans

● ODD improves readability by maintaining this separation

ODD protocol - limitations● designed to describe a definite model version

● we often have to compare different versions to identify the best among alternative submodels, or to learn about the significance of different model designs, assumptions, and parameterizations

● different publications are often based on different versions of the same model, with only a few of the models entities or processes changed

● it would be convenient to have a protocol that describes only the differences from a reference version of the model

● the usefulness of a proposed standard – ODD – was assessed by some of those who originally proposed this protocol (or tested it for the social sciences)● independent and more quantitative tests needed● e.g. ODD’s usefulness for making models replicable

ODD protocol - applications

ODD protocol - applications

Example:

Polhill, J. Gary, Parker, Dawn, Brown, Daniel and Grimm, Volker (2008).

Using the ODD Protocol for Describing Three Agent-Based Social Simulation Models of Land-Use Change.

Journal of Artificial Societies and Social Simulation 11(2)3

<http://jasss.soc.surrey.ac.uk/11/2/3.html>

http://jasss.soc.surrey.ac.uk/11/2/3.html

Documents

Agent-based modelling of complex systemsprac.im.pwr.wroc.pl/~szwabin/assets/abm/lec/10.pdf · make it so simple that there are obviously no deficiencies, and the other way is to make