Upload
todd-roy
View
32
Download
2
Tags:
Embed Size (px)
DESCRIPTION
Evaluating System Change Ventures. Innovations Toward a More Dynamic and Democratic Approach. Bobby Milstein Measurement Knowledge Network WHO Commission on Social Determinants of Health March 22, 2005. Plan for Today. CDC Framework for Program Evaluation - PowerPoint PPT Presentation
Citation preview
Evaluating System Change VenturesEvaluating System Change Ventures
Bobby MilsteinMeasurement Knowledge Network
WHO Commission on Social Determinants of HealthMarch 22, 2005
Bobby MilsteinMeasurement Knowledge Network
WHO Commission on Social Determinants of HealthMarch 22, 2005
Innovations Toward a More Dynamic and Democratic Approach
Innovations Toward a More Dynamic and Democratic Approach
Plan for TodayPlan for Today
CDC Framework for Program Evaluation
Difference between research and evaluation
Link to institutional change
Steps and standards
Innovations for System Change Ventures
Syndemic orientation
Place of simulation modeling
CDC Framework for Program Evaluation
Difference between research and evaluation
Link to institutional change
Steps and standards
Innovations for System Change Ventures
Syndemic orientation
Place of simulation modeling
Appreciating the Unique Character of Evaluative Inquiry
Appreciating the Unique Character of Evaluative Inquiry
“It is easier to find facts than it is to face them.”
-- Anonymous
“It is easier to find facts than it is to face them.”
-- Anonymous
SystematicMethods
SystematicMethods
Centers for Disease Control and Prevention. What procedures are available for planning and evaluating initiatives to prevent syndemics? Syndemics Prevention Network, 2001. Available at <http://www.cdc.gov/syndemics/overview-planeval.htm>.
Research
Questions of Fact(descriptions, associations, effects)
Questions of Fact(descriptions, associations, effects)
Evaluation
Questions of Values(merit, worth, significance)
Questions of Values(merit, worth, significance)
Institutional Climate for Evaluative InquiryInstitutional Climate for Evaluative Inquiry
Milstein B, Chapel T, Wetterhall S, Cotton D. Building capacity for program evaluation at the Centers for Disease Control and Prevention. New Directions for Evaluation 2002;2002(93):27-47.
“The CDC’s senior leaders understood that strengthening
evaluation capacity in public health would require a process of culture
change, including significant reforms to their own organization.”
“The CDC’s senior leaders understood that strengthening
evaluation capacity in public health would require a process of culture
change, including significant reforms to their own organization.”
The Evaluation ContinuumThe Evaluation Continuum
Ways of EvaluatingWays of EvaluatingInformal
Low
Formal
HighStakes InvolvedStakes Involved
Framework for Program EvaluationFramework for Program Evaluation
Milstein B, Wetterall S, CDC Evaluation Working Group. Framework for program evaluation in public health. MMWR Recommendations and Reports 1999;48(RR-11):1-40. Available at <http://www.cdc.gov/mmwr/PDF/RR/RR4811.pdf>.
“Both a synthesis of existing evaluation practicesand a standard for further improvement.”
“Both a synthesis of existing evaluation practicesand a standard for further improvement.”
Framework for Program EvaluationFramework for Program Evaluation
Milstein B, Wetterall S, CDC Evaluation Working Group. Framework for program evaluation in public health. MMWR Recommendations and Reports 1999;48(RR-11):1-40. Available at <http://www.cdc.gov/mmwr/PDF/RR/RR4811.pdf>.
“A practical, nonprescriptive tool, designed to summarizeand organize the essential elements of program evaluation.”
“A practical, nonprescriptive tool, designed to summarizeand organize the essential elements of program evaluation.”
Utility (7)Serve information needs of intended users
Feasibility (3)Be realistic, prudent, diplomatic, and frugal
Propriety (8)Behave legally, ethically, and with due regard for the welfare of those involved and those affected
Accuracy (12)Reveal and convey technically accurate information
Utility (7)Serve information needs of intended users
Feasibility (3)Be realistic, prudent, diplomatic, and frugal
Propriety (8)Behave legally, ethically, and with due regard for the welfare of those involved and those affected
Accuracy (12)Reveal and convey technically accurate information
Standards for Effective EvaluationStandards for Effective Evaluation
Joint Committee on Educational Evaluation, James R. Sanders (Chair). The program evaluation standards: how to assess evaluations of educational programs. 2nd edition ed Thousand Oaks, CA: Sage Publications, 1994.
Catalyst for Complementary ResourcesCatalyst for Complementary Resources
CDC Evaluation Working Group. Framework for program evaluation: adapted versions. Available at <http://www.cdc.gov/eval/framework.htm>.
Are We Posing Questions About Attribution or Contribution?
Are We Posing Questions About Attribution or Contribution?
“…if a program’s activities are aligned with those of other programs operating in the same setting, certain effects (e.g., the creation of new laws or policies) cannot be attributed solely to one program or another. In such situations, the goal for evaluation is to gather credible evidence that describes each program’s contribution in the combined change effort. Establishing accountability for program results is predicated on an ability to conduct evaluations that assess both of these kinds of effects.” p.11-12
“…if a program’s activities are aligned with those of other programs operating in the same setting, certain effects (e.g., the creation of new laws or policies) cannot be attributed solely to one program or another. In such situations, the goal for evaluation is to gather credible evidence that describes each program’s contribution in the combined change effort. Establishing accountability for program results is predicated on an ability to conduct evaluations that assess both of these kinds of effects.” p.11-12
Calls into question the conditions in which one focuses on a “program” as the unit of analysis
Calls into question the conditions in which one focuses on a “program” as the unit of analysis
Acknowledging PluralityAcknowledging Plurality
Efforts to Reduce Population Health ProblemsProblem, problem solver, response
Efforts to Organize a System that Assures Healthful Conditions for All Dynamic interaction among multiple problems, problem solvers, and responses
Efforts to Reduce Population Health ProblemsProblem, problem solver, response
Efforts to Organize a System that Assures Healthful Conditions for All Dynamic interaction among multiple problems, problem solvers, and responses
Bammer G. Integration and implementation sciences: building a new specialisation. Cambridge, MA: The Hauser Center for Nonprofit Organizations, Harvard University 2003.
“You think you understand two because you understand one and one. But you must also understand ‘and’.”
-- Sufi Saying
“You think you understand two because you understand one and one. But you must also understand ‘and’.”
-- Sufi Saying
A syndemic orientation clarifies the dynamic and democratic character
of public health work
The Syndemics Prevention NetworkThe Syndemics Prevention Network
The word syndemic signals special concern for many kinds of relationships:
mutually reinforcing health problems
health status and living conditions
synergy/fragmentation in the health response system
Learning within innovative ventures
Comprehensive Community InitiativesPhilanthropy
Legacy InitiativesState Tobacco Settlements
Efforts to Eliminate Health Disparities Government and Philanthropy
Responses to Unjust Conditions Broad-based Citizen Organizations
The word syndemic signals special concern for many kinds of relationships:
mutually reinforcing health problems
health status and living conditions
synergy/fragmentation in the health response system
Learning within innovative ventures
Comprehensive Community InitiativesPhilanthropy
Legacy InitiativesState Tobacco Settlements
Efforts to Eliminate Health Disparities Government and Philanthropy
Responses to Unjust Conditions Broad-based Citizen Organizations
Health
LivingConditions
Power toAct
Milstein B. Spotlight on syndemics. Centers for Disease Control and Prevention, 2001. <http://www.cdc.gov/syndemics>
“You think you understand two because you understand one and one. But you must also understand ‘and’.”
-- Sufi Saying
“You think you understand two because you understand one and one. But you must also understand ‘and’.”
-- Sufi Saying
Questioning the Character of Public Health WorkQuestioning the Character of Public Health WorkPUBLIC HEALTH WORK
InnovativeHealth
Ventures
SYSTEMS THINKING & MODELING (understanding change)
• What causes population health problems?
• How are efforts to protect the public’s health organized?
• How and when do health systems change (or resist change)?
PUBLIC HEALTH(setting direction)
What are health leaderstrying to accomplish?
SOCIAL NAVIGATION(governing movement)
Directing Change
Charting Progress
• Who does the work?• By what means?• According to whose values?
• How are conditions changing?• In which directions?
Formalizing an OrientationJoining Concepts and Methods
Formalizing an OrientationJoining Concepts and Methods
Syndemic Orientation
Network View
Connections
Proximity Data
What links to what?
Systems View
X Y
Leverage
Causal Data
What influences what?
Navigational View
Directed Public Work
Directional Data
Where are we going?
Changing (and Accumulating) Ideas in Causal Theory
What accounts for poor population health?
Changing (and Accumulating) Ideas in Causal Theory
What accounts for poor population health?
God’s will
Humors, miasma, atmosphere (“epidemic constitution”)
Poor living conditions, immorality (sanitation)
Single disease, single cause (germ theory)
Single disease, multiple causes (heart disease)
Single cause, multiple diseases (tobacco)
Multiple causes, multiple diseases (but no feedback dynamics) (social epidemiology)
Dynamic feedback among afflictions, living conditions, and public strength (syndemic)
God’s will
Humors, miasma, atmosphere (“epidemic constitution”)
Poor living conditions, immorality (sanitation)
Single disease, single cause (germ theory)
Single disease, multiple causes (heart disease)
Single cause, multiple diseases (tobacco)
Multiple causes, multiple diseases (but no feedback dynamics) (social epidemiology)
Dynamic feedback among afflictions, living conditions, and public strength (syndemic)
1880
1950
1960
1980
2000
1840
“When X and Y affect each other, one cannot study the link between X and Y and, independently, the link between Y and X and predict how the system will behave. Only the study of the whole
system as a feedback system will lead to correct results."
“When X and Y affect each other, one cannot study the link between X and Y and, independently, the link between Y and X and predict how the system will behave. Only the study of the whole
system as a feedback system will lead to correct results."
-- System Dynamics Society-- System Dynamics Society
The Feedback ThoughtThe Feedback Thought
System Dynamics Society. What is system dynamics? System Dynamics Society, 2002. Available at <http://www.systemdynamics.org/>.
Richardson GP. Feedback thought in social science and systems theory. Philadelphia: University of Pennsylvania Press, 1991.
System Dynamics Society. What is system dynamics? System Dynamics Society, 2002. Available at <http://www.systemdynamics.org/>.
Richardson GP. Feedback thought in social science and systems theory. Philadelphia: University of Pennsylvania Press, 1991.
X Y
A Very Particular DistanceA Very Particular Distance
“The macroscope filters details and amplifies that which links things together. It is not used
to make things larger or smaller but to observe what is
at once too great, too slow, and too complex for our eyes.”
“The macroscope filters details and amplifies that which links things together. It is not used
to make things larger or smaller but to observe what is
at once too great, too slow, and too complex for our eyes.”
Rosnay J. The macroscope: a book on the systems approach. Principia Cybernetica, 1997. <http://pespmc1.vub.ac.be/MACRBOOK.html
-- Joèel de Rosnay-- Joèel de Rosnay
Looking Through the MacroscopeLooking Through the Macroscope
Toward a Balanced System of Health ProtectionToward a Balanced System of Health Protection
SaferHealthierPeople Becoming
vulnerable
Becoming saferand healthier
VulnerablePeople Becoming
afflicted
Afflictedwithout
Complications Developingcomplications
Afflicted withComplications
Dying fromcomplications
Society's HealthResponse
Demand forresponse
Public Work
Adverse LivingConditions
GeneralProtection
TargetedProtection
PrimaryPrevention
SecondaryPrevention
TertiaryPrevention
From: Milstein B, Homer J. The dynamics of upstream and downstream: why is so hard for the health system to work upstream, and what can be done about it? CDC Futures Health Systems Workgroup; Atlanta, GA; 2003.
Balancing Two Major Areas of EmphasisBalancing Two Major Areas of Emphasis
SaferHealthierPeople
VulnerablePeople
Afflictedwithout
ComplicationsAfflicted with
ComplicationsBecomingvulnerable
Becoming saferand healthier
Becomingafflicted
Developingcomplications
Dying fromcomplications
Adverse LivingConditions
Society's HealthResponse
Demand forresponse
GeneralProtection
TargetedProtection
PrimaryPrevention
SecondaryPrevention
TertiaryPrevention
Public Work
World of Providing…
• Health education• Screening tests• Disease management • Pharmaceuticals• Clinical services• Physical and financial access• Etc…
Medical and Public Health Policy
MANAGEMENT OFRISKS & DISEASES
World of Transforming…
• Deprivation• Dependency• Violence• Disconnection• Environmental decay• Stress• Insecurity• Etc…
By Strengthening…
• Leaders and institutions• Foresight and precaution• The meaning of work• Mutual accountability• Plurality• Democracy• Freedom• Etc…
Healthy Public Policy & Public Work
DEMOCRATIC SELF-GOVERNANCE
Understanding Health as Public WorkUnderstanding Health as Public Work
SaferHealthierPeople
VulnerablePeople
Afflictedwithout
Complications
Afflicted withComplicationsBecoming
vulnerable
Becoming saferand healthier
Becomingafflicted
Developingcomplications
Dying fromcomplications
Adverse LivingConditions
Society's HealthResponse
Demand forresponse
GeneralProtection
TargetedProtection
PrimaryPrevention
SecondaryPrevention
TertiaryPrevention
-
Public Work-
Vulnerable andAfflicted People
Fraction of Adversity,Vulnerability and AfflictionBorne by Disadvantaged
Sub-Groups (Inequity)
PublicStrength
-
Citizen Involvementin Public Life
Social Division
How Can Test a Dynamic Hypothesis?How Can Test a Dynamic Hypothesis?
-- How can we learn about the consequences of actions in a system of this kind?-- Could the behavior of this system be analyzed using conventional epidemoiological methods (e.g., logistic or multi-level regression)?
SaferHealthierPeople
VulnerablePeople
Afflictedwithout
Complications
Afflicted withComplicationsBecoming
vulnerable
Becoming saferand healthier
Becomingafflicted
Developingcomplications
Dying fromcomplications
Adverse LivingConditions
Society's HealthResponse
Demand forresponse
GeneralProtection
TargetedProtection
PrimaryPrevention
SecondaryPrevention
TertiaryPrevention
-
Public Work-
Vulnerable andAfflicted People
Fraction of Adversity,Vulnerability and AfflictionBorne by Disadvantaged
Sub-Groups (Inequity)
PublicStrength
-
Citizen Involvementin Public Life
Social Division
Time Series ModelsDescribe trends
Multivariate Stat Models
Identify historical trend drivers and correlates
Patterns
Structure
Events
Increasing:
• Depth of causal theory
• Degrees of uncertainty
• Robustness for longer-term projection
• Value for developing policy insights
Increasing:
• Depth of causal theory
• Degrees of uncertainty
• Robustness for longer-term projection
• Value for developing policy insights
Dynamic Models
Anticipate future trends, and find policies that maximize chances
of a desirable path
Tools for Policy AnalysisTools for Policy Analysis
Average Number of Unhealthy Days per Month
2
4
6
8
1993 1995 1997 1999 2001
Year
2005 2025 2050
Redirecting the Course of ChangeQuestions from System Dynamics and Social Navigation
Redirecting the Course of ChangeQuestions from System Dynamics and Social Navigation
14% increase
Zack MM, Moriarty DG, Stroup DF, Ford ES, Mokdad AH. Worsening trends in adult health-related quality of life and self-rated health–United States, 1993-2001. Public Health Reports 2004;119(September-October):493-505.
How?Why?
Where?
Who?
Simulations for Learning in Dynamic SystemsThe Problem of Outside Assistance
Simulations for Learning in Dynamic SystemsThe Problem of Outside Assistance
Dynamic Hypothesis (Structure)Dynamic Hypothesis (Structure)
Behavior Over Time (Experiments)Behavior Over Time (Experiments)Affliction
prevalence& burden
Adverseliving
Publicstrength
R1
At-risk fraction
Afflictioncross-impacts
Effort to alleviate andprevent affliction
B1a
Effort to improveliving conditions
B1b
Effort to build public strength
B2
Social disparityR2c
R2b
R2a
R3a
Public work fraction
United efforts
Divided efforts
R3b
Outside assistance toalleviate and prevent
affliction
Outside assistanceto improve living
conditions
Outside assistanceto build public strength
Magnitude ofameliorative efforts
R4a
R4b B3b
B3a
KeyRectangle: Stock/state variableBlue arrow: same-direction linkGreen arrow: opposite-direction linkCircled “B”: balancing causal loopCircled “R”: reinforcing causal loop
conditions
12
10
8
6
4
0 2 4 6 8 10 12 14 16 18 20Time (years)
Affliction burden : BasicOptAffliction burden : BasicAF111Affliction burden : BasicLC111Affliction burden : BasicCS111
12
10
8
6
4
0 2 4 6 8 10 12 14 16 18 20Time (years)
Affliction burden : BasicOptAffliction burden : BasicAF111Affliction burden : BasicLC111Affliction burden : BasicCS111
Homer J, Milstein B. Optimal decision making in a dynamic model of poor community health. Proceedings of the 37th Hawaii International Conference on System Science; Big Island, Hawaii; January 5-8, 2004. Available at <http://csdl.computer.org/comp/proceedings/hicss/2004/2056/03/205630085a.pdf>.
Simulations for Learning in Dynamic SystemsDiabetes Dynamics in an Era of Epidemic Obesity
Simulations for Learning in Dynamic SystemsDiabetes Dynamics in an Era of Epidemic Obesity
Dynamic Hypothesis (Structure)Dynamic Hypothesis (Structure)
Behavior Over Time (Experiments)Behavior Over Time (Experiments)
Jones A, Homer J, Milstein B, Essien J, Murphy D, Sorensen S, Englegau M. Modeling the population dynamics of a chronic disease: the CDC's diabetes system model. American Journal of Public Health (in press).
People withUndiagnosed,Uncomplicated
Diabetes
People withDiagnosed,
UncomplicatedDiabetes
People withDiagnosed,Complicated
Diabetes
DiagnosingUncomplicated
Diabetes
People withUndiagnosedPreDiabetes
People withDiagnosedPreDiabetes
DiagnosingPreDiabetes
DevelopingComplications from
Undx diab
DevelopingDiabetes from Undx
PreD,People with
Undiagnosed,Complicated
Diabetes
DiagnosingComplicated
Diabetes
Dying from UndxComplications
People withNormal
GlycemicLevels
DiabetesDetection
Obese Fraction ofthe Population
Risk forPreDiabetes & Diabetes
Caloric Intake PhysicalActivity
PreDiabetesControl
DiabetesControl
PreDiabetesDetection
MedicationAffordability
Ability to SelfMonitor
Adoption ofHealthy Lifestyle
ClinicalManagement of
PreDiabetes
Clinical Managementof Diagnosed
Diabetes
LivingConditions
PersonalCapacity
PreDiabetesTesting for
Access toPreventive Health
Services Testing forDiabetes
PreDiabetesOnset
Recovering fromPreDiabete
s
Recovering fromPreDiabete
sDiabetesOnset
Dying fromComplications
DevelopingComplications
Deaths per Population
0.0035
0.003
0.0025
0.002
0.0015
1980 1990 2000 2010 2020 2030 2040 2050
Time (Year)
Blue: Base run; Red: Clinical mgmt up from 66% to 90%;Green: Caloric intake down 4% (99 Kcal/day);Black: Clin mgmt up to 80% & Intake down 2.5% (62 Kcal/day)
Base
Downstream
Upstream
Mixed
Striking an acceptable balance.
Iterative Steps in System Dynamics Simulation Modeling
Iterative Steps in System Dynamics Simulation Modeling
Enact PolicyBuild power and
organize actors to establish chosen policies
Enact PolicyBuild power and
organize actors to establish chosen policies
Learn About Policy Consequences
Test proposed policies, searching for ones that
best govern change
Learn About Policy Consequences
Test proposed policies, searching for ones that
best govern change
Run Simulation Experiments
Compare model’s behavior to expectations and/or data to
build confidence in the model
Run Simulation Experiments
Compare model’s behavior to expectations and/or data to
build confidence in the model
Convert the Map Into a Simulation Model
Formally quantify the hypothesis using allavailable evidence
Convert the Map Into a Simulation Model
Formally quantify the hypothesis using allavailable evidence
Create a Dynamic Hypothesis Identify and map the
main causal forces that create the problem
Create a Dynamic Hypothesis Identify and map the
main causal forces that create the problem
Identify a Persistent Problem Graph its behavior
over time
Identify a Persistent Problem Graph its behavior
over time
Milstein B, Homer J. Background on system dynamics simulation modeling, with a summary of major public health studies. Atlanta, GA: Syndemics Prevention Network, Centers for Disease Control and Prevention; February 1, 2005.
Learning In and About Dynamic SystemsLearning In and About Dynamic Systems
Benefits of Simulation/Game-based Learning
Formal means of evaluating options
Experimental control of conditions
Compressed time
Complete, undistorted results
Actions can be stopped or reversed
Visceral engagement and learning
Tests for extreme conditions
Early warning of unintended effects
Opportunity to assemble stronger support
Benefits of Simulation/Game-based Learning
Formal means of evaluating options
Experimental control of conditions
Compressed time
Complete, undistorted results
Actions can be stopped or reversed
Visceral engagement and learning
Tests for extreme conditions
Early warning of unintended effects
Opportunity to assemble stronger support
Complexity Hinders
Generation of evidence (by eroding the conditions for experimentation)
Learning from evidence (by demanding new heuristics for interpretation)
Acting upon evidence (by including the behaviors of other powerful actors)
Complexity Hinders
Generation of evidence (by eroding the conditions for experimentation)
Learning from evidence (by demanding new heuristics for interpretation)
Acting upon evidence (by including the behaviors of other powerful actors)
Sterman JD. Learning from evidence in a complex world. American Journal of Public Health (in press).
Sterman JD. Business Dynamics: Systems Thinking and Modeling for a Complex World. Boston, MA: Irwin McGraw-Hill, 2000.
“The complexity of our models vastly exceeds our ability to understand their implications without simulation."
-- John Sterman
“The complexity of our models vastly exceeds our ability to understand their implications without simulation."
-- John Sterman
“Simulation is a third way of doing science. Like deduction, it
starts with a set of explicit assumptions. But unlike deduction,
it does not prove theorems. Instead, a simulation generates
data that can be analyzed inductively. Unlike typical induction,
however, the simulated data comes from a rigorously specified
set of rules rather than direct measurement of the real world.
While induction can be used to find patterns in data, and
deduction can be used to find consequences of assumptions,
simulation modeling can be used as an aid to intuition.”
“Simulation is a third way of doing science. Like deduction, it
starts with a set of explicit assumptions. But unlike deduction,
it does not prove theorems. Instead, a simulation generates
data that can be analyzed inductively. Unlike typical induction,
however, the simulated data comes from a rigorously specified
set of rules rather than direct measurement of the real world.
While induction can be used to find patterns in data, and
deduction can be used to find consequences of assumptions,
simulation modeling can be used as an aid to intuition.”
-- Robert Axelrod-- Robert Axelrod
Axelrod R. Advancing the art of simulation in the social sciences. In: Conte R, Hegselmann R, Terna P, editors. Simulating Social Phenomena. New York, NY: Springer; 1997. p. 21-40. <http://www.pscs.umich.edu/pub/papers/AdvancingArtofSim.pdf>.
A Third Branch of ScienceA Third Branch of Science
Revisiting the FrameworkRevisiting the Framework
Simulation Modeling Offers
Support for multi-stakeholder dialogue
A larger conception of the “program” context
Another avenue for experimentation and visceral learning
Ability to track interrelated indicators (both states and rates)
An emphasis on pragmatism (learning through action)
Simulation Modeling Offers
Support for multi-stakeholder dialogue
A larger conception of the “program” context
Another avenue for experimentation and visceral learning
Ability to track interrelated indicators (both states and rates)
An emphasis on pragmatism (learning through action)
“Steps in the framework are starting points for tailoring an evaluation to a particular public health effort at a particular time.”
“Steps in the framework are starting points for tailoring an evaluation to a particular public health effort at a particular time.”
For Additional Informationhttp://www.cdc.gov/syndemics
Enhancing Learning Through SimulationEnhancing Learning Through Simulation
Sterman J. Business dynamics: systems thinking and modeling for a complex world. Boston, MA: Irwin McGraw-Hill, 2000.
Policy ResistancePolicy Resistance
Lee P, Paxman D. Reinventing public health. Annual Reviews of Public Health 1997;18:1-35.
Meadows DH, Richardson J, Bruckmann G. Groping in the dark: the first decade of global modelling. New York, NY: Wiley, 1982.
“At least six times since the
Depression, the United States has
tried and failed to enact a national
health insurance program.”
“At least six times since the
Depression, the United States has
tried and failed to enact a national
health insurance program.”
-- Lee & Paxman-- Lee & Paxman
“The tendency for interventions to be delayed, diluted, or defeated by the response of the system to the intervention itself.”
“The tendency for interventions to be delayed, diluted, or defeated by the response of the system to the intervention itself.”
-- Meadows, Richardson, Bruckman-- Meadows, Richardson, Bruckman
Flaws in Previous Attempts at Health System Reform in America (and Elsewhere)
Flaws in Previous Attempts at Health System Reform in America (and Elsewhere)
Heirich M. Rethinking health care: innovation and change in America. Boulder CO: Westview Press, 1999.
Kari NN, Boyte HC, Jennings B. Health as a civic question. American Civic Forum, 1994. Available at <http://www.cpn.org/topics/health/healthquestion.html>.
Piecemeal approaches that do not address the full scope of the problem
Comprehensive strategies that are opposed by special interests
Assumption that healthcare dynamics are separate from other areas of public concern
Conventional analytic methods make it difficult to
Observe the health system as a large, dynamic enterprise
Craft high-leverage strategies that can overcome policy resistance
Been thinking of health and healthcare primarily as nouns (i.e., commodities distributed to consumers), not as verbs (i.e., public work to be done by citizens)
Piecemeal approaches that do not address the full scope of the problem
Comprehensive strategies that are opposed by special interests
Assumption that healthcare dynamics are separate from other areas of public concern
Conventional analytic methods make it difficult to
Observe the health system as a large, dynamic enterprise
Craft high-leverage strategies that can overcome policy resistance
Been thinking of health and healthcare primarily as nouns (i.e., commodities distributed to consumers), not as verbs (i.e., public work to be done by citizens)
Basic Problem Solving OrientationsBasic Problem Solving Orientations
Sterman J. Business dynamics: systems thinking and modeling for a complex world. Boston, MA: Irwin McGraw-Hill, 2000.Sterman J. Business dynamics: systems thinking and modeling for a complex world. Boston, MA: Irwin McGraw-Hill, 2000.
“Side Effects”
Feedback View
Goals
Environment
Action
Goals ofOthers
Action ofOthers
“Side Effects”
Delay Delay
Delay
Delay
DelayDelay
Delay
Delay
Delay
Delay
Delay
Delay
Event Oriented View
Problem Results
Goals
Situation
Action
Building on Decades of SD Health StudiesBuilding on Decades of SD Health Studies
Disease epidemiology heart disease, diabetes, HIV/AIDS, cervical cancer, chlamydia, dengue fever, drug-resistant infections
Substance abuse epidemiology heroin, cocaine, tobacco
Health care patient flows emergency care, extended care
Health care capacity and deliveryHMO planning, dental care capacity, mental health care, disaster preparedness
Interactions between health capacity and disease epidemiology
chronic illness management, syndemics
Disease epidemiology heart disease, diabetes, HIV/AIDS, cervical cancer, chlamydia, dengue fever, drug-resistant infections
Substance abuse epidemiology heroin, cocaine, tobacco
Health care patient flows emergency care, extended care
Health care capacity and deliveryHMO planning, dental care capacity, mental health care, disaster preparedness
Interactions between health capacity and disease epidemiology
chronic illness management, syndemics
Homer JB, Hirsch GB. System dynamics modeling for public health: background and opportunities. American Journal of Public Health (in press).
Progress in Dynamic ModelingProgress in Dynamic ModelingProblem Focus Dynamic Focus
Grantmaking strategy in areas challenged by multiple afflictions
Designed to explore interactions between afflictions, living conditions, and public strength
Balancing upstream and downstream effort
Designed to understand an observed phenomenon, the 97% -- 3% split in health care expenditures
Diabetes in an era of increasing obesity
Most empirically supported
Fetal and infant health
Health care reform policies
Life-course dynamics of obesity
Transforming the Future of Diabetes…Transforming the Future of Diabetes…
"Every new insight into Type 2 diabetes...
makes clear that it can be avoided--and
that the earlier you intervene the better.
The real question is whether we as a
society are up to the challenge...
Comprehensive prevention programs
aren't cheap, but the cost of doing
nothing is far greater..."
"Every new insight into Type 2 diabetes...
makes clear that it can be avoided--and
that the earlier you intervene the better.
The real question is whether we as a
society are up to the challenge...
Comprehensive prevention programs
aren't cheap, but the cost of doing
nothing is far greater..."
Gorman C. Why so many of us are getting diabetes: never have doctors known so much about how to prevent or control this disease, yet the epidemic keeps on raging. how you can protect yourself. Time 2003 December 8. Accessed at http://www.time.com/time/covers/1101031208/story.html.
…in an Era of Epidemic Obesity…in an Era of Epidemic Obesity
Re-Directing the Course of ChangeQuestions from System Modeling and Social Navigation
Re-Directing the Course of ChangeQuestions from System Modeling and Social Navigation
20202010
Prevalence of Diagnosed Diabetes, US
0
5
10
15
1980 1985 1990 1995 2000
Mill
ion
peop
le
Data Source: CDC DDT and NCCDPHP. -- Change in measurement in 1996.
How?
Why?
Where?
Who?
Diabetes System Modeling ProjectWhere is the Leverage for Health Protection?
Diabetes System Modeling ProjectWhere is the Leverage for Health Protection?
Jones A, Homer J, Milstein B, Essien J, Murphy D, Sorensen S, Englegau M. Modeling the population dynamics of a chronic disease: the CDC's diabetes system model. American Journal of Public Health (in press).
People withUndiagnosed,Uncomplicated
Diabetes
People withDiagnosed,
UncomplicatedDiabetes
People withDiagnosed,Complicated
Diabetes
People withUndiagnosedPreDiabetes
People withDiagnosed
PreDiabetes
People withUndiagnosed,Complicated
DiabetesPeople with
NormalGlycemic
Levels
DiagnosingDiabetes
DiagnosingDiabetes
Diabetes Detection
Dying fromComplications
DevelopingComplications
Diabetes Control
PreDiabetes Detection
DiagnosingPreDiabetes
DiabetesOnset
PreDiabetes Control
PreDiabetesOnset
Recovering fromPreDiabetes
Recovering fromPreDiabetes
Obesity Prevention
Diabetes System Modeling ProjectWhere is the Leverage for Health Protection?
Diabetes System Modeling ProjectWhere is the Leverage for Health Protection?
People withUndiagnosed,Uncomplicated
Diabetes
People withDiagnosed,
UncomplicatedDiabetes
People withDiagnosed,Complicated
Diabetes
DiagnosingUncomplicated
Diabetes
People withUndiagnosedPreDiabetes
People withDiagnosed
PreDiabetes
DiagnosingPreDiabetes
DevelopingComplications from
Undx diab
DevelopingDiabetes from Undx
PreD,People with
Undiagnosed,Complicated
Diabetes
DiagnosingComplicated
Diabetes
Dying from UndxComplications
People withNormal
GlycemicLevels
DiabetesDetection
Obese Fraction ofthe Population
Risk forPreDiabetes & Diabetes
Caloric Intake PhysicalActivity
PreDiabetesControl
DiabetesControl
PreDiabetesDetection
MedicationAffordability
Ability to SelfMonitor
Adoption ofHealthy Lifestyle
ClinicalManagement of
PreDiabetes
Clinical Managementof Diagnosed
Diabetes
LivingConditions
PersonalCapacity
PreDiabetesTesting for
Access toPreventive Health
Services Testing forDiabetes
PreDiabetesOnset
Recovering fromPreDiabetes
Recovering fromPreDiabetes Diabetes
Onset
Dying fromComplications
DevelopingComplications
Deaths per Population
0.0035
0.003
0.0025
0.002
0.0015
1980 1990 2000 2010 2020 2030 2040 2050
Time (Year)
Blue: Base run; Red: Clinical mgmt up from 66% to 90%;Green: Caloric intake down 4% (99 Kcal/day);Black: Clin mgmt up to 80% & Intake down 2.5% (62 Kcal/day)
Base
Downstream
Upstream
Mixed
Striking an acceptable balance.
Using Available Data to Calibrate the ModelUsing Available Data to Calibrate the Model
Information Sources Data
U.S. Census Adult population and death rates
Health insurance coverage
National Health Interview Survey Diabetes prevalence
Diabetes detection
National Health and Nutrition Examination Survey
Prediabetes prevalence
Weight, height, and body fat
Caloric intake
Behavioral Risk Factor Surveillance System
Glucose self-monitoring
Eye and foot exams
Participation in health education
Use of medications
Professional Literature
Physical activity trends
Effects of control and aging on onset, progression, death, and costs
Expenditures
Diabetes System Modeling ProjectConfirming the Model’s Fit to History
Diabetes System Modeling ProjectConfirming the Model’s Fit to History
Jones A, Homer J, Milstein B, Essien J, Murphy D, Sorensen S, Englegau M. Modeling the population dynamics of a chronic disease: the CDC's diabetes system model. American Journal of Public Health (in press).
0%
2%
4%
6%
8%
1980 1985 1990 1995 2000 2005 2010
Diagnosed diabetes % of adults
Data (NHIS)
Simulated
0%
2%
4%
6%
8%
1980 1985 1990 1995 2000 2005 2010
Diagnosed diabetes % of adults
Data (NHIS)
Simulated
0%
10%
20%
30%
40%
1980 1985 1990 1995 2000 2005 2010
Obese % of adults
Data (NHANES)
Simulated
0%
10%
20%
30%
40%
1980 1985 1990 1995 2000 2005 2010
Obese % of adults
Data (NHANES)
Simulated
Diagnosed Diabetes % of AdultsObese % of Adults
Explaining the PastWhat Has Driven the Burden of Diabetes?
Explaining the PastWhat Has Driven the Burden of Diabetes?
Great Progress in Reducing the Burden
for the Average Person with Diabetes
Huge Growth in Number of People
with Diabetes
Overall, Total Diabetes Burden Held at Bay
Explaining the Past Deaths Due to Diabetes Have Fallen
Explaining the Past Deaths Due to Diabetes Have Fallen
Complications Deaths per Thous People w Diabetes40
30
20
10
0
1980 1985 1990 1995 2000 2005Time (Year)
People with Diabetes per Thousand Adults
100
90
80
70
60
501980 1985 1990 1995 2000 2005
Time (Year)
More people with diabetes
Deaths from Comps of Diabetes Per Thous Adults2.5
2
1.5
1
0.5
0
1980 1985 1990 1995 2000 2005Time (Year)
Model OutputModel Output
Model Output
Among people with diabetes, fewer dying every year
Combine to mean fewer U.S. adults dying 1980-2004
Anticipating the FuturePrevalence Under ‘Status Quo’ Assumptions
Anticipating the FuturePrevalence Under ‘Status Quo’ Assumptions
After adelay
Obese Fraction of Adult Population
0.6
0.45
0.3
0.15
0
Even if we assume the obesity epidemic has peaked…
People with Diabetes per Thousand Adults
130
110
90
70
50
1980 1990 2000 2010 2020 2030 2040 2050Time (Year)
Prevalence continues to increase.
Model OutputModel Output
1980 1990 2000 2010 2020 2030 2040 2050Time (Year)
People with Diabetes per Thousand Adults130
110
90
70
50
1980 1990 2000 2010 2020 2030 2040 2050Time (Year)
Complications Deaths per Thous w Diabetes40
30
20
10
0
1980 1990 2000 2010 2020 2030 2040 2050Time (Year)
Deaths from Complications of Diabetes Per Thousand Adults2.5
1.25
1980 1990 2000 2010 2020 2030 2040 2050Time (Year)
Then prevalence overwhelms the improved care to boost the burden
Anticipating the Future Deaths Under ‘Status Quo’ Assumptions
Anticipating the Future Deaths Under ‘Status Quo’ Assumptions
And if we can maintain current levels of care
but no continued improvement…
If prevalence continues to increase,
Policy ExperimentsPolicy Experiments
Continued downstream improvements
Upstream efforts
Downstream & Upstream
Continued downstream improvements
Upstream efforts
Downstream & Upstream
Deaths per Population0.0035
0.003
0.0025
0.002
0.0015
1980 1990 2000 2010 2020 2030 2040 2050Time (Year)
Downstream-Only InterventionDownstream-Only Intervention
Blue: Base run; Red: Clinical mgmt of diagnosed up from 66% to 90%
Base
Downstream
Disease control acts quickly but does not slow the growth in deaths.
Upstream-Only InterventionUpstream-Only InterventionDeaths per Population
0.0035
0.003
0.0025
0.002
0.00151980 1990 2000 2010 2020 2030 2040 2050
Time (Year)
Blue: Base run; Red: Clinical mgmt up from 66% to 90%;Green: Caloric intake down 4% (99 Kcal/day)
Downstream
UpstreamBase
Obesity prevention slows the growth but takes a long time to do so.
Deaths per Population0.0035
0.003
0.0025
0.002
0.00151980 1990 2000 2010 2020 2030 2040 2050
Time (Year)
Mixed InterventionMixed Intervention
Blue: Base run; Red: Clinical mgmt up from 66% to 90%;Green: Caloric intake down 4% (99 Kcal/day);Black: Clin mgmt up to 80% & Intake down 2.5% (62 Kcal/day)
Base
Downstream
Upstream
Mixed
Striking an acceptable balance.
QuestionsQuestions
Where might the results of simulation experiments fit in the hierarchies of evidence used for systematic reviews?
Where might the results of simulation experiments fit in the hierarchies of evidence used for systematic reviews?
Understanding Health as Public WorkUnderstanding Health as Public Work
Safer,Healthier
Population
VulnerablePopulation
Becomingvulnerable
Becoming nolonger vulnerable
Afflictedwithout
ComplicationsBecomingafflicted
Afflicted withComplications
Developingcomplications
Dying fromcomplications
Primary preventioneffect on incidence
-
Secondary preventioneffect on progression
-
Tertiary preventioneffect on complications
-
General protectioneffect on living
conditions
Targeted protectioneffect on vulnerability
reduction
GeneralProtection
TargetedProtection
TertiaryPrevention
SecondaryPrevention
PrimaryPrevention
Contacts withafflicted
DirectContagion
Vulnerable andAfflicted Popn
RiskDiffusionAdverse
LivingConditions
contributingto
Vulnerability
-
PublicStrength
Upstream work
Downstream work
Social disparity
-
SocialDivision
CitizenInvolvement
Effort to buildpublic strength
Social and economicadaptation to norms and
needs of risk popn
Professionalconcern
Public concern
-
Efforts to reduce ALC forreasons other than health
protection
-
Public workInstitutional/organizationalemphasis on disease rather
than vulnerability
-
Wickelgren I. How the brain 'sees' borders. Science 1992;256(5063):1520-1521.
How Many Triangles Do You See?How Many Triangles Do You See?
“Solutions” Can Also Create New Problems“Solutions” Can Also Create New Problems
Merton RK. The unanticipated consequences of purposive social action. American Sociological Review 1936;1936:894-904.
Forrester JW. Counterintuitive behavior of social systems. Technology Review 1971;73(3):53-68.
New Word for a Familiar PhenomenonNew Word for a Familiar Phenomenon
Singer M, Snipes C. Generations of suffering: experiences of a treatment program for substance abuse during pregnancy. Journal of Health Care for the Poor and Underserved 1992;3(1):222-34.
Singer M. 1994. AIDS and the health crisis of the US urban poor: The perspective of critical medical anthropology. Social Science and Medicine 39(7): 931-948.
Singer M. 1996. A dose of drugs, a touch of violence, a case of AIDS: Conceptualizing the SAVA syndemic. Free Inquiry in Creative Sociology 24(2): 99-110.
Singer M, Clair S. Syndemics and public health: reconceptualizing disease in bio-social context. Medical Anthropology Quarterly 2003;17(4):423-441.
“We have introduced the term ‘syndemic’ to refer to the set of synergistic
or intertwined and mutually enhancing health and social problems facing
the urban poor. Violence, substance abuse, and AIDS, in this sense, are not
concurrent in that they are not completely separable phenomena.”
“We have introduced the term ‘syndemic’ to refer to the set of synergistic
or intertwined and mutually enhancing health and social problems facing
the urban poor. Violence, substance abuse, and AIDS, in this sense, are not
concurrent in that they are not completely separable phenomena.”
-- Merrill Singer-- Merrill Singer
Strengths and Further FrontiersStrengths and Further Frontiers
Positioned Evaluation As…
Essential, not a question of whether, but rather how
Continuous, not left to the end
Inclusive, powerful roles for all stakeholders
Situational, decisions change based on participants, context, questions, uses, resources, etc.
Practical, judged against explicit standards
Humane, primarily about values and learning
Positioned Evaluation As…
Essential, not a question of whether, but rather how
Continuous, not left to the end
Inclusive, powerful roles for all stakeholders
Situational, decisions change based on participants, context, questions, uses, resources, etc.
Practical, judged against explicit standards
Humane, primarily about values and learning
Left Unexamined…
Focus on a singular “program” as the unit of inquiry (N=1 depth)
Limitations of logic (counterintuitive effects)
Dynamic aspects of public health work (e.g., better-before-worse patterns of change)
Democratic aspects of public health work (e.g., actions by citizens who don’t self-identify as health professionals)
Evaluative aspects of planning
Left Unexamined…
Focus on a singular “program” as the unit of inquiry (N=1 depth)
Limitations of logic (counterintuitive effects)
Dynamic aspects of public health work (e.g., better-before-worse patterns of change)
Democratic aspects of public health work (e.g., actions by citizens who don’t self-identify as health professionals)
Evaluative aspects of planning
“When we attribute behavior to
people rather than system structure
the focus of management becomes
scapegoating and blame rather than
the design of organizations in which
ordinary people can achieve
extraordinary results.”
“When we attribute behavior to
people rather than system structure
the focus of management becomes
scapegoating and blame rather than
the design of organizations in which
ordinary people can achieve
extraordinary results.”
-- John Sterman-- John Sterman
Sterman J. System dynamics modeling: tools for learning in a complex world. California Management Review 2001;43(4):8-25.
“The tendency to blame other people instead of the system is so strong
that psychologists call it the fundamental attribution error.”
“The tendency to blame other people instead of the system is so strong
that psychologists call it the fundamental attribution error.”
Beyond ScapegoatingBeyond Scapegoating
Utility (7)Serve information needs of intended users
Feasibility (3)Be realistic, prudent, diplomatic, and frugal
Propriety (8)Behave legally, ethically, and with due regard for the welfare of those involved and those affected
Accuracy (12)Reveal and convey technically accurate information
Utility (7)Serve information needs of intended users
Feasibility (3)Be realistic, prudent, diplomatic, and frugal
Propriety (8)Behave legally, ethically, and with due regard for the welfare of those involved and those affected
Accuracy (12)Reveal and convey technically accurate information
Standards for Effective EvaluationStandards for Effective Evaluation
Joint Committee on Educational Evaluation, James R. Sanders (Chair). The program evaluation standards: how to assess evaluations of educational programs. 2nd edition ed Thousand Oaks, CA: Sage Publications, 1994.
Institutional Climate for Evaluative InquiryInstitutional Climate for Evaluative Inquiry
Milstein B, Chapel T, Wetterhall S, Cotton D. Building capacity for program evaluation at the Centers for Disease Control and Prevention. New Directions for Evaluation 2002;2002(93):27-47.
Links to Cross-Cutting Trends
Expanding goals (length, quality, equity)
Wider ecological/systems view (assure the conditions for health)
Deepening democratic processes/outlook
Public accountability
Multi-sector action
Information systems integration
Public health research
Knowledge management
Systematic reviews
Performance standards
Ethical and moral considerations
Excellence in science and practice
Links to Cross-Cutting Trends
Expanding goals (length, quality, equity)
Wider ecological/systems view (assure the conditions for health)
Deepening democratic processes/outlook
Public accountability
Multi-sector action
Information systems integration
Public health research
Knowledge management
Systematic reviews
Performance standards
Ethical and moral considerations
Excellence in science and practice
“Living conditions are the everyday environment of people, where they live, play and work. These living conditions are a product of social and economic
circumstances and the physical environment–all of which can impact
upon health–and are largely outside of the immediate control of the individual.”
“Living conditions are the everyday environment of people, where they live, play and work. These living conditions are a product of social and economic
circumstances and the physical environment–all of which can impact
upon health–and are largely outside of the immediate control of the individual.”
-- World Health Organization-- World Health Organization
Definition:Living Conditions
Definition:Living Conditions
World Health Organization. Health promotion glossary. World Health Organization, 1998. Accessed July 15 at http://www.who.int/hpr/NPH/docs/hp_glossary_en.pdf.
Public work is sustained, visible, serious effort by a diverse mix of ordinary people that creates things of lasting civic or public significance.
Public work is sustained, visible, serious effort by a diverse mix of ordinary people that creates things of lasting civic or public significance.
Mitchell Siporin. Jane Addams memorial. Illinois Federal Art Project, WPA, 1936. Fine Arts Collection, General Services Administration.
Center for Democracy and Citizenship. The concept and philosophy of public work. Center for Democracy and Citizenship, 2001. Available at <http://www.publicwork.org/1_2_philosophy.html>.
Definition:Public WorkDefinition:Public Work
-- Center for Democracy and Citizenship-- Center for Democracy and Citizenship
1. The power of citizens to direct the course of change toward a negotiated set of valued conditions/goals.
2. Vitality of a society’s public sphere, the health of its polis.
1. The power of citizens to direct the course of change toward a negotiated set of valued conditions/goals.
2. Vitality of a society’s public sphere, the health of its polis.
Lyon D. Tottle House. . .occupied during a sit-in by some of America's most effective organizers. Washington, DC: Library of Congress; 2002.
Lyon D. Tottle House. . .occupied during a sit-in by some of America's most effective organizers. Washington, DC: Library of Congress; 2002.
Definition:Public Strength
Definition:Public Strength
Milstein B. Syndemic. In: Mathison S, editor. Encyclopedia of Evaluation. Thousand Oaks, CA: Sage Publications; 2004.
What are the Prospects for Using aSyndemic Orientation?
What are the Prospects for Using aSyndemic Orientation?
A way of thinking about public health work that focuses on connections among
health-related problems, considers those connections when developing
health policies, and aligns with other avenues of social change to assure the
conditions in which people can be healthy
A way of thinking about public health work that focuses on connections among
health-related problems, considers those connections when developing
health policies, and aligns with other avenues of social change to assure the
conditions in which people can be healthy
Complements single-issue prevention strategies, which can be effective for discrete problems but often are mismatched to the goal of assuring conditions for health in its widest sense
Incorporates 21st century systems science and political sensibilities, but the underlying concepts are not new. Still, the implications of adhering to this orientation remain largely unexplored.
Complements single-issue prevention strategies, which can be effective for discrete problems but often are mismatched to the goal of assuring conditions for health in its widest sense
Incorporates 21st century systems science and political sensibilities, but the underlying concepts are not new. Still, the implications of adhering to this orientation remain largely unexplored.
Seeing Conditions as FreedomsSeeing Conditions as Freedoms
Adverse living conditions are circumstances that inhibit people's freedom to be safe and healthy and develop their full potential
They include, at a minimum, any deviation from prerequisite conditions for life and human dignity (e.g., physical extremes, violence, deprivation, disconnection)
Phenomena like hunger, homelessness, joblessness, illiteracy, war, environmental decay, and various forms of injustice, including racism, are all examples of adverse living conditions
Adverse living conditions are circumstances that inhibit people's freedom to be safe and healthy and develop their full potential
They include, at a minimum, any deviation from prerequisite conditions for life and human dignity (e.g., physical extremes, violence, deprivation, disconnection)
Phenomena like hunger, homelessness, joblessness, illiteracy, war, environmental decay, and various forms of injustice, including racism, are all examples of adverse living conditions
Choice and Non-ChoiceChoice and Non-Choice
Levins R, Lopez C. Toward an ecosocial view of health. International Journal of Health Services 1999;29(2):261-93.
“Choices are always made from among
alternatives presented by the social
environment, or by circumstances that were
themselves not chosen…When we recognize the
elements of non-choice in choice, we can escape
the contradiction between social causation and
individual responsibility and understand the
interactiveness of the two.”
“Choices are always made from among
alternatives presented by the social
environment, or by circumstances that were
themselves not chosen…When we recognize the
elements of non-choice in choice, we can escape
the contradiction between social causation and
individual responsibility and understand the
interactiveness of the two.”
-- Richard Levins & Cynthia Lopez
-- Richard Levins & Cynthia Lopez
Learning In and About Dynamic SystemsLearning In and About Dynamic Systems
Benefits of Simulation
Formal means of evaluating options
Experimental control of conditions
Compressed time
Complete, undistorted results
Actions can be stopped or reversed
Visceral engagement and learning
Tests for extreme conditions
Early warning of unintended effects
Opportunity to assemble stronger support
Benefits of Simulation
Formal means of evaluating options
Experimental control of conditions
Compressed time
Complete, undistorted results
Actions can be stopped or reversed
Visceral engagement and learning
Tests for extreme conditions
Early warning of unintended effects
Opportunity to assemble stronger support
Barriers to Learning
Dynamic complexity
Time delays
Inadequate and ambiguous feedback
Poor reasoning skills
Defensive reactions
Inability and costs of experimentation
Barriers to Learning
Dynamic complexity
Time delays
Inadequate and ambiguous feedback
Poor reasoning skills
Defensive reactions
Inability and costs of experimentation
Sterman JD. Learning from evidence in a complex world. American Journal of Public Health (in press).
Sterman JD. Business Dynamics: Systems Thinking and Modeling for a Complex World. Boston, MA: Irwin McGraw-Hill, 2000.
“The complexity of our models vastly exceeds our ability to understand their implications without simulation."
-- John Sterman
“The complexity of our models vastly exceeds our ability to understand their implications without simulation."
-- John Sterman
Curve-Fitting Models
Dynamic Models
Curve-Fitting Models
Dynamic Models
Curve-Fitting Models
Dynamic Models
Insight Generating
Capacity
Descriptive Realism
Mode Reproduction
Ability
Transparency
Relevance
Ease of Enrichment
Fertility
Formal Correspondence
with Data
Point Predictive Ability
Source: Randers JR. Elements of the System Dynamics Method. Cambridge Mass: Productivity Press, 1980. p. xx.
Curve-Fitting Models
Dynamic Models
Different Modeling Approaches For Different Purposes
Different Modeling Approaches For Different Purposes
Logic Models
(flowcharts, maps or diagrams)
System Dynamics
(causal loop diagrams and simulation models)
Forecasting
Models
Articulate steps between program actions and results
Improve understanding about the possible effects of a policy over time
Focus on patterns of change over time (e.g., long delays, worse before better)
Make accurate forecasts of key variables
Focus on precision of point predictions and confidence intervals