Matching Evaluation to the Nature & Conceptualization of the Problem

Matching Evaluation to the Nature &

Conceptualization of the Problem

Simple, Complicated and Complex Problem Framings and the Implications for Evaluation

Brenda Zimmerman, Schulich School of BusinessYork University, Toronto, Canada

Presentation to the Joint Canadian Evaluation Society/American Evaluation Association Conference

Toronto Oct 29, 2005

Social Transformation – our definition An alteration of what is established by the

introduction of new elements or forms (including new ideas, practices, or resource flows). In particular the alteration of social relationships to allow for an improvement in or transformation of intransigent and broadly based social problems

Our approach to social innovation Connecting understanding of large scale

patterns or dynamics to action at the level of individuals or groups

Concerned with how to understand these interactions in such a way that we could help those trying to make a difference

The Decision to Use a Complexity Lens Complexity theory leads us to look at the

relationship between the micro and the macro It suggests how individuals act effectively in

arenas which they cannot control (complex is more than complicated)

Complexity theory points to what is happening “in the between”; in the relationships between things.

Implications for evaluation??

Following a Recipe A Rocket to the Moon Raising a Child

Complicated Complex

The recipe is essential Recipes are tested to assure replicability of later effortsNo particular expertise; knowing how to cook increases successRecipe notes the quantity and nature of “parts” neededRecipes produce standard productsCertainty of same results every time

Simple

Following a Recipe A Rocket to the Moon Raising a Child• Formulae are critical

and necessary• Sending one rocket

increases assurance that next will be ok

• High level of expertise in many specialized fields + coordination

• Separate into parts and then coordinate

• Rockets similar in critical ways

• High degree of certainty of outcome

Complicated Complex

The recipe is essential Recipes are tested to assure replicability of later effortsNo particular expertise; knowing how to cook increases successRecipes produce standard productsCertainty of same results every time

Simple

Following a Recipe A Rocket to the Moon Raising a Child

• Formulae are critical and necessary

• Sending one rocket increases assurance that next will be ok

• High level of expertise in many specialized fields + coordination



• Formulae have only a limited application

• Raising one child gives no assurance of success with the next

• Expertise can help but is not sufficient; relationships are key

• Can’t separate parts from the whole

• Every child is unique• Uncertainty of

outcome remains

Complicated Complex

The recipe is essential Recipes are tested to assure replicability of later effortsNo particular expertise; knowing how to cook increases successRecipes produce standard productsCertainty of same results every time

Simple

Following a Recipe A Rocket to the Moon Raising a Child• Formulae are critical

and necessary• Sending one rocket

increases assurance that next will be ok

High level of expertise in many specialized fields + coordination

• Separate into parts and then coordinate



• Formulae have only a limited application

• Raising one child gives no assurance of success with the next

• Expertise can help but is not sufficient; relationships are key

• Can’t separate parts from the whole

• Every child is unique• Uncertainty of outcome

remains

Complicated Complex

The recipe is essential Recipes are tested to assure replicability of later effortsNo particular expertise; knowing how to cook increases successRecipe notes the quantity and nature of “parts” neededRecipes produce standard productsCertainty of same results every time

Simple

• Sources for charts of next three slides

Begun, Zimmerman and Dooley, “Health Care Organizations as Complex Adaptive Systems”, in S. M. Mick and M. Wyttenbach (eds.), 2003 Advances in Health Care Organization Theory San Francisco: Jossey-Bass, pp 253-288

Also can be found at www.change-ability.ca

Complexity Science Established Science --- “complicated”

Holism ReductionismIndeterminism DeterminismRelationships among entities Discrete entitiesNonlinear relationships– critical mass thresholds

Linear relationships– marginal increases

Quantum physics– influence through iterative nonlinear feedback– expect novel and probabilistic world

Newtonian physics– influence as direct result of force from one object to another– expect predictable world

Understanding; sensitivity analysis PredictionFocus on variation Focus on averagesLocal control Global controlBehavior emerges from bottom up Behavior specified from top downMetaphor of morphogenesis Metaphor of assembly

Adapted from Dent 1999

Established Perspectives*

Complexity Science Perspective

TEMPORAL FRAMINGVIEW OF THE FUTURE

Relatively knowable Relatively unknowable

RELEVANCE OF HISTORY

None (transaction cost) to high (institutional). When high, history is deterministic.

High, but history may or may not be deterministic.

SPATIAL FRAMINGDOMAIN OF STUDY

Reified organization in the environment

Relationships among individuals, subsystems, systems

VIEW OF THE ENVIRONMENT

Outside the organization; evolves separately

Part of the domain of study; coevolves with the organization

LEVELS OF ANALYSIS

Single to few, relatively independent

Multitude of nested levels

CONSTRUCT FRAMINGSTRATEGY Relatively designed Relatively emergentSTRUCTURE Equilibrium; relatively

centralizedNon-equilibrium; relatively decentralized

KEY INFO. FOR THE ORGANIZATION

External environmental intelligence

Functioning of relationships

INFORMATION PROCESSOR

Reified organization Individuals; complex systems of individuals

Complexity Science Perspective

Implications for Research

TEMPORAL FRAMINGVIEW OF THE FUTURE Relatively unknowable Patterns may repeat, but

without predictive power. Anticipate surprise. Study emergence.

RELEVANCE OF HISTORY

High, but history may or may not be deterministic.

Requisite to study history (vs. cross-sectional only); conduct longitudinal analysis

SPATIAL FRAMINGDOMAIN OF STUDY Relationships among ind.,

subsystems, systemsStudy patterns of interaction among agents.

VIEW OF THE ENVIRONMENT

Part of the domain of study; coevolves with the org.

Study co-evolution of organization and environment

LEVELS OF ANALYSIS Multitude of nested levels View issue from multiple, nested levels of systems

CONSTRUCT FRAMINGSTRATEGY Relatively emergent Study changes in strategy and

conditions that facilitate changeSTRUCTURE Non-equilibrium;

relatively decentralizedAssess flexibility of structures; simple rules; min specs

PURPOSE OF ORGANIZATIONAL RELATIONSHIPS

Learning; co-creation of meaning; functioning of relationships

Assess degrees of co-participation, learning, sharing; study quality of relationships

KEY INFORMATION FOR THE ORGANIZATION

Functioning of relationships

Study quality of relationships

INFORMATION PROCESSOR

Individuals; complex systems of individuals

Study individuals and coalitions, vs. reified organization

http://www.nationalgeographic.com/resources/ngo/maps/view/brazilt.html

The Brazil Case Study

annual per capita income is less than $5000

In the 1980s, Brazil’s AIDS problem was worse than South Africa’s

By 2000, South Africa’s HIV infection rate was 25% whereas Brazil’s was 0.6%

In 1992, the World Bank predicted that Brazil would have 1.2 million AIDS cases by 2000

…but the actual count was 0.5 million.

Brazil Analysis is from Begun, Zimmerman and Dooley, 2002




The Brazil Complex Problem How do you respond to an AIDS epidemic in a

developing country? No money for expensive drugs

problems of affordability of drugs A very iffy health care system

problems of treatment High levels of illiteracy

problems of compliance even if there were drugs High levels of poverty and hunger

problems of nutritional needs for drugs

Making the Brazil Problem Complicated

“What will drug costs be for our infected population?”

“What resources are needed to manage drug therapies for illiterate patients?”

“What resources are needed to assure compliance with drug associated nutrition in this population?”

“What are the resources needed for an effective prevention program?”

World Bank Responds to AIDS as Complicated

Meaningful solutions require sophisticated, integrated national health care systems

We cannot provide treatment to all when the drug costs are so high

We cannot afford resources to manage treatment compliance

With our limited resources, we should focus more on prevention than treatment

It will therefore take a long time for the problem to work itself through

Brazil Implicitly Recognized the Complexity

Began to address the AIDS issue head-on in 1994

Were unwilling to accept the answers of the World Bank

Hence – had to change the questions… Changing the questions changes the focus,

changes what is “analyzed”, changes what is seen as possible/impossible

The Brazil Questions Assume Complexity

“What will drug costs be for our infected population?”

“What resources are needed to manage drug therapies for illiterate patients?”

“How can we reduce costs so that we can provide treatment to all who need it?”

“What methods of

communication will work to convey the drug therapy routine to a patient – even a homeless, illiterate patient?”

World Bank Questions

Brazil Questions

The Brazil Questions

“What resources are needed to assure compliance with drug associated nutrition in this population?”

“What are the resources needed for an effective prevention program?”

“If food is an issue, how can we ensure greater compliance with the routine by linking up with charities that can provide food at the right times of day?”

“How can we achieve our prevention goals while treating all of those currently infected?”

World Bank Questions

Brazil Questions

The Brazil Conclusions

Meaningful solutions require sophisticated, integrated national health care systems

We cannot provide treatment to all when the drug costs are so high

We cannot afford resources to manage treatment compliance

With limited resources, focus more on prevention than treatment

It will therefore take a long time for the problem to work itself through

Find ways to use the resources we have to respond to the problem

Provide drugs to all by finding ways to reduce drug costs

Use our informal system to train people to care for themselves

Prevention will be part of the treatment

Seek short and long term results

World Bank Conclusions

Brazil Conclusions

The Brazil Responses

A stable container: Brazil built on existing infrastructure (natural network existed) A somewhat shaky health system of

hospitals and clinics Added to by 600 NGOs, churches (hubs)

Free drugs to all AIDS patients: faced down drug companies in pursuit of national interest

The Brazil Responses - cont’d

Patients managed their own drugs Illiterate people were taught by local

“trustworthy” folks (hubs were sought)

Free treatment spread prevention ideas No labeling of those affected Prevention information readily available

Brazil saw the value in the relationships as the key to change

The Brazil AIDS story - optimism COURAGE– to challenge

WTO, USA, large pharmaceuticals, World Bank

COMPLEXITY – lived with the complex nature of the society, and the AIDS problem

RELATIONSHIPS – used the power of existing relationships to learn and enhance their connectedness




Matching Evaluation to Cycles of

Intervention & Program Development

Ecocycle & Panarchy as a Framework

The ecocycle as a metaphor for successful social innovation Based on the work of ecologists studying

natural systems (C.S. Hollings in particular) Suggests four distinct stages in a “life cycle”,

linked to two key dynamics: connectedness or sameness and stored capital or potential

BIRTH

GROWTH

MATURITY

The birth, growth, destruction and renewal of a forest

Variety CONNECTEDNESS Sameness

Exploitation1

Rele

ased

PO

TEN

TIAL

St

ored

04/22/23

exploitation

The Psychosocial place called “exploitation” Triggers: Choice! Groups of people have converged on some promising

products or processes and are devoting time and energy to these. Decision to invest in one or more and build capacity

People: the implementers come into their own. Invest in people who love to take ideas and develop them into reality, the team-builders and the engineers come into their own.

Experience: Excitement, flow, high energy, doors opening, long hours, time flying by….often correlated with peak experiences. A time of intense activity, flow and productivity with a steep organizational “learning curve”

Evaluation: “Formative” – look for a continuous feedback systems as the start-up phase moves towards greater goal clarity and efficiency in delivery

Management: Supporting capacity building. Careful attention to the need for training, hiring and developing systems to support bringing a program “to market” and moving it up the scale. Evaluation geared to providing feedback for increasing effectiveness and efficiency.

Traps: Failure to build sufficient capacity or sufficient “lock-in” to achieve economies of scale and delivery efficiency



Exploitation1

Rele

ased

PO

TEN

TIAL

St

ored

04/22/23

conservation

The Psycho-social place called “conservation” Triggers: Reaching a level of efficiency and learning where

improvements become incremental. The “mature” phase, productivity high and measurable, peak “earnings” and efficiency.

People: Invest in those excellent at management, system design excel, others may be getting restless, or even a little bored.

Experience: satisfaction at success, the pride in the well oiled machine, anxiety about mushrooming demand, loss of momentum

Evaluation: Summative evaluation – success based on goals should be measurable by quantitative or qualitative means.

Management: awareness of vulnerability, standing still (again), openess to creative destruction. Encourage “letting go”

Traps: rigidity, loss of peripheral vision, over-commitment. Fear of change.



Exploitation1

Rele

ased

PO

TEN

TIAL

St

ored

Creative Destruction

Conservation

Creative destruction

Not devastation of “soil” Crisis…. from root word “to sift” Creation requires destruction To honor creation… open to destruction Externally imposed Internally driven

04/22/23

Creativedestruction

04/22/23

Creativedestruction

The psychosocial space called “creative destruction” Triggers: most clearly signaled by the discontinuation, destruction or

collapse of a given program or process and the simultaneous release of resources and energies for other things.

People: those who thrive on crisis, on new beginnings are happy here – others may be depressed or in mourning

Experience: of confusion, identity crisis,, change in relationships, anxiety, elation.

Management: providing reassurance, “standing still”, ‘active” reflection, contemplation, listening.

Evaluation: organizations should be “harvesting”: identifying lessons learned and thinking about implications for the future.

Traps: Fear and guilt can act to isolate – communication is needed.

Outputs: Focus on new ideas, investment of capital in exploration.

RenewalAlso known as Exploration orReorganization

04/22/23

Renewal/exploration

The psychosocial space called “exploration” Triggers: The need for innovation. Development of multiple “random walks”,

experiments, initiatives which lead to little in the way of measurable outcomes for considerable time

People: people who learn by doing are happy here; invest in individuals who are entrepreneurs/ innovators with a track record.

Experience: Reflection moving to experimentation, lots of false starts and sometimes frustration and mounting anxiety about inputs/output ratios

Management: Generate resources and connections and exchange of information, encourage experimentations and learning, have faith in the self-organizing properties of information and groups. Introduce new ingredients as required. Exert increasing pressure for output.

Evaluation: geared to capturing learning, building transparent accounts of innovation.

Traps: “spinning wheels”, the incapacity to select a course and commit resources to it to the required degree. Ungovernable competition.

Conservation 2


Weak CONNECTEDNESS Strong

Exploitation1

Renewal4

3Creative

Destruction

Litt

le

CAP

ITAL

STO

RED

M

uch



Exploitation1

Rele

ased

PO

TEN

TIAL

St

ored

Creative Destruction

ConservationRenewal

POVERTYTRAP

RIGIDITYTRAP

VarietySameness

Released

Stored

The psycho-social space of poverty traps Organization, processes, initiatives: a sense

of spinning wheels, initiatives and interactions are infrequent or are not generative

Experience: “chronic disaster”, increasing hopelessness or helplessness, as sense of the poverty of the very ground of being

Psycho-social experience of rigidity traps Organizations, systems initiatives: rule-

bound, unresponsive, resistant to change, too may resources tied up in non-productive systems.

Experience: a kind of rigor mortis, nothing new seems to happen, creativity discouraged, emphasis on evaluation and performance

Stored

Released

Variety Sameness

Do we evaluate Resiliency? Connectedness is an indicator of how much similarity

vs. variety characterizes the behavior/composition of the system

Potential (or capital) is an indicator of the degree to which resources (time, energy, money, skill, knowledge) are committed to existing projects as opposed to available to attach to new projects

Resilience is a measure of the ability of the system to avoid “traps”, i.e. getting stuck in any phase

Developmental Evaluation

Formative Evaluation

Summative Evaluation

Documents

Matching Evaluation to the Nature & Conceptualization of the Problem