Developing a Spatial Evaluation Framework to Address Social and

Health Problems

Perspectives on Impact Evaluation: Approaches to Assessing Development Effectiveness

2009

Derek Powell, Independent Consultant, Formerly Deputy Director General of the South African Department of

Provincial and Local Government, S. Africa

Julia Koschinsky, Arizona State University

Sanjeev Sridharan, University of Toronto

Outline

Why think spatially?

How will spatial thinking help us with program planning/policy response? Examples

Examples of spatial approaches to policy from S. Africa

How can spatial thinking help with evaluative activities? Example of key spatial concepts

Towards a spatial framework to integrate monitoring, evaluation and program/policy planning?

1. Why think spatially

• Because understanding the spatial dimensions that drive the process that generated relevant differences between areas and spatial concentrations can improve the design, evaluation and effectiveness of an intervention

• Because spatial analysis has the potential to address problems of institutional fragmentation and silo thinking

• Because interventions can be designed to explicitly address the spatial nature of problems such as – rural-urban inequalities (National Spatial Development

Perspective: SA)

1. Why think spatially

• Because the spatial dimension of an intervention can have an independent or interacting effect – E.g., concentration of poverty/neighborhood effects debates in US – e.g., does concentration of low-income residents in large-scale

public housing in underdeveloped areas make it harder to escape poverty?

– E.g., does moving subsidized tenants to “areas of opportunity” make it more likely to escape poverty?

1. Under what conditions can thinking spatially make sense?

In the presence of: Spatial Heterogeneity: Relevant differences between

areas e.g., water quality in informal settlements vs

downtown Spatial Dependence: Concentrations in space

e.g., geographic clusters of residents with a contagious disease

E.g., townships with lower-quality public services set up to serve well-served wealthier inner-city areas

1. An example: Why space, inequality and redress are inter-linked in SA

– Apartheid used space to separate black and white citizens

– Communities spatially separated

– Democratic government inherited legacy of inequality

– Huge racial inequality – per capita wealth, ownership, skills, services

– Former ethnic homelands deeply impoverished (14 million people)

– Poverty, unemployment, inequality are high, institutional capacity weak – especially in rural areas and former homelands

2. How will spatial thinking help us with program planning/policy

response? Examples

• A spatial approach is particularly well suited to integrate information on interventions from multiple jurisdictions – E.g., spatial visualization: African HIV rates in context

http://www.gapminder.org/downloads/gapminder-hiv-chart-2009/– Gap analysis: : http://www.gapminder.org

/downloads/world-health-chart/ – Possible spatial spillovers: http://www.healthmap.org/en – E.g. Neighborhood Early Warning Systems: advance indicators

of nhood decline (http://www.knowledgeplex.org/showdoc.html?id=39186)

2. Example of an Monitoring and Evaluation Question

The Centralized Planners problem:

Consider a State with multiple localities/jurisdictions. As an example in the case of Virginia we have 135 counties/cities.

How can the State planner benefit from a knowledge of spatial patterns of key indicators in the 135 counties? Can a methodological tool help the State planner help identify counties with exemplary trajectories of key indicators?

Between generalization and single cases

2. Problem: Learning from localities

• States generally do a poor job of actively learning from “innovative” practices within localities.

• The standard assumption is that transfer of knowledge and “innovations” move from States to localities. There is a more limited focus on learning from the localities.

2. Distribution of Children’s Risk: 1995

(2) Towards spatial analysis

• …We need some analysis to help identify patterns: towards a typology of patterns

• A typology of five clusters:– Not significant– High-High– Low-Low– High-Low– Low-High

(2) Local Patterns of Children’s Risk: 1995

(2) Local Patterns of Children’s Risk: 2001

3. Examples of project

Example from S. Africa

An interpretation of the spatial realitiesinterpretation of the spatial realities and the implications for government intervention

A shared understanding of the national space shared understanding of the national space economyeconomy by describing the spatial manifestations of the main social, economic and environmental trends

National Spatial Development Perspective

A set of principles and mechanisms for guiding infrastructure investmentinfrastructure investment and development development decisionsdecisions

Purpose of NSDP

Poverty, inequality and deprivation are manifested in space. Can allocation of resource and allocation of resource and directing investment happen in a coordinated directing investment happen in a coordinated and spatially-targeted wayand spatially-targeted way ? ?

This implies a greater appreciation of spatial This implies a greater appreciation of spatial challengeschallenges

Given our objective to grow the economy, create jobs, address poverty and promote social cohesion, the NSDP assists government in confronting three fundamental planning questions:

Where should government direct its direct its investment investment and and development development initiativesinitiatives to ensure sustainable and maximum impact?

How can govt as a whole:

− Capitalise on complementarities and facilitate consistent decision-making?

− Move beyond mere focusing on integration and coordination procedures to establishing processes and mechanisms that would bring about strategic co-ordination, interaction and alignment?

What kinds of spatial forms and arrangements are more conducive to the achievement of our objectives of democratic nation building and social and economic inclusion?

Purpose of NSDP

Approach

Spatial marginalisation from economic opportunities is still a significant feature of our space economy that need to be addressed in order to reduce poverty and inequality & ensure shared growth

Apartheid spatial planning ensured that the mass of our people were located far from social and economic opportunities

Spatial configuration of SA is not only the product of growth but also apartheid spatial planning – a disjuncture exist between where people live and where economic opportunities exist

The spatial concentration of growth in South Africa reveals: South Africa is not unique…

Approach

Regions which already have some economic success are more likely to grow than other regions – this is because individuals, firms, industries and regions LEARN

Successful learning occurs when institutions and incentives work and institutions are locally specific

Unfocused infrastructure spending doesn’t improve GDP growth

Unfocused human resource development doesn’t improve GDP growth

Success achieved through focused and polarised investment

International case studies and theory further shows…

NATIONAL SPATIAL

DEVELOPMENT PERSPECTIVEApproach

Location is critical for the poor to exploit opportunities for growth

Areas with demonstrated economic potential provide greater protection due to greater diversity of income sources

Govt needs to ensure that poor are able to benefit fully from growth and employment in these areas

Areas with demonstrated economic potential are most favourable for overcoming poverty

NSDP Assumptions

The poor, concentrated around economic centres have greater opportunity to gain from economic growth

The poor are making rational choices about relocating to areas of opportunity

NSDP principles

Contribute to the broader growth growth and development policy and development policy objectivesobjectives of government

The NSDP puts forward a set of normative principles in order to:

Surface the spatial dimensions of of social exclusionsocial exclusion and inequalityand inequality

Lift the burden of unequal and inefficient unequal and inefficient spatial arrangements placed on the spatial arrangements placed on the statestate e.g.:

−high transport subsidies)

−communities (especially the poor) who incur huge transaction costs by having to commute large distances to and from work

Applying the NSDP

Providing a common platform for common platform for structured dialoguestructured dialogue

Undertaking rigorous analysis of the Undertaking rigorous analysis of the space economyspace economy to identify areas of economic significance & concentrations of poverty and relative & dynamic comparative advantage of localities

Applying and using the NSDP within within the three spheres the three spheres of governmentof government (e.g. in PGDSs, IDPs, Sector Plans) means the following:

Applying the NSDP will enable government to:Applying the NSDP will enable government to:

Focus investment and development interventions to ensure maximum and sustainable impact

Capitalise on complementarities & facilitate consistent and focussed decision-making

Move beyond mere focusing on integration & coordination procedures towards strategic coordination, interaction and alignment

4. Space and Evaluation

• Indian elephant example: – Aspatial intervention = Indian gov. built long fence for lots of

money to prevent human-elephant conflicts.– Spatial intervention: Spatial tools to track and predict these

conflicts allow for spatially targeted interventions (fences only in areas with high risk of conflict) that are both more effective and cheaper

– http://ecoworldly.com/2008/10/13/intelligent-african-elephants-using-google-earth-and-cellphone-text-to-report-own-truancy/

5. Good spatial models of integrating planning, monitoring

and evaluation

• Geographic visualization of quantitative results easier to communicate to the public

• Incl. in areas with lower literacy rates

• Public input in and use of spatial web service systems (e.g., http://indiabiodiversity.org)

4. So, how can spatial methods help in developing a monitoring and evaluation

system?

• Exploratory spatial data analysis (Anselin, 2003, 2002)

Absence of strong theory Methods for discovery, not only for testing Movement away from global models,

towards local associations

4. Global vs. Local patterns

• Definitions from Virginia's example:• Global spatial patterns examine the average spatial

relationship across all of the regions in Virginia’s counties and cities. Local spatial patterns examine the spatial patterning in certain regions of Virginia

• The challenge of defining “neighbors.”

• Why focus on global vs. local patterns?• Global Moran / Local Indicators of Spatial Association

1 2 3

4 5 6

7 8 9

Rook Criteria: 2,4,6,8 Bishop Criteria: 1, 3, 7, 9Queen Criteria: 1,2,3,4,6, 7, 8, 9

4. The Challenge of operationalizing spatial contiguity: What are the neighbors of 5?

(Anselin,1997)

4. Local Indicators of Spatial Association (LISA)

• Identify Hot Spots• Significant local clusters

• Significant local outliers• High surrounded by low and vice versa

• Indicate Local Instability• Local deviations from global patterns

5. So What?

5. Relevance of Approach

• From a policy perspective, one of the strengths of our approach is it takes the systemic aspects of interventions seriously

• Building a monitoring system with a focus on learning and communication

5. AN EXAMPLE: TOWARDS A DYNAMIC SYSTEM OF COMPREHENSIVE STATE-LEVEL

PLANNINGDevelop Programs for

“Problem” Counties/Learn from “Exemplary”

Counties

6. Response/LearningDevelop a Geographical

Information System:

State-level Monitoring of Juvenile Crimes, Social Disorganization

Measures, Protective Factors, and Local Planning Efforts

1. Monitoring

To Expand and Improve

State Data Systems

Use Learning

Develop Explanations for “Problem”/“Exemplary”

Crime Trends

5. Diagnosis

Spatial and Growth Models of Juvenile Crime

Trends

2. Understanding Context

Communicate with “Problem”/“Exemplary”C

ounties

4. Communication

Identify “Problem”/“Exemplary”

Counties

3. Identification

5. Understanding complex combinations of interventions

• “If the primary causal mechanism underlying crime trajectories can be found in factors such as single family households, racial heterogeneity and economic deprivation, all linked to the social disorganization perspective, then a much wider set of social interventions would be required to change the form of trajectories at crime hot spots…a complex combination of interventions might be required to have a meaningful and long term impact on crime as hot spots.” (Weisburd et al., 2004)

• Understanding the “complex combinations of interventions” that might characterize the “exemplary” counties is a focus of the intensive case studies.

5. Potential problems/promises

• The problems with social indicators• Don Campbell on the “corrupting effect of using

social science indicators”

• Leveraging social indicator databases• Learning through principled discovery• Embedded processes of change

5. Role of Central Organization

• Diffusion of innovations• Promoting cross-locality learning• Moving beyond a fixation with

performance towards learning• A focus on patterns—not simply on levels

of outcomes• Localities as connected units

• sridharans@smh.toronto.on.ca