I M P A C T , L E A R N I N G A N D I N N O V A T I O N : T O W A R D S A R E S E A R C H A N D P R A C T I C E A G E N D A F O R T H E F U T U R E

I N S T I T U T E F O R D E V E L O P M E N T S T U D I E S , B R I G H T O N ( U K )

M A R C H 2 6 , 2 0 1 3

J O H N T . G R O V E , M . A . , P H . D . B I L L A N D M E L I N D A G A T E S F D N . *

* C U R R E N T A F F I L I A T I O N . T H I S W O R K W A S C O N D U C T E D B E T W E E N 2 0 0 6 A N D 2 0 1 1 W H E N J G W A S P A R T O F T H E C E N T E R S F O R D I S E A S E C O N T R O L & P R E V E N T I O N A N D A S A N I N D E P E N D E N T

R E S E A R C H D E G R E E

M O D E L I N G W A S C O N D U C T E D B Y D R . J A C K H O M E R

Aiming for utility in ‘systems-based evaluation’: A research-based framework for practitioners

Outline

Defining SBE

Research Objectives, Methods, Stance

Key Features and Utility of SBE

The Zambia Case: Strategic Evaluation with SD

Preparatory (Pre-Model) Phase Findings

SD Model Highlights & Reflections

Post-Model Findings

Recommendations for Practice

What Is Systems-Based Evaluation?

Systems-Based Evaluation (SBE) is the application of specific methods that are influenced by systems concepts and methodologies to the task of evaluation.

These methods articulate and analyze the interrelationships and interdependencies of real-life factors, including perspectives and boundaries, in a process of intervention toward action.

Research Scope & Objectives

The broad objectives of the research were:

1. To investigate the utility of system dynamics (SD) in terms of how it does or does not add value for evaluation of the ART strategy in Zambia.

2. To identify useful principles and methods for the broader application of SBE to other health- services research.

Methods & Philosophical Stance

The research questions and utility framework were informed by relevant literature, in-depth interviews, extensive field notes, and a 6-year personal research journal.

Data collection and analysis for the research was guided by narrative-based inquiry and critical-hermeneutics philosophy:

Gadamer’s (1998) role of language;

Habermas’s (1976) communicative action;

and Ricoeur’s (1985) threefold mimesis.

Literature: Key Features of SBE Approaches

Three unique core concepts in application together are essential for an SBE:

Interrelationships

Perspectives

Boundaries

• (MIDGLEY 2000: 45; WILLIAMS & HUMMELBRUNNER 2010: 3).

Utility Framework: Transformative vs Representative

Tensions of SBE utility:

‘Now that I understand that the problem is a system, I can see

or consider (evaluate) what’s needed to possibly change it. I may be the one to make the change or I can help advocate for someone else to do so.’ (Representative)

‘I was part of describing the problem and the system, I am part of the system, I care about the system, and I value certain factors and connections. I will now go create and be part of the new system.’ (Transformative)

Utility Framework: Transformative & Representative

Utility Framework: Spectrum of Utility— From Alignment to Action

For Utility, we mediate Representative and Transformative on a spectrum:

aligning pre-understandings

developing a shared comprehension of the dynamic situation and components

providing space for argument on the validity of opinions and data and reinterpretation of inter-relationships, perspectives, and boundaries.

Move to action

Trigger, Comprehend, Validate, Re-Interpret, Act

Case Study: Zambian ART Scale-Up Strategy & Context

2005–2009 :

Zambian Ministry of Health’s ART scale-up strategy aimed to rapidly expand HIV/AIDS services

Overarching goal to achieve dramatic change in the epidemiological profile of HIV/AIDS in the country.

Complexity of the Zambian ART scale-up plan raises need for dynamic and systems-oriented view in the midst of linear target-setting.

Opportunities:

1) Apply a systems lens, and 2) Study the process

Preparatory Research Data and Findings

Data revealed three common themes:

1. Critique of current M&E practice for ART

2. Necessity for boundary-setting for the system of interest

3. Interest in systems-oriented approaches but cautious about new methods.

Preparatory Research Data and Findings

So we’re good at getting the clinical stuff, and we’re good at counting things and following patients, but we’re bad at the more psycho/social stuff … ‘Psycho/social’ is a term of art, which, I don’t really know what it means exactly. When I say it I’m talking about taking care of the patient’s non-medical needs, so: counselling and making sure that the systems and structures are in place in the family, in order to take care of the family unit. In the point of view of things we measure, we’d like to know how well those other services are doing, because they affect the clinical outcomes that we’re trying to deliver.

- Rep from lead NGO partner

Preparatory Research Data and Findings

Case Study: Strategic Evaluation

Objectives:

to assess the typical point at which the ART patient load in any given clinic has reached beyond its capacity to provide sustainable, long-term care

to determine how increasing the number and/or type of supportive services networked with an ART clinic reduces the patient load of the clinical site

to ascertain what level of access to and participation in supportive services is necessary to sustain ART for clients into the future

Strategic evaluation is for: ‘evaluating relationships among elements of strategy’, including ‘coherence of different strategy elements, complexity of interrelationships, understanding of interdependence and interrelationships, and evidence of systems thinking and complexity understandings’ (Patrizi: 2010: 99)

SD Model Highlights

SD Model Highlights

800,000

600,000

400,000

200,000

0

1990 2000 2010 2020 2030 2040

Year

Treating OI’s

Screening

Enrolling

Monitoring

Hou

rs p

er y

ear

Non-HIV tasks

The Monitoring “Bulge”

Strategic Evaluation Conclusions

1. Zambia’s strategic goal to ‘prevent, halt and begin to

reverse the spread and impact of HIV and AIDS by 2015’ might be plausible for Lusaka if the huge need for additional clinical staff is met.

2. Unless supportive service workers can perform

particular tasks (e.g. screening & monitoring) tasks, their presence may actually increase patient load.

Post-Model Data and Findings

SD model: 1. readily triggered stakeholders to reflect on their

existing boundaries (i.e., their beliefs and values) 2. stimulated reinterpretation of interrelationships 3. generated new possible boundaries of the ART care

system Stakeholders:

1. had comfort with and an understanding of the model’s structure talked about concerns with the perceived complicatedness of the approach

2. expressed concern about the potential for false assumptions to be reified in the model calculations

3. placed confidence/trust in investigators to be rigorous

Post-Model Data and Findings

Benefits SD model triggered discussion and elaboration of

system boundaries among participants, trending towards unified understandings.

SD model generated new and unfolding understandings through dialogue.

Constraints Process did not lead to transformative action could have been more useful had stakeholders been

engaged in intensive boundary-setting dialogue both during the development and after the completion of the model

Discussion: Utility of SBE

Participants asked for more time and opportunities to understand modeling and analysis.

Need to collect a broad variety of perspectives at very outset of modeling process.

SBE demands planner/evaluator to emphasize representative and/or transformative process aims.

Determining system’s boundaries is methodological requirement/asset for/of systems-based approaches: transformative power of this discussion in itself, where it can be an evaluative intervention.

SBE can be facilitated with an aim to move stakeholders to consensual coordinated action.

Sometimes things fall apart…

Conclusions for SBE Practice

Acknowledgments Government of the Republic of Zambia, Ministry of Health CDC Zambia Bill and Melinda Gates Foundation Patricia Rogers Bob Williams John Barton Jack Homer Countless others…..

Dissertation is available at:

http://researchbank.rmit.edu.au/eserv/rmit:160050/Grove.pdf

Thank You