Centre for Primary Health Care and Equity

Health Impact Assessment A triumph over common sense?

Ben Harris-RoxasConsultant, Harris-Roxas Health

Conjoint Lecturer, University of New South Wales, Sydney

Health Section Co-Chair, International Association for Impact Assessment

A detailed paper on this talk is available from

www.harrisroxashealth.com/2012/10/korea2012

Background

•We work in increasingly resource constrained systems and all

interventions are expected to demonstrate their usefulness – this

includes HIA

• The use of HIA may seem like “common sense”. It’s already being

used; it has already demonstrating its usefulness. We have good

examples.

Background

• But common sense isn’t good enough. We need to convincingly

demonstrate HIA’s effectiveness.

• This effectiveness can be thought of narrowly, in terms of

changes to decisions and implementation, and evidence of

recommendations being adopted.

• It’s also important to look at HIA’s ability to change ways of

working, understanding and learning.

This presentation

• Draws on a before-and-after study (prospective multiple case

study):

• Two similar equity focused HIAs of service plans

(obesity prevention and management strategies)

• One HIA was conducted, one was screened out

• 23 interviews, conducted before and after the HIAs,

document analysis

The effectiveness of HIA

• The effectiveness of HIA is complex and affected by many factors

• Colleagues at UNSW and I developed a conceptual framework

that attempts to look at the broad range of factors that can be

affected through HIAs, but also the factors that affect them

Source: Harris-Roxas B, Harris E (2012) The Impact and Effectiveness of Health Impact Assessment: A conceptual framework, Environmental Impact Assessment Review: accepted, in press. doi:10.1016/j.eiar.2012.09.003

The need to look at perceptions of

effectiveness

• It was obvious to us from previous research that perceptions of

an HIA matter.

• Perceptions determined the extent to which people attributed

changes to an HIA.

• Perceptions can give rise to conflict.

Source: Harris-Roxas B, Harris P, Harris E, Kemp L (2011) A Rapid Equity Focused Health Impact Assessment of a Policy Implementation Plan: An Australian case study and impact evaluation, International Journal for Equity in Health, 10(6), doi:10.1186/1475-9276-10-6.. http://www.equityhealthj.com/content/10/1/6

Example: What is the purpose of an HIA?

“I think people felt when recommendations came in, that they saw as a critique, or not that they were a critique, because different... They were like ‘Oh, but it wasn’t a proper plan anyway, it was just, you know, we were just trying to get the money, and that was our goal at that time, just get the money, and we said we’d do this, but not sure if we really will’.”

“We didn’t have a shared understanding of why we were undertaking it. Our purposes were probably different[...], and maybe that’s where they don’t work, but if you have two differing purposes, unless you can fully appreciate what those two different purposes are, maybe it doesn’t work out as well as it could... I think there was a feeling that, well, we could get something out of [the EFHIA]. There were probably two rationales for why it would be useful. One is that we could get some, a critique if you like, or some feedback about, through an equity lens, on the strategies that we had proposed. And the second one was that it would perhaps serve a process of helping people who are more engaged in the consultation process.”

Example: What is the purpose of an HIA?

“In a way, it was about improving the quality of the document, it was actually quite important to be able to debate some of the issues.”

Example: What is the purpose of an HIA?

“There were also some things in [the EFHIA report] that, I guess, implied, that we wouldn’t consider, some issues that I think can be dealt with in careful planning, and careful implementation, and the intention, as I said before, if the [ABHI implementation plan] was really about ‘this is the flavour of where we’re going with this’ we’re going to have to obviously have greater implementation plans around each of these strategies, we’ve only got sixty pages to do it in.”

Example: We would have known it anyway

(the recommendations are common sense)

“There are quite dichotomous views about what people believe about HIAs. Some people believe there is a place [for HIAs], blah, blah, blah and they’re fantastic. Other people believe [these issues are addressed as] part of a good planning process, and there’s some there are in between those two.”

Example: We would have known it anyway

(the recommendations are common sense)

Are common sense and HIA synonymous?

• Not really! What seems obvious in hindsight wasn’t really at the

time – this is known as hindsight bias.

• I found several examples of hindsight bias in this study, where

people expressed disappointment that the HIA didn’t result in

radically new information, even though that isn’t what they

hoped for in the interviews before the HIA

(more detailed examples in the paper)

Conceptual challenges in evaluating HIA

So, given:

1. perception of HIA’s effectiveness matters, and

2. we know hindsight bias applies to perceptions of HIA;

how should we address this problem when evaluating HIA?

(Keep in mind almost all evaluations of HIAs

have relied on retrospective/historical accounts)

Challenge 1: Narrative fallacy

• We create narratives (stories) to explain events.

• Stories are created after the event, often to explain what

happened, e.g. “this is why the HIA failed/succeeded”.

• We tend to create stories to suit we want events to be

understood, and this leads to narrative fallacy.

• We cant rely on stories to explain what actually happened

(stories often sound like they’re about different events!)

Challenge 2: Creeping determinism

• Sounds complicated but is essentially the tendency to discount

information and learning because “we knew it all along” or

“the HIA was always going to end up not making a difference”.

• It highlights the need to have clear descriptions about the

purpose and desired outcomes of an HIA before it starts.

What’s required: Prospective evaluation

• We lack counterfactuals – the “what if?” cases

• Comparative prospective evaluation/natural experiments of

similar proposals when HIAs were done

• won’t allow perfect comparisons

• but will allow some comparisons.

• Data also needs to be collected before, during and after HIAs

to attempt to address the problems of narrative fallacy and

creeping determinism.

What’s required: HIAs of health proposals

• There’s an assumption health addresses health well in

planning but not much evidence of this.

• HIAs with a focus on distributional impact and equity have

been useful in getting the health sector to think about the

impacts of health services and programs in a different way.

• Capacity gets built in the health sector, not just to do HIAs but

to set ToRs, review completed HIAs and advocate for HIA’s use

(and work intersectorally)

What’s required: HIAs of health proposals

• This study was only possible of strong relationships and

credibility in the health sector. It would have been more

difficult working with other sectors.

• If health agencies/professionals want to promote Health in All

Policies they will also need to demonstrate they’re serious

about Health in Health Policies.

Conclusions

• HIA has the ability to alter not only plans and documents but

also perceptions and understandings, often in ways that are

only poorly accounted for by participants in retrospect

(looking backwards).

• Impact assessment, at its best, provides useful

recommendations that are timely and linked to

implementation.

• The recommendations can seem obvious!

Conclusions

• Narrative fallacy and creeping determinism can make it seem

like we knew this information all along.

• Common sense is often anything but common, but it can seem

that way in retrospect.

• HIA may be a way to triumph over common sense.

ben@harrisroxashealth.com

@ben_hr or @hiablog

healthimpactassessment.blogspot.com

linkedin.com/in/benharrisroxas

These slides are available at

www.slideshare.net/benharrisroxas