Health Systems Research Centre,Department of Sociology, University of Limerick – Annual Conference 2008

Evidence, policy and management in the English NHS

Kieran Walshe

Professor of Health Policy and Management

Manchester Business School, UK

kieran.walshe@mbs.ac.uk

Overview

• The rise and rise of evidence based medicine

• The spread of the evidence based “movement” into other areas – social policy, criminal justice, education and so on

• The “rise” of evidence-based policy and management

• Moving towards evidence-based healthcare policy and management

Evidence-based medicine

• Evidence-based medicine is “the conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients.” (Sackett 1996)

• Many studies showed that only 20-50% of clinical decisions based on “best evidence”

• Influence of out of date textbooks, teaching, so-called experts often greater than the evidence from well designed research studies

Underuse, overuse and misuse

• Underuse – not using an intervention known to be effective, like post-MI thrombolytic therapy

• Overuse – using an intervention known to be ineffective, like asymptomatic third molar extractions (wisdom teeth)

• Misuse – using an intervention for which there is inadequate evidence of effect, like some prostheses used in hip replacement

The achievements of evidence-based medicine

• Huge improvements in the technical stuff – getting evidence and making it available in usable forms at the right time to clinical decision makers (Cochrane, Best Evidence, etc)

• Almost complete acceptance by the professionals of the intellectual case for EBM – quickly mainstreamed

• Institutionalisation in healthcare systems – in areas like health technology assessment for new interventions, formularies, clinical practice guidelines

The spread of the evidence-based movement

• Social policy – does reducing social security benefits mean more claimants seek/find work? Does direct payment for social care produce better outcomes?

• Education – what is the most effective way of teaching children to read? Does class size make a difference to teaching outcomes?

• Criminal justice – do custodial sentences for young offenders reduce reoffending rates? Do “Megan’s Law” regimes for paedophiles reduce reoffending or improve child protection?

The rise of evidence based policy

The rise of evidence-based policy

• “Social science should be at the heart of policy-making… We need social scientists to help determine what works and why, and what types of policy initiatives are most likely to be effective.”

David Blunkett, Secretary of State for Education

• “No-one with the slightest common-sense could take seriously suggestions by university researchers that homework is bad for you.”

David Blunkett, Secretary of State for Education

The rise of evidence-based policy

How does the process work?

• Knowledge production

• Problem solving

• Politics and tactics

• Enlightenment

What influences policy decisions?

• Experience and expertise

• Lobbyists and pressure groups

• Habit, precedent and tradition

• Values and ideologies

• Resources and pragmatism

• ..and evidence

The rise of evidence-based policy

• Push – getting research evidence “out there” in forms that policymakers and managers can understand and use through briefings, summaries, presentations…

• Pull – generating demand for evidence, developing skills in appraisal and research use, focusing on the future leadership cadre

• Intermediation – getting researchers and managers or policymakers in the same rooms, discussing the same issues, in the same language – developing brokerage and interchange mechanisms

But what about healthcare management?

• Underuse – skillmix substitution such as taking some work from doctors that can be done by nurses an others

• Overuse – resorting to mergers and reorganisations as a way to respond to problems of performance

• Misuse – implementation of quality programmes (such as TQM, CQI, BPR, lean, six sigma…)

But what about healthcare management

• Managers are less interested in what really works than clinicians?

• Managers are willing to adopt the latest fad or fashion at the airport bookstall?

• Anyone can (and does) claim to be a management expert?

• The evidence may be out there, but its often hard to find and contradictory?

• Stories have more power to convince than dry statistics?

Whom to believe: books on management• In search of excellence:

lessons from America’s best run companies

• Charisma: seven keys to developing the magnetism that leads to success

• Love is the killer app: how to win business and influence friends

• The quest for authentic power: getting past manipulation, control and self limiting beliefs

• The myth of excellence: why great companies never try to be the best at everything

• Leading quietly: an unorthodox guide to doing the right thing

• Business is combat: a guide to winning in modern business warfare

• What would Machiavelli do? The ends justify the meanness

Pfeffer and Sutton (2006)

Clinical Managerial

• Highly professionalised and homogenous culture, coherence in attitudes and beliefs

• Entry restricted to those who have completed formal training, formal body of specialist knowledge

• High value placed on scientific knowledge and research, strong research/practitioner links

• Open and heterogeneous culture, diversity of attitudes and beliefs

• Entry open to whoever is appointed to management role, no defined formal body of knowledge

• High value placed on personal experience and learning, research viewed with suspicion, few research/practitioner links

Clinical Managerial

• Strong biomedical, empirical paradigm, focus on quantitative and experimental methods

• Belief in generalisability and objectivity of findings

• Well organized and bounded literature, amenable to review and synthesis

• Weak social sciences paradigm, more use of qualitative methods

• Tendency to see research findings as subjective, context dependent and less generalisable

• Poorly organised and bounded research literature, not easy to review

Clinical Managerial

• Many decisions about patient care taken every day

• Decisions made by individuals often with few constraints

• Decisions homogenous – involve applying common body of knowledge to specific circumstances

• Long tradition of using decision support systems – guidelines, handbooks,formularies

• Often rapid feedback on results of decision

• Fewer decisions taken, generally about bigger and less clear cut issues

• Decisions often made by groups, negotiated, and constrained

• Decisions heterogeneous – involve different evidence base each time

• Not much use of decision support systems of any kind

• Results of decision often far into the future, and hard to measure or link to decision

About the NIHR Service Delivery and Organisation research programme• Created in 2000, and is a (small) part of NIHR which

brings together DH investment in health research

• Focused on serving the research needs of the NHS management community

• So far has spent over £40m on over 120 research projects

• Spent around £8.4m in 2007/08 – planned to rise to £12m pa by 2010/11

• Website at www.sdo.nihr.ac.uk contains all our research, publications and lots more

The NIHR Service Delivery and Organisation Programme

improves health outcomes for people by:

• Commissioning research and producing evidence that improves practice in relation to the organisation and delivery of health care, and

• Building capacity to carry out research amongst those who manage, organise and deliver services and improve their understanding of research literature and how to use research evidence.

About the NIHR SDO programme

Key criteria for research priorities

• Expressed need for research from the NHS

• Capacity to generate new knowledge

• Organisational focus consistent with SDO mission

• Areas of sustained interest and intent

• Not covered by others (RfPB, PRP, HTA…)

• Actionable findings – likely to result in real changes

• Building on work that SDO has undertaken already

Evidence based management is…

• Treating your organisation as an unfinished prototype

• Promoting innovation through careful experimentation

• Focusing on facts, not conventional wisdom or stories

• Using sound logic and careful analysis of data

• Recognising the strengths and weaknesses of innovations

• Being suspicious of “breakthrough ideas”, fads and fashion

Evidence based management is not…

• Copying – just doing what (is said to have) worked elsewhere without understanding or testing it

• Doing what worked – or seemed to – in the past

• Following deeply held and often unchallenged beliefs and ideologies

• Jumping on the latest management bandwagon

• Ignoring the evidence and following your instincts

Evidence for management decision-making

Theoretical Empirical

Experiential

Evidence for management decision-making

• Empirical evidence – “does it work?”

– Data from before and after implementation, or from intervention and control sites, on key outcomes

• Theoretical evidence – “how and why does it work?”

– Explicit programme logic explaining how the intervention is meant to work, and what effects it may or will have

• Experiential evidence – “when and for whom does it work?”

– Views, opinions and perspectives of those involved and key stakeholders on the intervention, process of implementation and its outcomes

Example: pay for performance

• A very fashionable idea – P4P initiatives started in the US and have been adopted in the UK and elsewhere

• Leapfrog employers group, Medicare P4P programmes, a third of HMOs…

• UK Quality and Outcomes Framework – ties 25% of GP remuneration to whole set of quality indicators

Pay for performance: theoretical evidence

• What are the underlying theories behind paying for performance?

– Motivational effect (people will work harder if incentivised to do so by financial reward)

– Informational effect (people will be more aware of the “outcomes that matter” and will be able to prioritise them)

– Selection effect (people with the requisite skills will be attracted by the financial reward on offer)

Pay for performance: theoretical evidence

• Motivation – extrinsic and intrinsic motivation, importance of context, effects of financial incentives on social cohesion

• Informational – extent of individual control and interdependence in performance

• Informational - measurement issues - incentivisation and perverse behaviours – hitting the target but missing the point, effect on unincentivised activities, etc

• Selection – potential conflict between primary financial motivation and other aims/purposes

Pay for performance: empirical evidence

• Christianson, Leatherman and Sutherland (2007). Financial incentives, healthcare providers and quality improvement: a review of the evidence. London: Health Foundation.

• Reviewed 36 studies of financial incentives to healthcare providers – found very limited and mixed effects on performance

• Difficult to measure effects of P4P separately from other health system changes

• P4P results in pay dispersion and greater inequality

Pay for performance: experiential evidence

• Physicians involved in P4P experiments found it “burdensome and time-consuming” and were more interested in how to be “a good doctor”

• Physicians comply – reluctantly – but doubt the accuracy of the P4P measures and are unwilling to act in ways that conflict with patient expectations or their own views

• Social, behavioural and cultural consequences of P4P underresearched and not well understood

Pay for performance: conclusions

• P4P works best in areas where work is about individual effort, work outcomes can be clearly measured, and there is existing strong financial/extrinsic motivation

• P4P works least in areas where work is interdependent, outcomes are hard to measure, and there are strong non-financial, professional or altruistic motivations – like healthcare!

• P4P can work with very modest levels of reward, paid on a team rather than individual basis, with attention to the behavioural/social effects, and alongside other forms of incentives

Conclusions

• How can we expect clinicians to practice EBM when managers and policymakers don’t?

• Managerial decisions can have a huge impact on patient care, and should be treated just as seriously as clinical decisions

• Evidence-based policy and management is not impossible – but requires skills/competencies, data and resources – push, pull and intermediation

• Evidence based policy and management requires a culture change in the healthcare policy and management community – that’s the biggest challenge