Title page

Title pageEvidence Check:

Commissioning better research syntheses

Sally Redman,

Danielle Campbell,

Gai Moore

Sax Institute Mission

To improve health and increase the quality and performance of health services and programs by:

building excellent policy and practice focused research and

increasing the impact of this research on health policy, programs and services

Is research used to inform policy in NSW?

• Policy Maker Survey - 38 senior policy makers from the NSW Department of Health and Area Health Services

• Researcher Survey - 41 senior researchers from 29 Universities and research centres across NSW

0

10

20

30

40

50

60

70

80

90

%

Policy makers Researchers

Proportion perceiving a HIGH need to improve the use of research in policy

Source: Policy Maker Survey and Researcher Survey

Perceived facilitators of evidence in policy

NSW policy makers

Better access to researchers

Better access to research/reviews

Policy-relevant research

Innvaer et al, 2002(Review of 24 interview studies of health policy makers)

Personal contact with researchers

Timely relevance

Inclusion of summaries & recommendations

Increasing the impact of research:Some goals of the Sax Institute

Increase research in areas where NSW Health has knowledge priorities

Give policy makers more timely and effective access to research findings

Bring policy makers and researchers closer together to more easily understand each others expertise and priorities

Identify research findings that are of potential importance to policy and assist researchers to package and communicate their findings in ways that can influence the policy agenda.

Ability to find research to inform policy in last 12 months

68% found brief research summaries difficult or impossible to find

84% found more systematic reviews difficult or impossible to find

Difficulty in finding research summaries nominated as one of greatest barriers to using evidence in policy

Policy Maker Survey

What goes wrong?Policy makers…

Reviews are:

Too slow

Too focused on researcher’s interests rather than policy question

Too time consuming to commission

Too long with key messages buried

Focused on need for more research rather than best summary of current position

Policy makers’ feedback about research reviews

“I don’t think we use reviews much at all – people don’t even think of it”

“A couple of times either I didn’t specify clearly enough what I wanted or the specs changed and what they produced...was awkward for the Department”

“Also there’s the timing issue, you don’t want to have to put a lot of effort into a tendering process that won’t get you a product when you really need it”

What goes wrong?Researchers..

Commissioned reviews:

Are often buried and cannot be published

Are frustrating because policy makers do not know what they want and change their mind

Compete with research through urgent tender requests

Prop’n of researchers perceiving that little value is placed on efforts to get research into policy

0

10

20

30

40

50

60

70

80

90

%

When funding bodies assess trackrecord

When universities consideremployment/promotion

Source: Researcher Survey

Evidence Check

To enable policy makers to more readily access high quality reviews of research suited to their needs

Will also:• Encourage researchers to undertake reviews for

government• Result in greater knowledge of research expertise in

NSW among policy makers

Knowledge Brokers

Evidence Check

Diagnosis ReviewIdentification of expertise

Commissioning Tool

Partnership and production guidelines

RADAR

Preferred Provider Rules

Evidence Check: Diagnosis

Evidence Check Commissioning Tool:

Series of structured questions to assist policy makers to analyse and make explicit their needs

Knowledge brokers:

Individuals with experience in policy AND research - trained and supported to work between researchers and policy makers

Commissioning Tool

1. Background and purpose of the policy• How will the review be used, at what stage in the

policy process, and by whom?• What is the political context?• What risks are associated with the policy issue?

2. Questions to guide the review• What are the specific questions that need to be

answered (define: intervention, target population and outcome)?

Commissioning Tool3. Timeframe and funding

• Is an answer required immediately, in the short term, or in the long term?

• What funds are available?

4. Depth of analysis of evidence• Systematic review or a summary of key findings?• Are strength and recency of evidence important?

5. Format of final product• How long should the report be?• What level of language is appropriate?

Case Study: Advanced Care Planning

Evidence to support a policy approach to comprehensive advancedcare planning (ACP) for end-of-life choices

Is there evidence ofinterventions that

support a community-wide approach to

ACP for end-of-lifecare?

What types ofinterventions that

support a community-wide approach toACP have been

implemented andproven effective?

Is there evidence ofinterventions for othersensitive issues that

could provide lessonsto support a

community-wideapproach to ACP?

Evidence need initiallydefined by policy makers

Review questions agreedafter meeting with broker

Findings from scopingexercise by broker

Recommendation

A recent meta-analysis provides a basis for identifying key features of interventions, and other reviews of ACP and palliative care may support these findings. Reviews of approaches to other issues (eg

domestic violence) may also describe relevant strategies

Commission researcher to: identify relevant high-quality meta-analyses and reviews; identify key features of effective interventions

in community settings; and develop recommendations for a policy and program approach in NSW

Case Study: Type 2 Diabetes Prevention

Evidence to support the development of strategies to reduce theincidence of type 2 diabetes in CALD communities

Are there CALDgroups in NSW withincreased incidenceof type 2 diabetes

or increasedprevalence of risk

factors?

What are the factorsthat may contribute to

higher risk levels inspecific CALD groups?

What evidenceexists for interventions

that are likely to beeffective in reducingtype 2 diabetes riskfactors in specific

CALD groups?




Recommendation

Risk factor prevalence and disease incidence data is available fromgovernment bodies and diabetes associations. Data on factors thatinfluence risk and incidence in specific CALD groups can be derived

from these sources and the peer reviewed literature

Commission researcher to undertake a review and critical analysis ofepidemiological data on risk factor and disease prevalence, and

evidence on the most effective prevention approaches for specificCALD groups

Knowledge Brokers

Evidence Check


Commissioning Tool


RADAR


Evidence Check: Identifying expertise

RADAR: Database

38 research centres and Universities that are members of the Sax Institute used to generate a database of expertise

Selecting reviewers: Preferred provider

Competitive tender avoided by decision rules for selecting individuals with best expertise based on discipline base (eg epidemiology, health economics), content knowledge (i.e. diseases, settings, interventions, and populations), and review experience

Prop’n of researchers interested in commissioned research reviews

0

10

20

30

40

50

60

70

80

90

%

Funded by governmentto conduct review in

last 2 years

High interest inconducting

commissioned reviews

High interest in joiningregister of reviewers

Source: Researcher survey

Knowledge Brokers

Evidence Check


Commissioning Tool


RADAR


Evidence check: the review

Range of review types depending on policy issue(eg systematic review, review of evaluations, ‘headline’ review)

Guidelines for reviewers(eg CHSRF 1:3:25 format)

Translation process by knowledge broker

Feedback from the pilot study

Provided a valuable framework for critical thinking and discussion among policy makers

Helped policy makers move from a general issue to specific questions

Helped policy makers anticipate the potential challenges and risks that could result in a product that does not meet policy needs

Outcomes so far

7 reviews using Evidence Check commissioned so far by NSW Health in broad range of areas: e.g. health workforce, diabetes prevention, HIV services, obesity, population health priorities

Rapid increase in demand over the past few months

2 groups that used Evidence Check during pilot/early stages have returned with new project requests

Case Study: Health Workforce

Evidence to support improved workforce planning and educationfor future health needs especially overseas trained doctors

What are thecharacteristics of

newly appointed non-specialist doctorstrained overseas or in Australia?

What skills andknowledge gaps canbe identified in newly

appointed doctors trained overseas and in Australia ?

How are thesedoctors perceived bythe community andthe health system?




Recommendation

Little evidence exists to clarify: the extent to which NSW doctors aresupported and trained; community perceptions of types of doctors;

barriers and facilitators of effective workforce participation; and flow-oneffects of new training/recruitment methods.

Commission researcher to provide an objective, descriptive andanalytical profile of doctors practicing in NSW who have been trained

in Australian medical schools or overseas.

Documents

Title page