ABSTRACT
Applied prevention research centres (APRC) are important parts of public health efforts
to prevent chronic disease and promote healthy living. How to measure their practical impacts
upon society remains poorly understood. This study aimed to identify indicators considered by a
diverse set of stakeholders to be most important for capturing the practical impacts of APRCs
(outside of contributions to new knowledge); and, to identify opportunities for adaptation and
further development of measures for these most important indicators. A modified Delphi
approach was used to gather the perspectives of centre leaders, funders and knowledge users
associated with 36 APRCs from diverse international settings. An initial set of 22 decision
making and capacity development indicators were gathered from existing research impact
frameworks. During a three round Delphi process, panelists rated these indicators on importance
and feasibility, proposed refinements to existing indicators and developed new indicators. Only
those indicators rated above average on importance were retained between rounds. This process
identified eight indicators that were rated as highly important and highly feasible for collection,
such as the number of APRC projects driven by policy needs, the number and quality of
knowledge exchange activities, and citations of APRC research in public policy documents.
Seven indicators were rated as highly important but with low feasibility, such as measures of
APRC reputation, evidence of contributions to the field of prevention research, and the influence
of the APRC’s work over time on the knowledge, skills and commitment of policy and practice
partners. These indicators may be suitable for future methods development.
1
INTRODUCTION
Research evaluation may be conducted for four broad purposes: to inform advocacy
efforts; to meet accountability requirements; to analyse and understand how, where and why
research is effective; and to inform funding allocation decisions (Guthrie et al. 2013; Penfield et
al. 2014). In many jurisdictions, interest in research evaluation is increasing, driven by a greater
focus on accountability, good research governance, a need to make better choices about limited
funding for research activities, and a desire to limit waste in conducting research (Greenhalgh et
al. 2016; Guthrie et al. 2013). Research evaluation therefore provides an important means for
demonstrating the value of investing in research activities, informing the planning of research
institutions and funding agencies, as well as rewarding past experience and incentivizing future
activity (Greenhalgh et al. 2016; Guthrie et al. 2013; Upton et al. 2014).
Within the field of research evaluation are specific efforts to examine and document
research impact, particularly the ‘societal’ impact arising from research activities (Bornmann
2013, 2016; Penfield et al. 2014). As noted by Bornmann, societal impact is concerned with “the
assessment of (a) social, (b) cultural, (c) environmental, and (d) economic returns (impact and
effects) from results (research output) or products (research outcome) of publicly funded
research” (Bornmann 2013).” Societal impact therefore recognizes the importance of research
that leads to marketable and consumable products or services (Bornmann 2013, 2016).
Despite its importance, assessing or measuring societal impact is difficult. Social,
cultural, environmental, and economic impacts are not mutually exclusive, such as evidenced by
the multiple contributions made by a new medical treatment that improves quality of life,
reduces absenteeism, and increases economic productivity (Bornmann 2013). Societal impacts
may be intended or unintended, and confined to a particular target area or population, or extend
2
more broadly. Increasingly, research carried out in one location may have far reaching effects
beyond those in an intended area be they governments, industries, clinicians, or individual
citizens. Finally, societal impacts often require years or decades to become apparent. As a result,
drawing causal links between a particular research project or activity and a definable societal
impact is often extremely difficult.
For some disciplines, these linkages are more readily apparent, such as between research
in engineering and economic impacts (Upton et al. 2014). For other disciplines, such as applied
research efforts related to public policymaking, social sciences, or applied public health, the links
between research and societal impact are more difficult to describe, and influenced by a plethora
of uncontrollable factors (Greenhalgh et al. 2016). In these, and other similar fields, evidence
from research is often used “conceptually (for general enlightenment) or symbolically (to justify
a chosen course of action), rather than instrumentally (feeding directly into a particular policy
decision)” (Greenhalgh et al. 2016). A key contributing factor is the complexity of the decision
making process itself, which exerts powerful effects on how research evidence from social
science can be used. As a consequence, social science research is often used to bring attention to
and highlight the complexity of a situation or problem, for which multiple responses may exist
(Greenhalgh et al. 2016).
Given these challenges, there are some who question the usefulness of assessing the
societal impact of research at all, and suggest the potential for such assessments to negatively
influence the type of research that is undertaken (Penfield et al. 2014). For example, concerns
exist that assessing research impact may encourage research on topics and questions for which
research impact may be more readily identified, for which economic impacts (and products) may
be more easily generated, and which fit the interests and priorities of powerful donors (Johnston
3
1995; Penfield et al. 2014; Stuckler et al. 2011). This research may come at the expense of more
exploratory and creative research, and/or that which is not easily translated into quantified
societal impacts. Recent studies suggest that while researchers and the public both place a high
value on research with societal impacts (Mulligan and Conteh 2016; Pollitt et al. 2016), concern
exists particularly among the public that research should not be conducted solely for the pursuit
of economic gains (Miller et al. 2013).
Therefore, measuring the societal impact of research, without compromising the creative
pursuit of new knowledge, requires nuanced measurement approaches that capture a range of
societal impacts, alongside those measures of new knowledge generation. Existing research
impact frameworks offer insights into some of the different ways these societal impacts may be
understood and measured.
Research impact frameworks
Recent reviews have synthesized insights into the growing number of available research
impact frameworks (Banzi et al. 2011; Boaz et al. 2009; Buykx et al. 2012; Greenhalgh et al.
2016; Milat et al. 2015; Penfield et al. 2014). Among the most commonly cited frameworks are
six examined in detail by the RAND corporation: the Canadian Academy of Health Sciences
(CAHS); Excellence in Research for Australia (ERA); National Institute of Health Research
(NIHR) Dashboard; Productive Interactions framework; Research Excellence Framework (REF);
and STAR METRICS (Guthrie et al. 2013). Many of these approaches have been informed by
Buxton and Hanney’s ‘Payback’ framework, originally developed in the UK to examine returns
or ‘paybacks’ from investment in research (Buxton and Hanney 1996).
4
The Payback framework contains five measurement dimensions: (1) knowledge
production; (2) benefits to future research and research use (research targeting, capacity building
and absorption); (3) political and administrative benefits (informing policy and product
development); (4) health sector benefits; and (5) broader economic benefits (Buxton and Hanney
1996; Hanney et al. 2000; Hanney 2005). These dimensions are situated in an input-output model
of how each can be best assessed. This model contains multiple stages (e.g. research needs
assessment, primary research outputs, secondary research outputs etc.), with interfaces between
the research system and the reservoir or stock of knowledge, and the broader political,
professional and industrial environment or society in which research is conducted. As noted by
Hanney, this emphasizes the need for research that meets the “…needs of potential users and
engages the interest of leading researchers, and is then fed back into the wider environment in a
way that increases the chances of the research being utilized”(p.11)(Hanney 2005).
The Payback framework and others have now been widely applied in a range of contexts.
From this work, most consistency appears to relate to measures of knowledge production,
particularly those metrics focused on research publications and funding. For publications,
common measures include the number of publications, quality of publication and citation data,
while funding specific measures relate to the number of applications made to funding agencies or
the total value of funding support secured from various sources (research councils as well as
industry) (Australian Research Council 2016; Higher Education Funding Council for England
2014; Panel on Return on Investment in Health Research 2009).
Measuring the societal impact of research, including impacts on policy and product
development, capacity development, and decision making is comparatively more difficult.(Boaz
et al. 2009; Greenhalgh et al. 2016; Penfield et al. 2014). As noted by Buxton and Hanney, it is
5
important to be able to demonstrate that research has improved the quality and depth of an
information base, as well as influenced the decisions made by those responsible for policy and
practice and their capacity to do so (Buxton and Hanney 1996). Capacity development is
considered to relate to personnel, the acquisition of funding, and investment in research
infrastructure, and may include improving the skills and competencies of staff, creating larger
and more comprehensive datasets, enhancing centre reputation, and fostering cross fertilization
of ideas (Panel on Return on Investment in Health Research 2009). Decision making relates to
decisions made by those in the broad fields of health, research, the health product industry, and
by the general public. This may be evidenced in practitioner behavior, clinical management
guidelines, how resources are allocated, regulatory decisions, media coverage or research and
development agendas (Panel on Return on Investment in Health Research 2009).
Applications of the Payback framework have demonstrated multiple ways that the
societal impacts of research may be captured, including as they relate to the work of specific
research centres (Graham et al. 2012; Hanney et al. 2000; Wooding et al. 2014). For example,
Hanney et al. describe the impact of two research centres; one focused on substance misuse and
the other on community and primary care (Hanney et al. 2000). Results from applying the
Payback model to the work of these centres suggested both had demonstrated impacts on
capacity development and decision making. In part, these impacts are reflected by use of a
centre’s research in needs assessments and project specifications, as well as evidence that those
in policy make deliberate efforts to engage in centre activity (Hanney et al. 2000). These types
of impacts appear to be particularly important for clinical or applied research centres, including
those with a focus on chronic disease prevention (Wooding et al. 2014).
6
Applied Prevention Research Centres (APRC)
In recent years, many high income countries such as Canada, the USA, Australia and the
UK, have established a variety of applied prevention research centres (APRC) to tackle the rising
burden of chronic conditions through prevention research (e.g. the Healthy Populations Institute
(Canada), Prevention Research Centre at St. Louis (USA), the Australian Prevention Partnership
Centre (Australia), Development and Evaluation of Complex Interventions for Public Health
Improvement (UK)). While there is no single organizational model applicable to all APRCs,
many are characterised by the explicit inclusion and engagement of research, policy and practice
perspectives; foci on scientific and practical contributions; and capacity development among
research scholars and policy/practice partners. APRCs are often affiliated with universities, and
may or may not be housed within traditional university structures. Activities are understandably
diverse within APRCs and involve gathering and interpreting surveillance data; developing,
testing and evaluating preventive interventions; mobilizing knowledge from research and
practice; and developing and delivering training programs (Greenlund and Giles 2012). As
investments in such centres continues to grow, being able to describe and document their societal
impacts – beyond new knowledge – is becoming increasingly important. At the same time, the
challenges of measuring the impact of centres is being acknowledged, including the range of
indirect outcomes that centres influence, the difficulty in linking specific pieces of centre
research with societal impacts, and the often lengthy time required for societal impacts to
develop (Scott et al. 2011). The result is a need and desire to understand and improve the societal
impact of APRCs, but limited methods and tools for doing so in the specific context of APRCs.
This study represents an early and exploratory response to this problem and builds on
existing research impact frameworks such as Payback and CAHS. It seeks to contextualize these
7
frameworks to the specific circumstances of APRCs, and in doing so, provide new insights into
those indicators that may be important and feasible for assessing the societal impacts of these
centres. Specifically, this study aims to identify:
1. Those indicators considered by a diverse set of stakeholders to be most important for
capturing the practical impacts of applied prevention research centres; and,
2. Opportunities for adaptation and further development of measures for these most
important indicators.
In doing so, this study tailors available research impact frameworks to the particular
research domain of applied prevention research, and explicitly engages diverse and relevant
perspectives – i.e., research funders, scientific leaders of research centres, knowledge users of
APRCs. Findings from this work are intended to provide inputs for catalyzing new conversations
with members of participating communities, and discerning promising directions for evaluating
practical impacts of APRCs.
METHODS
This study adopted a modified Delphi approach (Day and Bobeva 2005; Goodman 1987;
Hsu and Sandford 2007). The Delphi technique is commonly used for gaining group consensus
on a given topic or theme, and typically involves multiple rounds of a questionnaire delivered
either online or by post (Day and Bobeva 2005; Goodman 1987; Hsu and Sandford 2007). A
traditional Delphi commences with exploratory questions that are used to refine subsequent
rounds. In this instance, a modified Delphi technique was used based on a review of indicators in
available research evaluation frameworks. The design of this study was guided by an Advisory
Group and involved three rounds.
8
Advisory Group
The Advisory Group (n=12) was convened at the commencement of the study. The
Advisory Group included international experts from Canada, Australia, the UK, the USA, and
the Netherlands with expertise in chronic disease prevention research, prevention policy and
practice, research funding, research governance, and assessing research impact. The purpose of
the Advisory Group was to assist in defining the characteristics of APRCs, identifying centres
who met these criteria and assist in recruiting members of APRCs to participate in the study.
The group also helped to identify evaluation frameworks to inform the study. The group
communicated via email with small group teleconferences conducted as needed.
Delphi panelists
An ‘expert panel’ is critical for the Delphi method (Keeney et al. 2001). Given the focus
of this study on APRCs, a two-step process was used to identify expert participants: (1)
identifying APRCs of relevance to the study; and (2) identifying expert panelists associated with
those centres.
Identifying applied prevention research centres (APRC)
Through a combination of targeted online searches and consultation with the Advisory
Group, 36 APRCs were identified internationally that met the definition outlined in Figure 1.
While the search strategy did not specifically use “chronic disease” as a key word, those centres
that did not include a focus on chronic diseases were excluded. Using this definition, the
websites of identified research centres were reviewed to determine if the information provided
within the mission, vision, objectives, services and or additional material such as the annual
report could be considered to fall within the proposed definition of APRCs. The website and
9
document review were completed by one team member and discussed with two additional team
members to determine a final list of APRCs.
Identifying expert panelists
The leaders of identified centres were found through a review of available websites and
associated documentation. Centre leaders were considered those holding positions of directors,
executive directors, CEOs or another comparable title. The first round of the Delphi (see below)
was sent to these centre leaders, who were asked during that round to provide the names and
business email addresses for their core funders and up to two policy/practice ‘knowledge users’
with whom they collaborate. This combination of centre leaders, core funding agencies, and
knowledge users comprised the expert panel in this Delphi.
Expert panelists were recruited through an initial study invitation sent via email that
introduced the study and provided participants with information about what participation
required. After receiving confirmation of their participation, participants were sent an email with
a link to the round 1 questionnaire. Two reminders were sent for each round of the
questionnaire. The round 1 questionnaire was open for 3 weeks. Only panel members who
returned the round 1 questionnaire were sent the round 2 questionnaire, which was administered
1.5 months after round 1 using the same approach as the round 1 questionnaire. The round 3
questionnaire was administered using the same approach as rounds 1 and 2 and was administered
2.5 months after round 2.
The study was reviewed and received ethics clearance through a University of Waterloo
Research Ethics Committee (ORE#20421).
Conducting the Delphi process
10
Round 1
The initial list of indicators for the round 1 questionnaire was informed by a review of
research impact frameworks. Indicators of impact were compiled from the six frameworks
included in the RAND report, beginning with the CAHS framework as this was considered by
the Advisory Group as most closely aligned with the field of research (i.e. chronic disease
prevention). As this study focused on the practical / societal contributions from APRCs, the
initial set of indicators focused on the ‘capacity development’ (CD) and ‘decision making’ (DM)
categories of CAHS. Therefore, the survey did not include indicators on the scientific
contributions of APRCs to new knowledge (e.g. peer-reviewed publications). Once the relevant
indicators from the CAHS had been identified, the REF, ERA, STAR METRICS, Productive
Interactions, and the NIHR Dashboard were reviewed for additional indicators considered by the
Advisory Group as relevant for inclusion. Only those indicators that were applicable at an
institutional level (i.e. the centre level) were included.
In round 1, participants were emailed a link to the survey containing a list of 22
indicators subdivided into CD (9 indicators) and DM (13 indicators). Participants were asked to
rate each indicator on its importance using a nine point Likert scale with anchors of 1 = ‘not at
all important’ and 9 = ‘extremely important’. A comment box was available for each indicator to
allow participants to suggest how they would refine the indicator to make it most useful to
evaluations of APRCs. Participants were asked to complete the ratings from their own
perspective as a centre leader, funder or knowledge user collaborator. Following the rating of
indicators, an open ended question asked participants to provide any additional indicators
specific to the evaluation of APRCs that they considered to be important but which were not
included in the list of 22 indicators.
11
Round 2
Wording refinements suggested by participants to original indicators in round 1 were
incorporated into ‘refined’ indicators (meaning that one original indicator could have been
developed into multiple ‘refined’ indicators with links to an original indicator). Completely new
ideas not captured in any original indicator were developed into new indicators, using the words
of the participant(s) where possible. The round 2 questionnaire provided participants with the
mean importance ratings for each of the original indicators and requested participants provide
importance ratings for refined indicators (n = 49, 27 CD and 22 DM) and newly developed
indicators (n = 15, 12 CD and 3 DM). An open-ended question was included for each set of
indicators to provide comments or suggestions related to the refined and new indicators.
Round 3
In contrast to rounds 1 and 2, round 3 focused on feasibility ratings for those indicators
deemed by participants to be most important. Based on rounds 1 and 2 results, the mean rating of
importance for all indicators (original, refined and newly nominated) was calculated. All
indicators falling above this mean were identified and reviewed by three of the authors (CW, LS,
BR) to determine a set of indicators that each represented a unique idea. Full agreement was
required among these authors, with disagreements resolved through open discussion. Where two
indicators addressed a similar concept as judged by the three authors, the indicator with the
highest mean rating was retained. These procedures generated a list of 39 unique indicators rated
as above average in importance by participants (26 CD and 13 DM indicators). The authorship
team considered the feasibility of collecting indicators to be an important marker of their
usefulness in practice. As such, round 3 asked panelists to rate the feasibility of measuring these
indicators using a nine point Likert scale with anchors of 1 = ‘not at all feasible’ and 9 =
12
‘extremely feasible’. Open ended questions for each set of indicators (CD and DM indicators)
asked participants to identify promising methods and/or data sources (both existing and
potential) and potential challenges to using these indicators for evaluating APRCs.
In addition to ratings of importance and feasibility, round 3 also provided panelists with
opportunity to provide an overall comment on methodological challenges associated with the
indicators. This included opportunity to describe important considerations, potentially useful
sources of evaluative data, and approaches for collecting new data of relevance to the evaluation
of APRCs. Qualitative data gathered from participants were collated into a brief table and
analyzed thematically.
Analysis
Of the 39 unique indicators with above average ratings of importance, we identified two
subsets: (1) those indicators with the 5 highest importance ratings and with feasibility ratings
also falling above the mean; and (2) those indicators with the 5 highest importance ratings but
with feasibility ratings falling below the mean. Indicators in subset 1 represent those which
panelists believe are important to measure for APRCs and which are considered feasible to
measure with current methods, while those indicators in the second subset represent important
indicators for APRCs, but for which current measurement methods may not be suitable. Given
this study explicitly engaged a small number of panelists with expert knowledge of APRCs (i.e.
centre leaders, funders and knowledge users), comparisons between panelists was not possible.
RESULTS
As noted, 36 APRCs were identified and invited to participate in this study, of which 22
agreed to participate. Of the 36 panel members (22 centre leaders, 6 funders and 8 knowledge
users) who agreed to participate in the study, 27 individuals completed round 1 (75%), 23
13
completed round 2 (85% - based on round 1 sample, 13 centre leaders; 4 funders; 6 knowledge
users) and 20 completed round 3 (87%) (11 centre leaders; 4 funders; 5 knowledge users).
More than half of participating APRCs had less than 25 full or part time employees
(63%), and an approximate annual operating budget of $7.3M USD (range $750,000 - $19.7M
USD). Fifty-six percent of participating APRCs had been in operation for more than 10 years.
Panelist from funding organizations represented government and charitable sectors that provided
funding support for infrastructure, capacity development, project and program grants, and
contracts. All participating funding organizations had been in operation for more than 10 years.
Knowledge users brought a variety of perspectives, most prominently advocacy, program
implementation and evaluation (5 of 7 knowledge user panelists). Other areas of focus for
knowledge users included health services, planning and policy development (4 of 7 knowledge
user panelists). Six of the knowledge user panelists reported that their organization had been in
operation for more than 10 years.
A total of 22 CD and DM indicators were selected from research impact evaluation
frameworks and included in the round 1 questionnaire (Table 1). CD indicators (n=9) broadly
related to issues of student training and employment, centre staffing structure and professional
development, centre and project funding, and collaborations with external partners. DM
indicators (n=13) covered domains relevant to the use of research in public policy and programs,
citation of research in guidelines and other policy documents, and reported use of research in and
outside health domains. In round 1, panelists proposed refinements to 9 of the original CD
indicators and 13 of the original DM indicators in order to increase their specificity for APRCs.
This resulted in 27 refined CD indicators (Table 2a) and 22 refined DM indicators (Table 2b)
that had origins in the originally circulated indicators. In addition, panelists proposed a further 12
14
new CD indicators and 3 new DM indicators. These newly identified CD indicators related to
concepts of co-production with research, policy and practice perspectives, field building,
reputational indicators, and specific indicators for knowledge exchange activities. Newly
developed DM indicators focused on the engagement of APRC staff in formal decision making
committees, specific details on the organizations influenced by APRC activities, and sustained
impact of centre research over time.
Round 1 generated 64 new or refined indicators in CD and DM categories. Using the
specified cut-off values, 25 indicators were discarded from round 2, leaving 39 indicators with
above average importance ratings that were then rated by panelists for their feasibility in round 3
(Tables 3a and 3b).
Examination of round 3 results revealed 8 indicators (5 CD and 3 DM indicators) with
ratings for importance and feasibility above the mean (Table 4). These indicators related to
aspects of centre funding, the responsiveness of centre projects to user needs, knowledge
exchange, citations in policy documents and centre staff contributing to decision making bodies.
Round 3 analyses also identified 7 indicators with importance ratings above the mean, but
feasibility ratings below the mean (5 CD and 2 DM indicators) (Table 4). As noted in Table 4,
these indicators relate broadly to concepts of centre reputation, influence of centre activities on
knowledge users, and contributions of the centre to scientific fields and policy development.
Qualitative data highlighted a number of challenges in advancing the evaluation of
APRCs. Panelists highlighted the lack of funding available for supporting evaluative activities,
including for contracting external and independent evaluations (despite their potential value to
funding agencies). Panelists also referred to the ambiguity present in many of the indicators
themselves, highlighted by one panelist:
15
“There are many definitional challenges, such as what counts as applied prevention
research; what is relevant knowledge and skills for applied prevention research; what are types
of 'influence' and 'use' of centre products/activities; what defines "co-production"?”
These challenges were seen as particularly problematic for those indicators requiring
large amounts of complex qualitative data. As highlighted by the following quote, panelists
noted the challenges in gathering relevant evaluative data from collaborators and potential
beneficiaries:
“It requires significant resources and skills to do well, as an arm's length skilled
interviewer must be paid to collect qualitative data of this sort from a wide range of such
stakeholders, many of whom are reluctant to be interviewed or say much when they are.”
Panelists also suggested that the users of knowledge generated by APRCs are not always
easily identified and are not included in an easily accessed database or repository. Even where
such individuals were identifiable, panelists raised concerns about overburdening collaborators
and partners in collecting evaluative data (particularly qualitative data), as well as a potential
lack of interest among policy and practice partners in participating in evaluation activities (as
noted by the quote above). In addition, some panelists expressed potential challenges in
gathering insightful evaluative data from partner organizations experiencing high rates of staff
turnover (i.e. limited organizational memory).
In addition to the challenges of evaluating APRCs, panelists also identified a number of
ways in which high quality data may be gathered as part of APRC evaluation efforts. Panelists
advocated for a range of data sources to be used, suggesting a ‘triangulation’ of methods and
approaches. These data sources included routine national data repositories containing aggregate
data from defined geographic areas, as well as readily available data in centre blogs, social media
16
data, committee minutes and ministerial press releases. While making use of available data was a
key consideration suggested by panelists, a number of data collection methods were suggested
for gathering new evaluative data. These included interviews with relevant individuals and
teams, surveys of partner organizations, and social network analyses (Knoke and Yang 2008).
An overall approach, such as offered by Contribution Analysis (Mayne 2012), was considered to
be one potentially useful strategy for gathering and organizing diverse sources of data to examine
centre impact. An important consideration is the alignment of these methods to the domains of
preferred CD and DM indicators.
DISCUSSION
This study has identified a number of existing indicators, refined indicators and new
indicators that may be useful for evaluating the impact of APRCs. Many indicators that rated
highly on importance represent refined versions of similar indicators in existing frameworks. For
example, while the originally circulated indicators included two focused on the skills and
experience of centre staff, the refined indicators specified the skills of centre staff to the specific
needs of prevention research, the availability of career paths in prevention centres, and the
professional development opportunities provided to centre staff that are relevant to prevention.
Other originally circulated indicators that did not rate highly on importance (e.g. those related to
funding, student training and partnerships) were refined into highly rating indicators more
closely aligned to the contexts of prevention (e.g. stability of funding for indirect costs, students
employed in relevant fields following graduation, the quality of partnerships and not just their
quantity).
17
These results share some similarities with measures contained in existing frameworks,
and highlight how common concepts related to capacity, policy impact, and health status may be
tailored to the context of APRCs. For example, the Payback framework’s dimension on research
targeting, capacity development and absorption appears to relate to this study’s indicators on the
influence of the centre’s training on public and population health practitioners, and how the
centre’s work influences more junior researchers. Payback dimensions focused on informing
policy and product development are expressed in indicators related to the contributions of the
centre’s work to policy development, implementation and evaluation, and the sustained impact of
the centre’s work on decision making that affects the public’s health. Similarly, the Payback
dimension on knowledge production appears to be related to indicators from this study focused
on the contributions of APRCs to the field of prevention research.
Applications of the Payback framework, and associated frameworks such as CAHS, have
demonstrated how these tools may be adapted for use in specific contexts, as well as some of the
key activities required to achieve impact. For example, the CAHS framework (based on the
Payback framework) was adapted by Graham et al. for use in evaluating the development and
implementation of Alberta Innovates-Health Solutions: a Canadian-based, publicly funded
provincial health research and innovation organization (Graham et al. 2012). As noted by the
authors, through both retrospective and prospective application of the CAHS, additional concepts
were added, including domains for reach, training and mentorship, as well as specific indictors
related to budget variance, partnership expenditures, and the number of programs and services
delivered (Graham et al. 2012). As per the adaptations made by Graham et al., results from the
present study highlight specific adaptations that may be useful in the context of APRCs,
particularly around indicators for assessing knowledge mobilization such as those focused on
18
knowledge exchange activities, co-production with policy and practice partners, invitations to
join policy and practice decision making events, and formal and informal relationships with
those in policy and practice.
Consistent with existing research impact frameworks, these indicators highlight the
crucial role of sustained engagement, partnerships and co-production for achieving research
impact. Similar results are suggested by empirical applications of the Payback framework,
including in evaluating the impact of Heartstart Scotland: a national program attempting to
introduce automatic defibrillators into all Scotland’s ambulances (Wooding et al. 2014). In that
analysis, an explicit and sustained effort to build relationships that linked and exchanged the
perspectives of health researchers and potential users was identified as crucial in achieving
impact (Wooding et al. 2014). These concepts resonate with the present findings, and are in
keeping with the theoretical foundations underlying the core functions of APRCs. These
functions are grounded in concepts from traditions such as engaged scholarship, integrated
knowledge-to-action, and partnership-based research (Davies et al. 2008; Greenhalgh and
Wieringa 2011; Moser et al. 2016; Van de Ven 2007). While unique in their own right, these
traditions each highlight the importance of engagement with those that benefit from the research
generated by APRCs. While this is in part reflected by output measures such as the citation of
APRC research in guidelines, policy documents and by those working in policy and practice
settings, results here reinforce other research impact frameworks, and suggest potential value in
also measuring the processes of engagement, and the quality of relationships that exist between
researchers and partners. Cultivating such relationships is a critical role for APRCs and a
significant investment of time and resources to enable use of research knowledge.
19
The distinction between process, outcome and output measures is important for debates
around measuring the impact of APRCs, particularly as there is variability in how these terms are
used: for example, what some consider as process markers may be classified by others as
proximal outcomes. While greater consistency exists related to what constitutes longer-term
outcomes in health (e.g. the incidence and prevalence of a disease or risk factor in a population),
measurement of these outcomes is challenging for two reasons: the time lag required to generate
a detectable change in the health of populations, and the attribution of any change to the work of
an APRC or prevention researcher (Pollack 2011). Process measures (or proximal outcome
measures) have the advantage of measuring the more immediate and often incremental work of
APRCs activities, and are critical for capturing the ongoing and sustained nature of preventive
research that influences those in positions to create positive change (Samuel and Derrick 2015).
Such markers also recognize the indirect nature of many policy contributions, the tendency for
drivers of policy change to often go un-cited, and the potential for high quality research evidence
to be dismissed by powerful political forces (Upton et al. 2014).
Process markers identified in the present study are consistent with the measurement
domains suggested by the Productive Interactions framework originating from the Netherlands
(Spaapen and Van Drooge 2011). As noted, this framework focuses on the role of productive
interactions between researchers and policy/practice partners. As per Spaapen et al., productive
interactions are considered as those that occur between researchers and society, and which “lead
to efforts by stakeholders to somehow use or apply research results or practical information or
experiences” (p 212) (Spaapen and Van Drooge 2011). The social impacts that result from this
knowledge are changes in behaviour that may relate to human well-being and/or the relationships
that exist between people or organizations (Spaapen and Van Drooge 2011).
20
Unlike other impact frameworks, the Productive Interactions approach is tailored to the
needs and contexts of specific groups and teams, limiting its utility for making comparisons, and
hence, it’s capacity to inform funding allocation decisions. As such, Productive Interactions has
been noted for its primary role in fostering learning and improvement within teams over time
(Guthrie et al. 2013).
The utility of the Productive Interactions framework for learning and improvement,
highlights the different uses that research evaluations might have, including for accountability as
noted above, as well as for informing advocacy and analysis/understanding of research activities
(Guthrie et al. 2013). As noted by Guthrie et al., different approaches to evaluation tend to lend
themselves to different purposes. Quantitative approaches tend to be most useful for gathering
longitudinal data and making comparisons across centres and over time; formative approaches
tend to focus on learning and improvement with a flexible and comprehensive approach to
evaluation but with limited utility for drawing comparisons; approaches with a high burden on
those agencies conducting the evaluation are used infrequently and typically capture large
amounts of qualitative data; while those requiring low levels of expertise in their deployment
often have a low burden on participants, and are therefore more commonly employed (Guthrie et
al. 2013).
Therefore, the ease with which data may be collected for a specific suite of indicators
influences who will use those indicators, for what purposes, and how frequently they will collect
data on one or more indicators. Results from the present study have identified a subset of
indicators that panelists rated as both important and feasible for collection. Many of these
indicators are quantitative measures, such as the number of centre projects driven by expressed
policy/practice needs of engaged organizations; the number of partners/collaborators (by sector)
21
and products and contributions of collaboration; the number and type of knowledge exchange
activities undertaken by the centre; or evidence of the centre’s contributions to supporting
decision making processes and groups (e.g., participation of centre staff on steering groups,
Ministerial Working groups, government committees etc.). Capturing data on these and other
similar indicators helps to quantify the performance of APRCs and is comparatively simple to
gather through regular tracking procedures within many APRCs. As such, these data may be
built into ongoing monitoring efforts, allowing performance to be tracked over time as well as
between centres, and may therefore be of appeal to those evaluation efforts with few resources or
limited specialist skills available. A common set of quantitative metrics may also be of appeal to
centre funders with interests in conducting comparative analyses across centres. Yet these
quantitative methods are unlikely to capture the full range of societal impacts an APRC might
have or desire to have.
Alongside these important and feasible quantitative measures, this study also identified a
sub-set of indicators with high importance ratings, yet low feasibility ratings. These measures
include a wide range of foci, including aspects of centre reputation, temporal changes over time
in how people think, work and behave as a result of centre activities, the contribution of a centre
to building the field of prevention research, and sustained impact of the centre’s work on policy,
programs and practice. For many APRCs, these are key domains of activity, yet as noted by
panelists in this study, are regarded by many as being difficult to capture. Gaining useful insights
into these aspects of centre performance likely requires multi-modal methods, including desk
analysis, panel assessments, interviews and case studies, which include the perspectives of those
within and external to APRCs (Cohen et al. 2015; Milat et al. 2015). Narrative case studies, such
as those gathered by the 2014 Research Excellence Framework, have been noted to succeed in
22
“capturing the complex links between research and impact” (Higher Education Funding Council
for England 2014). Through the REF, 1621 qualitative case studies from the health and
biomedical research fields were submitted for evaluation by expert panels, with 91% considered
by panelists as being outstanding or very considerable in terms of the significance of their
impact. The most successful case studies were considered as those involving a compelling
narrative that linked research to impact; verifiable evidence of the link; and how the impact has
spread from immediate to distant beneficiaries (Higher Education Funding Council for England
2014). Critically, these details cannot be captured through sole use of quantitative metrics
(Higher Education Funding Council for England 2014). Yet gathering these and similar data
requires a high level of expertise, time and financial support: resources not always available to
many evaluation practices.
Given the importance of these indicators, advancing the evaluation of APRCs may be
well served through investing in methods development work that captures the contribution of
such centres to societal and economic goals. A number of contribution-based approaches have
been described in the literature, including contribution mapping (Kok and Schuit 2012),
contribution analysis (Mayne 2012) and an extension of the latter, the Research Contribution
Framework (Morton 2015) A companion article in this journal issue (Riley et al.), reports on the
experience of an APRC in Canada – the Propel Centre for Population Health Impact – in
applying contribution analysis methods to three case studies as part of developmental work for a
centre-wide evaluation intended to serve learning, improvement, and accountability purposes
(Riley et al. 2017). The study adapts and extends Mayne’s six steps to contribution analysis in
an effort to increase their relevance to evaluating research impacts on public health policy, with
potential application more broadly to evaluating societal impacts of APRCs (Mayne 2012).
23
Exploring this and other approaches to describing contribution rather than attribution, is an
important area for future research to better understand the societal impacts of research centres.
Strengths and limitations
This study engaged a broad group of panelists, from multiple jurisdictions, and with
varying perspectives on APRCs (i.e. prevention researchers, funders and knowledge users
working with APRCs). The response rate between rounds remained high throughout the study.
While the study did not seek to identify a final suite of indicators for assessing APRCs, it did
succeed in describing subsets of indicators considered by panelists as being more important
and/or more feasible than others.
While efforts were made to engage diverse perspectives, there were too few panelists to
meaningfully explore any between group differences. Participation was particularly low for those
in funding agencies or those collaborating with APRCs as knowledge users. Future studies may
seek to explore these perspectives through alternative qualitative methods, such as in-person
focus groups or key-informant interviews. Such approaches may provide a more engaging forum
for eliciting the opinions of those funding and collaborating with APRCs.
The indicators identified in this study may be used to provide particular insights into the
impact of APRCs. They cannot stand alone however, especially data on individual indicators. To
more fully understand impact, the indicators identified in this study need to be placed in context
– individually and as a set – which includes the interplay of performance indicators, and the
multi-level influences on APRCs such as broader institutional climates and cultures, local
community settings, and broader socio-economic conditions. In addition, this study has not
considered over what time periods change in the identified indicators may be expected.
Therefore, some of the indicators identified in this study may be suitable for annual monitoring,
24
while others may require longer time frames for change to take place. As noted above, a next
step may be to more deeply understand how those leading APRCs, funding APRCs and
collaborating with APRCs interpret and value different measures, for different purposes, and
over different time periods.
Conclusions
Applied prevention research centres are important parts of societal efforts to promote
healthy living and prevent chronic disease. Documenting, describing and comparing the diverse
impacts such centres can have on advancing new knowledge, improving decision making, and
developing capacity, is therefore important. The challenge of measuring these impacts –
particularly in relation to the effects of APRCs on decision making and capacity development –
are shared with other fields of research, such as those focused on public policy, social sciences,
or applied public health. While previous frameworks such as CAHS and Payback are potentially
useful for understanding the impacts of APRCs, no specific set of indicators exists for measuring
the impact of APRCs. As such, this study sought to gather expert perspectives on promising
measurement domains focused on the practical impacts of APRCs. Findings from this work are
consistent with existing research impact frameworks, and highlight measurement domains that
are both important and feasible, including the number of APRC projects driven by explicit policy
needs, the number and quality of knowledge exchange activities, and citations of APRC research
in public policy documents. Findings also suggest important measurement domains that require
future methods refinement, including measures of centre reputation, evidence of contributions to
the field of prevention research, and the influence of the centre’s work over time on the
knowledge, skills and commitment of policy and practice partners. Methods such as Contribution
Analysis hold promise for advancing the measurement of APRCs and warrant further
25
investigation in future empirical studies. Improving our understanding and measurement of the
practical impacts of APRCs is important for both learning and accountability: findings from this
study provide new insights into what such measures may include, and where future effort may be
usefully directed.
26
REFERENCES
Australian Research Council (2016), 'Excellence in Research for Australia',
<http://www.arc.gov.au/excellence-research-australia>, accessed.
Banzi, R., et al. (2011), 'Conceptual frameworks and empirical approaches used to assess the
impact of health research: an overview of reviews', Health Res Policy Syst, 9, 26.
Boaz, A., Fitzpatrick, S., and Shaw, B. (2009), 'Assessing the impact of research on policy: A
literature review', Science & Public Policy, 36 (4), 255-70.
Bornmann, L. (2013), 'What is societal impact of research and how can it be assessed? A
literature survey', Journal of the American Society for Information Science and
Technology, 64 (2), 217-33.
--- (2016), 'Scientific Revolution in Scientometrics: the Broadening of Impact from Citation to
Societal', in C. R. Sugimoto (ed.), Theories of Informetrics and Scholarly
Communication (Berlin: de Gruyter), 347-59.
Buxton, M. and Hanney, S. (1996), 'How can payback from health services research be
assessed?', J Health Serv Res Policy, 1 (1), 35-43.
Buykx, P., et al. (2012), ''Making evidence count': a framework to monitor the impact of health
services research', Aust J Rural Health, 20 (2), 51-8.
Cohen, G., et al. (2015), 'Does health intervention research have real world policy and practice
impacts: testing a new impact assessment tool', Health Res Policy Syst, 13, 3.
Davies, H., Nutley, S., and Walter, I. (2008), 'Why 'knowledge transfer' is misconceived for
applied social research', J Health Serv Res Policy, 13 (3), 188-90.
Day, J. and Bobeva, M. (2005), 'A generic toolkit for the successful management of Delphi
studies', Electron J Bus Res Methodol, 3, 103-16.
27
Goodman, C. M. (1987), 'The Delphi technique: a critique', J Adv Nurs, 12 (6), 729-34.
Graham, K.E.R., et al. (2012), 'Evaluating health research impact: Development and
implementation of the Alberta Innovates - Health solutions impact framework.', Research
Evaluation, 21, 354-67.
Greenhalgh, T. and Wieringa, S. (2011), 'Is it time to drop the 'knowledge translation' metaphor?
A critical literature review', J R Soc Med, 104 (12), 501-9.
Greenhalgh, T., et al. (2016), 'Research impact: a narrative review', BMC Med, 14, 78.
Greenlund, K. J. and Giles, W. H. (2012), 'The Prevention Research Centers program: translating
research into public health practice and impact', Am J Prev Med, 43 (3 Suppl 2), S91-2.
Guthrie, S., et al. (2013), 'Measuring research: A guide to research evaluation frameworks and
tools', (Santa Monica, CA: RAND).
Hanney, S. (2005), 'Developing and aplying a framework for assessing the payback from
medical research', Kiel Institute's International Research Conference on New Technology
and National Health Systems (Kiel).
Hanney, S., Packwood, T., and Buxton, M. (2000), 'Evaluating the benefits from health research
and development centres', Evaluation, 6 (2), 137-60.
Higher Education Funding Council for England (2014), '2014 REF: Assessment framework and
guidance on submissions. Panel A criteria', (London HEFCE).
Hsu, C.C. and Sandford, B.A. (2007), 'The Delphi Technique: Making Sense Of Consensus',
Practical Assessment, Research and Evaluation, 12 (10), 1-8.
Johnston, R. (1995), 'Research impact quantification', Scientometrics, 34 (3), 415-26.
Keeney, S., Hasson, F., and McKenna, H. P. (2001), 'A critical review of the Delphi technique as
a research methodology for nursing', Int J Nurs Stud, 38 (2), 195-200.
28
Knoke, D. and Yang, S. (2008), Social Network Analysis (2 edn.; Thousand Oaks: Sage).
Kok, M. O. and Schuit, A. J. (2012), 'Contribution mapping: a method for mapping the
contribution of research to enhance its impact', Health Res Policy Syst, 10, 21.
Mayne, J. (2012), 'Contribution analysis: Coming of age?', Evaluation, 18 (3), 270-80.
Milat, A. J., Bauman, A. E., and Redman, S. (2015), 'A narrative review of research impact
assessment models and methods', Health Res Policy Syst, 13, 18.
Miller, F. A., et al. (2013), 'Do canadian researchers and the lay public prioritize biomedical
research outcomes equally? A choice experiment', Acad Med, 88 (4), 519-26.
Morton, S. (2015), 'Progressing research impact assessment: A ‘contributions’ approach',
Research Evaluation, 24 (4), 405-19.
Moser, D. , Ream, T. C., and Braxton, J. M. (2016), Scholarship reconsidered: Priorities of the
Professoriate (expanded edition) (San Francisco: Jossey-Bass).
Mulligan, J. A. and Conteh, L. (2016), 'Global priorities for research and the relative importance
of different research outcomes: an international Delphi survey of malaria research
experts', Malar J, 15 (1), 585.
Panel on Return on Investment in Health Research (2009), 'Making an Impact: A Preferred
Framework and Indicators to Measure Returns on Investment in Health Research',
(Ottawa: Canadian Academy of Health Sciences (CAHS)).
Penfield, T., et al. (2014), 'Assessment, evaluations, and definitions of research impact: A
review', Research Evaluation, 23, 21-32.
Pollack, H. A. (2011), 'Prevention and public health', J Health Polit Policy Law, 36 (3), 515-20.
29
Pollitt, A., et al. (2016), 'Understanding the relative valuation of research impact: a best-worst
scaling experiment of the general public and biomedical and health researchers', BMJ
Open, 6 (8), e010916.
Riley, B.L., et al. (2017), 'Adapting and extending contribution analysis methods to evaluate the
impacts of research on three tobacco control policies in Canada', Research Evaluation,
Under Review.
Samuel, G.N. and Derrick, G. E. (2015), 'Societal impact evaluation: Exploring evaluator
perceptions of the characterization of impact under the REF2014', Research Evaluation,
24 (3), 229-41.
Scott, J.E., et al. (2011), 'An evaluation of the Mind Body Interactions and Health Program:
assessing the impact of an NIH program using the Payback Framework', Res Eval 20 (3),
185-92.
Spaapen, J. and Van Drooge, L. (2011), 'Introducing “Productive Interactions” in Social Impact
Assessment', Research Evaluation, 20 (3), 211-18.
Stuckler, D., Basu, S., and McKee, M. (2011), 'Global health philanthropy and institutional
relationships: how should conflicts of interest be addressed?', PLoS Med, 8 (4),
e1001020.
Upton, S., Vallance, P., and Goddard, J. (2014), 'From outcomes to process: evidence for a new
approach to research impact assessment', Research Evaluation, 23 (4), 352-65.
Van de Ven, A.H. (2007), Engaged Scholarship: A Guide for Organizational and Social
Research (Oxford: Oxford University Press).
Wooding, S., et al. (2014), 'Understanding factors associated with the translation of
cardiovascular research: a multinational case study approach', Implement Sci, 9 (1), 47.
30