NCCMT Spotlight Webinar: MetaQAT

Follow us @nccmt Suivez-nous @ccnmo

Funded by the Public Health Agency of Canada | Affiliated with McMaster UniversityProduction of this presentation has been made possible through a financial contribution from the Public Health Agency of Canada. The

views expressed here do not necessarily reflect the views of the Public Health Agency of Canada..

MetaQAT

Presenters:

Laura Rosella, PhD, MHsc

Catherine Bornbaum, PhD

March 21, 2017 1:00 – 2:30 PM ET

Follow us @nccmt Suivez-nous @ccnmo2

Housekeeping

Use Chat to post comments and/or

questions during the webinar

• ‘Send’ questions to All (not

privately to ‘Host’)

Connection issues

• Recommend using a wired

Internet connection (vs.

wireless),

• WebEx 24/7 help line

• 1-866-229-3239

Participant Side

Panel in WebEx

Chat


After Today

The PowerPoint presentation (in English and French)

and English audio recording will be made available.

These resources are available at:

PowerPoint: https://www.slideshare.net/NCCMT/presentations

Audio Recording:

https://www.youtube.com/user/nccmt/videos

https://www.slideshare.net/NCCMT/presentations

https://www.youtube.com/user/nccmt/videos


How many people are watching

today’s session with you?

Poll Question #1

A. Just me

B. 1-3

C. 4-5

D. 6-10

E. >10


Your profession? Put a √ on your answer (or RSVP via email)

/

Epidemiologist Management (director,

supervisor, etc.)

Allied health

professionals (nurse,

dietician, dental

hygenist, etc.)

Librarian Physician / Dentist Other

5


MetaQAT

http://www.nccmt.ca/resources/search/243

Episode 32

6

http://www.nccmt.ca/resources/search/243

NCC

Infectious

DiseasesWinnipeg, MB

NCC

Methods

and ToolsHamilton, ON

NCC Healthy

Public PolicyMontreal, QC

NCC

Determinants

of HealthAntigonish, NS

NCC

Aboriginal

HealthPrince George, BC

NCC

Environmental

HealthVancouver, BC

7

http://www.nccid.ca/en/home

http://www.nccid.ca/en/home

http://www.nccmt.ca/


http://www.healthypublicpolicy.ca/2/Home.htm

http://www.healthypublicpolicy.ca/2/Home.htm

http://www.nccdh.ca/

http://www.nccdh.ca/

http://www.nccah-ccnsa.ca/

http://www.nccah-ccnsa.ca/

http://www.ncceh.ca/

http://www.ncceh.ca/

Registry of Methods and Tools

Online Learning

Opportunities

WorkshopsMultimedia

Public Health+

Networking and

Outreach

NCCMT Products and Services

8


Poll Question #2

How familiar are you with the

method or tool we are discussing

today?

A. I am not familiar with the method or tool

B. I have heard of the method or tool

C. I have used the method or tool


Presenter

Laura Rosella, PhD, MHsc

Canada Research Chair in Population

Health Analytics

Assistant Professor, Dalla Lana School of

Public Health, University

of Toronto

Scientist, Public Health Ontario

Adjunct Scientist, Institute for Clinical

Evaluative Sciences


Presenter


Research Associate, Dalla

Lana School of Public Health,

University of Toronto

Adjunct Research Professor,

Western University

A public health approach to appraising evidence:

Meta-tool for Quality Appraisal (MetaQAT)

Laura Rosella, PhDDalla Lana School of Public HealthPublic Health Ontario

March 21, 2017

Project team

• Laura Rosella, Scientist

• Beata Pach, Manager, Library Services

• Carolyn Bowman, Senior Research Coordinator

• Sarah Morgan, Library Technician

• Tiffany Fitzpatrick, Epidemiologist

• Vivek Goel, Scientist

13

Objectives

1. Rationale and development of the PHO MetaQAT tool

2. Orientation of the tool and how to use it

14

15

Idea/request for review topic

Formulation of specific review question

Literature search

Application of inclusion & exclusion criteria

Quality appraisal of included studies

Data extraction and analysis

Synthesis or conclusions

Will start directly here for review of one study

Some iteration may be required

Can be done at the same timei.e. when documenting appraisal you can

also add desired data extraction fields

“Critical appraisal is the process of carefully and systematically examining research to judge its trustworthiness, and its value and relevance in a particular context.”• Judiciously assessing quality of evidence is challenging

• Requires expertise (preferably methods and content)

• If we are attempting to inform an evidence base that will inform public health decisions – it needs to be a judicious and transparent process

• Transparency is critical – not only will it enable us to better support our recommendations but also challenge our own assumptions to ensure the science is as strong as possible

• The “why” is often as important as the outcome

Quality (critical) appraisal

Source: Burls A. What is critical appraisal? 2nd ed. [Internet]. Newmarket, UK: Hayward Group; 2009 [cited 2015 Mar 2]. Available fromhttp://www.bandolier.org.uk/painres/download/whatis/What_is_critical_appraisal.pdf

16

http://www.bandolier.org.uk/painres/download/whatis/What_is_critical_appraisal.pdf

Appraisal should be about understanding

• Both strengths and weaknesses

• We need to make decisions, important to use evidence to fullest extent possible

• Understand how evidence can be used appropriately

17

Subjective is not a bad word

• Judgments are made based on appraisal principles

• Using the same principles, similar issues will be identified

• No one right answer, that doesn’t mean anything goes!

• Transparency is key: another person should be able to understand why you made your assessment

18

Transparency

• Importance of documentation – if others know what you appraised and why, they can ‘appraise your appraisal’

• Transparency is key: another person should be able to understand why you made your assessment

• Think of it as a way that one can audit your analysis of the evidence – akin to auditing code following a statistical analysis

19

Internal Validity

• Elwood: measure of how easily a difference in an outcome between two groups can be attributed to the effects of an exposure or intervention

• L. Green: “the essence of rigor”

External Validity

• The process of generalizing the findings of the study to the population from which the sample was drawn (or even beyond that population to a more universal statement)

• Without internal validity a study cannot have external validity

• L. Green: “ the essence of relevance”20

Internal and external validity

Internal validity can be influenced by...

• Study design

• Measurement properties of the variables

• Study recruitment/response rates

• Selection pressures

• Sampling strategy

• Setting

• Investigators/funding

• Among others factors…

• Preferred term: RISK OF BIAS (RoB)21

22

23www.riskofbias.info

http://www.riskofbias.info/

24

Risk of Bias says nothing about the intended use of the evidence

• Application of the evidence is important in public health

• Context matters – where the evidence was generated, where it is going to be applied

• Importance of different factors when appraising evidence should be guided by the intended application

25

Contemporary view of “evidence-based”

• Evidence-informed• The idea that evidence (in the scientific sense) informs practice but that

other factors are important

• Weight of evidence versus strength of evidence• EBM traditionally focuses on strength of evidence (i.e. RCT) and

discounts indirect evidence from other sources

• Larry Green: “ If We Want More Evidence Based Practice, We Need More Practice Based Evidence”

26

Evidence Informed Public Health (EIPH)

• “public health endeavor in which there is an informed, explicit, and judicious use of evidence that has been derived from any of a variety of science and social science research and evaluation methods”

• Acknowledges the many factors, beyond simply the evidence, influence decision-making

• “EIPH is a complex, multi-disciplinary process that occurs within dynamic and ever-changing communities and encompasses different sectors of society”

http://www.nccmt.ca/uploads/media/media/0001/01/b331668f85bc6357f262944f0aca38c14c89c5a4.pdf

27


Beware the evidence pyramid

A Hierarchy of Strength of Evidence for Treatment/ Intervention Decisions (Quantitative Research)

29

Relative Strength of Evidence (with 1 being

the strongest)

Type of Evidence

1 • systematic review of randomized controlled trials

2 • systematic review of observational studies addressing population health important outcomes

3 • single randomized trial

4 • single observational study addressing population important outcomes

5 • physiologic and epidemiological study (e.g., study of infection prevalence, prevalence of cardiovascular risk factors)



What’s better? • A large and rigorous prospective cohort study or a small poorly

conducted randomized control trial?

• A systematic review of dozens of well-conducted case control studies or one large randomized control trial?

• Large and rigorous randomized controlled trial done in Japan or a time-series analysis in Calgary? (assuming you live in Toronto)

30

Critical appraisal and the evidence hierarchy

What’s better? • A large and rigorous prospective cohort study or a small poorly

conducted randomized control trial?

• A systematic review of dozens of well-conducted case control studies or one large randomized control trial?

• Large and rigorous randomized controlled trial done in Japan or a time-series analysis in Calgary? (assuming you live in Toronto)

• A well conducted study that’s ‘lower’ on the evidence hierarchy may actually be better...every study needs to be interpreted and assessed in its own right

31

Critical appraisal and the evidence hierarchy

32

Points to remember:1. Overall quality appraisal ≠ internal validity

only2. Risk of bias assessment speaks mainly to

internal validity 3. The study’s quality is only one consideration

in the weight of evidence

Skills and knowledge required for quality appraisal

• Knowledge of topic area

• Knowledge and experience with the specific methods used

• Knowledge of intended application

• Experience with appraisal

• Excellent documentation skills - record why decisions were made

• Note: These traits can be covered by a team

33

TOOLS, TOOLS, EVERYWHERE

34

A systematic review of 121 published critical appraisal tools • 87% of critical appraisal tools were specific to a research

design with most tools having been developed for experimental studies

• 49% of the critical appraisal tools summarized the quality appraisal into a numeric summary score

• Few critical appraisal tools had documented evidence of validity of their items, or reliability of use

• “We found no gold standard critical appraisal tool for any type of study design”

35

Source: Katrak P, Bialocerkowski AE, Massy-Westropp N, Kumar S, Grimmer KA. A systematic review of the content of critical appraisal tools. BMC Med Res Methodol. 2004 [cited 2014 Aug 7];4:22. Available from: http://www.biomedcentral.com/content/pdf/1471-2288-4-22.pdf

http://www.biomedcentral.com/content/pdf/1471-2288-4-22.pdf

Suggested benefits of appraisal tools frameworks

• Checklists that ensure that we don’t overlook important considerations (i.e. Aide-mémoire)• CAN limit what we do look for depending what’s in the tool

• Standardize the approaching to reviewing the evidence• This can be easily challenged because many of the judgements are

subjective – so I'd rather know why you judged something a particular way versus what your judgement was

36

A comparison of toolsVoss and Rehfuess

QUALITY APPRAISAL IN SYSTEMATIC REVIEWS OF PUBLIC HEALTH INTERVENTIONS: AN EMPIRICAL STUDY ON THE IMPACT OF CHOICE OF TOOL ON META-ANALYSIS

Recommendations:

1) Testing of a broader set of QATs on a more up-to-date systematic review of a public health intervention covering a wide range of epidemiological study designs

2) Research into the development of a reliable QAT with broad applicability across study designs.

37

Source: Voss PH, Rehfuess EA. Quality appraisal in systematic reviews of public health interventions: an empirical study on the impact of choice of tool on meta-analysis. J Epidemiol Community Health. 2013;67(1):98-104.

Features of a Gold Standard QATVoss and Rehfuess

1) Instead a set of companion QATs comprising common components and study design-specific components could initially be developed for standard designs and then expanded to address more complex designs.

2) An overall summary score may lead to information loss and oversimplification. Instead, the authors propose a careful quantification of domains (rather than individual questions) in a weighted checklist.

3) Should address both internal and external validity with a clear distinction between these.

4) Much more effort should be invested into the development and testing of appropriate and specific questions and answer categories; a manual can help with their correct interpretation.

5) Subject-specific adaptation of selected questions makes quality appraisal more precise and reliable than a one size fits all QAT.

38

Source: Voss PH, Rehfuess EA. Quality appraisal in systematic reviews of public health interventions: an empirical study on the impact of choice of tool on meta-analysis. J Epidemiol Community Health. 2013;67(1):98-104.

METAQAT PROCESS

39

MetaQAT project stages

40

Stage 1: Gather information

Stage 2: Put information

together

Stage 3a: Draft the meta-tool

Stage 3b: Refine draft meta-tool

Stage 4: Pilot testing

Stage 5: Validation

Stage 1: Gather information

• Searched for existing tools• Databases

• Grey literature

• Guidance and background documents on critical appraisal in public health (NICE, ECDC, NCCMT, Cochrane)

• Websites

• Universities with public health programs

• NCCMT repository of critical appraisal tools

• Consult with experts

• Consult with PHO staff

41

Summary on tools to guide appraisal

• Many tools exist (most not validated)

• Most from evidence-based medicine perspective • Internal validity (or risk of bias) focus

• Many use numeric scales or scores• Not valid, not recommended by Cochrane Collaboration

• Many tools appropriate for specific contexts (topic, project, study design)

• Something more flexible was required

42

What kind of tool?

• One appropriate for our context• Various types of reviews

• Rapid reviews, evidence briefs, systematic reviews, etc.

• Various types of content

• Chronic disease, infectious disease, environmental health, health promotion, injury prevention, etc.

• Strong focus on the applicability of the evidence

• Not just the quality

• Not reinventing the wheel• Build on existing content

43

What tool for PHO?

• Many tools appropriate for specific contexts (topic, project, study design)

• Resources available to find tools• NCCMT tool repository

• University SA library

• PHO needs:• Compatible with all of the types of evidence we use

• Compatible with all of the types of projects we do

• Consistent with current best practice

• No numeric scores; acceptable to field

• Promotes a transparent process

• Outcome of stage 1: not a critical appraisal tool (too limited)

44

Stage 2: Putting the information together

• Group tools by study design• Compare the questions between tools

• Similarities across tools within each design

• Compare these similarities across study designs• Core process is similar across tools

• Coverage of process matches OPHLA guide

• Generic, applies to all evidence types

• Generic framework emerges• All questions can be grouped into four categories

• Still need study specific detail for high level of rigour

• Not a critical appraisal tool, a quality assessment process

45

Stage 3a: Develop draft meta-tool

• Develop framework and select companion tools

• Companion tools – pragmatic approach• Widespread use in public health literature

• Preference to tools with documented validation

• Compromise – reporting tools widely used

• Ease of use

• Revise based on scientific advice

46

Stage 3b: Refine draft meta-tool

• Feedback sessions• Staff selected for feedback sessions

• Sample article assessed, interpretations of questions

• Revisions made for clarity, reorganization

• Revised version presented to third group

• Revisions made

• Tool approved for pilot testing and further development and evaluation

47

Stage 4: Pilot testing

• Introduced to teams for trial use in specific projects• Variety of topics and types of projects

• Training methods developed• Introduction to tool as project reached appraisal stage

• Evaluation• Fit of tool for projects and staff, requirements for full implementation

48

Selected pilot testing outcomes

INTEGRATION OF DIVERSE GROUPS OF EVIDENCE

• Applied to projects with a heterogeneous mix of evidence

• The flexibility of MetaQAT allowed for the integrated assessment of evidence across methods and designs

• Facilitated screening decisions

• Broadened scope of appraisal promoted appropriate consideration of context versus internal validity alone

• Maintains consistent process of appraisal across a group of evidence that includes grey literature and surveillance literature

49

Stage 5: Validation

• Two groups of appraisers• One group used MetaQAT, other group appraised strengths and

weaknesses of several studies

• Written appraisals analysed• Content analysis

• Both groups identified similar issues

• MetaQAT group identified public health relevance

50

Meta-tool

• The “meta” concept is not a single tool/framework (traditional sense), it is a process guide which orients the user to the appropriate use of several appraisal tools, and places them within a larger framework to guide their use

• A tool of tools: Appraisal framework that incorporates existing tools for design-specific risk of bias assessment

• Documentation – records the appraisal process for transparency

51

MetaQAT structure

52

Relevancy

Companion tools

AMSTARCASPTRENDCONSORTMixed MethodsAGREE IINavigation GuidePHO Guide for Grey Literature

Reliability

Validity

Applicability

Appraisal Framework

Rosella, L. Bowman, C., Pach, B., Morgan, S., Fitzpatrick, T., Goel, V. The development and validation of a meta-tool for quality appraisal of public health evidence: Meta Quality Appraisal Tool (MetaQAT). Public Health. 2016;136:57-65.

MetaQAT structure

Domain Idea

Relevancy

• Determine if the item being appraised is sufficiently related to your topic or research question.

• In a structured systematic review, relevancy can also be considered in the inclusion and exclusion criteria.

Reliability

• Evaluate the completeness of reporting; complete reporting is necessary in order to conduct a thorough examination of quality.

• Lack of detail regarding the conduct of the study or report may be indicative of lower quality information.

Validity• The likelihood and magnitude of error or bias in a study. • “Risk of Bias” or “Internal Validity”

Applicability• Consider how the evidence might be applied to public health practice.• Also can consider more broadly what can be learned from the evidence and

how you can apply that knowledge to public health decision making.

53Rosella, L. Bowman, C., Pach, B., Morgan, S., Fitzpatrick, T., Goel, V. The development and validation of a meta-tool for quality appraisal of public health evidence: Meta Quality Appraisal Tool (MetaQAT). Public Health.

The form uses the term “study” to refer to the work being appraised and is meant to include all study types, including research published as grey literature, as syntheses, and any other type of evidence being considered.

MetaQAT appraisal form

54

• Domain

• Main question

• Prompting questions• Guide thinking about

the main question

• Stimulate thought, not limit

• Written answer

• Optional check boxes


• Written answer format

• Written summary of key points

• Tick box options available BUT

• Written summary is the answer to the question (many formats available)

55


• Branch point in process – to design-specific companion tools IF DESIRED• In validity domain, question b) Is the research methodology free from bias?

• Design-specific detail via companion tools IF DESIRED

56

• Can also create context-specific prompts

Summarizing the appraisal of an item

• Textual summary of key strengths and weaknesses by domain and question• This information will provide a good understanding of the quality of the

item

• E.g. which is more helpful? 8/10 or “the study group was small but characteristics were very similar to our local population”

• We are working on guidance to help with summary/synthesis language

• “The use of scales for assessing quality or risk of bias is explicitly discouraged in Cochrane reviews”

57

Source: Higgins JPT, Green S, editors. Cochrane handbook for systematic reviews of interventions. Version 5.1.0 [Internet]. London: Cochrane Collaboration; 2011 [cited 2015 Feb 10]. Section 8.3.3, Quality scales and Cochrane reviews. Available from: http://handbook.cochrane.org

MetaQAT user guide

• The MetaQAT document includes relevant background information and user guidance – please read! (url to follow)

• For more details:

Rosella L, Bowman C, Pach B, Morgan S, Fitzpatrick T, Goel V. The development and validation of a meta-tool for quality appraisal of public health evidence: Meta Quality Appraisal Tool (MetaQAT). Public Health. 2016;136:57-65. Available from: http://www.publichealthjrnl.com/article/S0033-3506(15)00437-0/abstract

58

http://www.publichealthjrnl.com/article/S0033-3506(15)00437-0/abstract

Hard work is hard

• The process of systematically appraising evidence and documenting this process is difficult and can be time consuming but these challenges may exist regardless of the tool/framework

59

60

https://www.publichealthontario.ca/en/ServicesAndTools/Pages/Critical-Appraisal-Tool.aspx

Screen shots to follow


61

62

63

References

• Rosella L, Bowman C, Pach B, Morgan S, Fitzpatrick T, Goel V. The development and validation of a meta-tool for quality appraisal of public health evidence: Meta Quality Appraisal Tool (MetaQAT). Public Health. 2016;136:57-65. Available from: http://www.publichealthjrnl.com/article/S0033-3506(15)00437-0/abstract

• Voss PH, Rehfuess EA. Quality appraisal in systematic reviews of public health interventions: an empirical study on the impact of choice of tool on meta-analysis. J Epidemiol Community Health. 2013;67:98-104.

• Heller RF, Verma A, Gemmell I, Harrison R, Hart J, Edwards R. Critical appraisal for public health: A new checklist. Public Health. 2008;122:92-98.

• Jüni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of clinical trials for meta-analysis. JAMA. 1999;282(11)1054-1060.

• Katrak P, Bialocerkowski AE, Nassy-Westropp N, Kumar VSS, Grimmer KA. A systematic review of the content of critical appraisal tools. BMC Med Res Methodol. 2004 [cited 2013 Dec 16];4(22): . Available from: http://www.biomedcentral.com/1471-2288/4/22

64

http://www.publichealthjrnl.com/article/S0033-3506(15)00437-0/abstract

http://www.biomedcentral.com/1471-2288/4/22

Information management

• Excel version of form is available as template

65

66

67

68

69

NCCMT Spotlight on Methods and Tools:Using the MetaQAT for Quality Appraisal in a

Systematic Review

Catherine Bornbaum, PhDManaging Director, Population Health Analytics Research Program

Research Associate, Dalla Lana School of Public Health, University of Toronto

Adjunct Research Professor, Health & Rehabilitation Sciences, Western University

Background• Failure to optimize the use of research evidence may result in

reduced quality of care1, inefficient use of resources2,3, and poorer health outcomes for individuals and communities4

• Given the challenges with effectively translating knowledge between researchers, practitioners, and decision-makers5, knowledge translation experts advocated for an intermediary person to mitigate challenges in knowledge sharing6,7

• Knowledge brokers work collaboratively with key stakeholders to facilitate the transfer and exchange of evidence and context

• Key roles:• Knowledge manager, linkage agent, capacity builder

70

To understand how knowledge brokers operate in health-related practice, and assess if use of a knowledge broker could facilitate effective knowledge translation initiatives.

Objectives: 1. To improve conceptualization of the knowledge brokering role in

health-related settings; and• Test theory of knowledge brokering in practice (confirmatory

thematic analysis8)

2. To determine whether knowledge brokers contribute to effective knowledge translation in health related settings

• Explore outcome data for evidence of changes in knowledge, skill, policy and practice9

• Critically appraise evidence using the Meta Quality Appraisal Tool (MetaQAT)

71

Goal

Inclusion and Exclusion Criteria

• Inclusion Criteria• Reports on an actual application of knowledge brokering in practice

• Health-related setting (e.g., public health, clinical setting, rehabilitation)

• Document available in English

• Exclusion Criteria• Reports on the theoretical assumptions about the practice of

knowledge brokering only

• Study protocols

72

MethodSearch strategy

Data Sources

• Databases• MEDLINE; EMBASE; PsychINFO; CINAHL; SCOPUS; SocINDEX; Health

Business Elite

• Grey literature• Knowledge translation networks: Canadian Foundation for Healthcare

Improvement, National Collaborating Centre for Methods and Tools

• Health-focused organizations: CIHR, Canadian Public Health Information, Health Evidence, Ontario Public Health Unit, World Health Organization

• Hand searching • E.g., Implementation Science, BMC Health Service Research, etc.

• Reference lists

73

Method

74

Ide

nti

fica

tio

nSc

ree

nin

gEl

igib

ility

Incl

ud

ed

Records identified through databases

(n = 7,022)

Records identified through other

sources (n = 340)

Records identified through databases

(n = 502)

Records identified through other

sources (n = 71)

January 2014 November 2014

Records after duplicates removed

(n = 6,363)

Records screened (n = 227)

Full-text articles assessed for

eligibility (n = 46)

Full text articles excluded (n = 181)

Records excluded

(n = 6,136)

Articles included in synthesis (n = 26)Unique projects included in synthesis (n = 20)

Articles included in synthesis (n = 3)Unique projects included in synthesis (n = 3)

Total articles (n = 29)Total unique projects (n = 23)

Records after duplicates removed

(n = 573)

Records screened (n = 13)

Full-text articles assessed for

eligibility (n = 12)

Records excluded (n = 560)

Full text articles excluded

(n = 1)

MetaQAT USER INSIGHTS

• Adapted MetaQAT forms to study context prior to conducting critical appraisal

• Modified “hint questions” where appropriate

• Adapted for Excel

MethodCritical appraisal

• Why we chose the MetaQAT to conduct critical appraisal • Permitted analysis of diverse research designs (e.g., qualitative, quantitative,

grey literature) and health-related settings

• Four step critical appraisal framework to assess: relevancy, reliability, validity and applicability

• Research design-specific companion tools for quantitative, qualitative and mixed method research design appraisal

75

Conducting Critical AppraisalUsing an Excel-based template to organize findings

76

Additional companion tools required?

Assessments of Relevancy, Validity, Reliability and

Applicability

Conducting Critical AppraisalCompleting companion tools

77

Where applicable, complete the companion tool before completing the MetaQAT

78

Presenting MetaQAT Results PowerPoint format

Project Relevance Reliability Validity Applicability

A16,17 Relevant setting and population

Insufficient description of population, intervention, outcomes

Potential bias in design, sampling, outcomes

Insufficientdescription of context/outcomes

B18 Relevant setting and population

Insufficient description of population and intervention

Potential bias in sampling, outcomes (unsystematic reflection)

Difficult to generalize

C19 Relevant setting and population

No description of analytic methods

Potential bias in sampling, outcomes (no KB perspective)

Insufficientdescription of context/outcomes

D20-22 Relevant setting and population

No significant concerns No significant concerns No significant concerns

E23 Relevant setting and population

Insufficient description of population, intervention, outcomes

Potential bias in design, sampling, data sources

Insufficient data to support applicability

F24,25 Relevant setting and population

Insufficient description of population and intervention

Potential bias in design, data sources, exclusion criteria

Insufficient description of KB role/outcomes

G26-28 Relevant setting and population

No significant concerns Chance cannot be ruled out (no CIs presented)

No significant concerns

79

Presenting MetaQAT Results Manuscript Format

Interested in our findings?

80

Review available from: https://implementationscience.biomedcentral.com/articles/10.1186/s13012-015-0351-9

https://implementationscience.biomedcentral.com/articles/10.1186/s13012-015-0351-9

• When conducting a meta-synthesis, quality appraisal tool needs to be broad enough to account for various design-specific concerns

• Although designed for use in public health contexts, MetaQAT is applicable to diverse health-related contexts

• E.g., clinical, rehabilitation, health policy settings

• The “Applicability” appraisal was highly useful for assessing utility of knowledge brokering strategies – a key component of our review

• Overall, MetaQAT was the only tool capable of meeting our broad needs relative to methodological and contextual factors

81

Lessons Learned

Public Health Ontario Library Services:


82

Ready to give it a try?


Acknowledgements• We wish to acknowledge the Canadian Institutes of Health Research for

their support of this research program (KAL-129895)

• We wish to thank Public Health Ontario Research Librarians, Allison McArthur and Domna Kapetanos for their assistance with the design and conduct of the search strategy

83

Thank you!

Want to connect? Feel free to reach out:


[email protected]

@CathBornbaum

84

mailto:[email protected]

References

1. Jernberg T, Johanson P, Held C, Svennblad B, Lindbäck J, Wallentin L et al.: Association between adoption of evidence-based treatment and survival for patients with ST-elevation myocardial infarction. Journal of the American Medical Association 2011, 305: 1677-1684.

2. David D, Davis M, Jadad A, Perrier L, Rath D, Ryan D et al. (Eds):The case for knowledge translation: Shortening the journey from evidence to effect. In British Medical Journal 2003, 327: 33-35.

3. Madon T, Hofman K, Kupfer L, Glass R: Public health: Implementation science. Science 2007, 318:1728-1729.

4. Chalmers I (Eds):If evidence-informed policy works in practice, does it matter if it doesn't work in theory? In Evidence & Policy: A journal of Research, Debate and Practice 2005, 1: 227-242.

5. Graham I, Logan J, Harrison M, Straus S, Tetroe J, Caswell W et al. (Eds):Lost in knowledge translation: Time for a map? In Journal of Continuing Education in the Health Professions 2006, 26:13-24.

6. Dobbins M, DeCorby K, Twiddy T (Eds):A knowledge transfer strategy for public health decision makers. In Worldviews on Evidence-Based Nursing 2004, 1: 120-128.

7. Ward V, House A, Hamer S: Knowledge brokering: The missing link in the evidence to action chain?Evidence & Policy 2009, 5: 267-279.

8. Popay J, Roberts H, Sowden A, Petticrew M, Arai L, Rodgers M, et al.: Guidance on the conduct of narrative synthesis in systematic reviews: A product from the ESRC methods programme. 2006.

9. Kujbida G, Stratton J. Effective knowledge translation tactics for increasing the use of health status and surveillance data. 2014. Retrieved from: www.peelregion.ca/health/library/pdf/effective-kt.pdf

85

http://www.peelregion.ca/health/library/pdf/effective-kt.pdf


Your Comments/Questions

• Use Chat to post comments

and/or questions

• ‘Send’ questions to All (not

privately to ‘Host’)

Chat

Participant Side

Panel in WebEx


Poll Question #3

Could this method or tool be useful

in practice?

A. Very useful

B. Somewhat useful

C. Not at all useful

D. Don’t know


Your Feedback is Important

Please take a few minutes to share your thoughts

on today’s webinar.

Your comments and suggestions help to improve

the resources we offer and plan future webinars.

The short survey is available at:

https://nccmt.co1.qualtrics.com/SE/?SID=SV_23n

CMSgc5sz9QJD

https://nccmt.co1.qualtrics.com/SE/?SID=SV_23nCMSgc5sz9QJD


Poll Question #4

What are your next steps? (Check all

that apply)

A. Access the method/tool referenced in the

presentation

B. Read the NCCMT summary about the

method/tool described today

C. Consider using the method/tool in

practice

D. Tell a colleague about the method/tool


Join us for our next webinarSpotlight on Methods and Tools: Knowledge

Translation and Implementation Video Series

Date: Wednesday April 5, 2017

Time: 1:00 – 2:30pm EST

Interested in a tool to support planning program

development? Do you plan program implementation? Are you

interested in an overview of the key concepts regarding

implementation of evidence-based practices? Join us for a

webinar to learn about the Knowledge Translation and

Implementation Video Series.

Register at: https://health-evidence.webex.com/health-

evidence/onstage/g.php?MTID=e4a0468604363b8e349ce95

b2277da0f6

https://health-evidence.webex.com/health-evidence/onstage/g.php?MTID=e4a0468604363b8e349ce95b2277da0f6


Funded by the Public Health Agency of Canada | Affiliated with McMaster UniversityProduction of this presentation has been made possible through a financial contribution from the Public Health Agency of Canada. The

views expressed here do not necessarily reflect the views of the Public Health Agency of Canada..

For more information about the National Collaborating Centre for Methods and Tools:NCCMT website www.nccmt.ca

Contact: [email protected]


mailto:[email protected]