What is a C2 Review Protocol? Presented by: Harris Cooper University of Missouri-Columbia Larry V. Hedges University of Chicago

What is a C2 Review Protocol?

Presented by: Harris Cooper

University of Missouri-Columbia

Larry V. HedgesUniversity of Chicago

A Review Protocol is a document that:

Sets out the reviewers’ intentions with regard to the topic and the methods to be used in carrying out a proposed review

Is meant for inclusion in the Campbell Database of Systematic Reviews

How were the Guidelines for a C2 Review Protocol Established?

The C2 Protocol Guidelines were: Drafted by the Methods Working Group at

the request of the Steering Committee using the Cochrane Collaboration Protocol Guidelines and other sources as guides

Reviewed (via e-mail) by attendees of the C2 Inaugural Meeting, sent to approximately 80 people, 7 of whom provided feedback

Reviewed by the Steering Committee Approved by the Steering Committee

(December 22, 2000)

Why are C2 Review Protocols Necessary?Preparing a review is a complex process

that comprises many judgments and decisions.

The methods to be used should be established beforehand (to the extent possible) because:

The studies to be included are usually identified after they have been completed

The results of many of these studies may already be known to the reviewers

Why are C2 Review Protocols Necessary?Therefore, it is important to make the

review process as well-defined, systematic, and unbiased as possible while maintaining a practical perspective.

Requiring detailed protocols is one way to: Allow the editorial referees to provide guidance

and advice Prevent problems from occurring during the

review process Ensure that final products will meet the

standards of the Campbell Collaboration

What Should a C2 Protocol Contain?

A protocol for a Campbell Review should consist of the following sections:

1. Cover Sheet2. Background for the Review3. Objectives for the Review


4. Methods Criteria for inclusion and exclusion of studies in

the review Search strategy for identification of relevant

studies Description of methods used in the component

studies Criteria for determination of independent

findings Details of study coding categories Statistical procedures and conventions Treatment of qualitative research


5. Timeframe6. Plans for Updating the Review7. Acknowledgements8. Statement Concerning Conflict of

Interest9. References10. Tables

What Should a C2 Protocol Cover Sheet Contain?

The cover sheet of the protocol should include: The title of the review The names of the reviewers Contact information for the lead

reviewer Sources of support

What should a C2 Protocol Background Section Contain?

The background section of the protocol should present: An overview of the theoretical,

conceptual, and/or practical issues surrounding the research problem

A general description of prior reviews, the controversies these reviews have created or left unresolved, and which of these will be the focus of the new review effort

The background section: Sets out the context of an already

formed body of knowledge Provides the rationale for the

review Explains why the questions being

asked are important Sets the stage for the empirical

results that follow

The background section should contain: A conceptual discussion of the research

problem A brief overview of the research

question including its theoretical, practical, and methodological history

The qualitative and historical debates surrounding the research question

A discussion of previous reviews of the research topic

The background section should answer questions such as: From where does the problem, approach,

and/or intervention in the research come? Do debates exist surrounding the

meaning of the problem or utility of the intervention?

Do theories predict how the major variables involved in the review will be related to one another?

Do different theories or philosophies of treatment yield conflicting predictions?

What Should a C2 Protocol Objectives Section Contain?

All Campbell Collaboration Reviews are undertaken to gather, summarize and integrate empirical research so as to help people understand the evidence.

Within this overarching framework, reviews can, for example, be meant to:

Produce general statements about relationships and treatment effects through the synthesis of individual study results

Find reasons for conflicting evidence

Explain variations in practice

Within this overarching framework, reviews can, for example, be meant to:

Answer questions, using variations in studies, that could not have been answered in the individual component studies

Review the evidence on the subjective experience of an intervention

Build connections between related areas of research

In setting out the objectives, reviewers should keep in mind that Campbell Reviews must:

Help people make practical decisions about social behavioral interventions and public policy

Address the choices (practical options) people face when deciding about whether or not to adopt a policy or practice

Relate to outcomes that are meaningful to people making decisions about public policy

What should a C2 Methodology Section Contain?

The methods section should describe operationally how the review will be conducted.


The methods sections of a C2 review will differ considerably from that of a primary research study.


Most methods section will need to address several separate issues:

Criteria for inclusion and exclusion of studies in the review

Search Strategy for identification of possibly relevant studies

Description of methods used in primary research

What should a C2 Methodology Section Contain? Criteria for determination of

independent findings Details of study coding strategies Statistical procedures and conventions

that may be used How qualitative research will be treated

in the review

How should a C2 Protocol Address Questions of Study Inclusion/Exclusion?

The protocol should explain the criteria that will be applied to determine relevance of studies uncovered by the search.

It is critical that these criteria be explicated in advance.


This section of the protocol should answer question such as:

What characteristics of studies will be used to determine if a study was relevant to the topic of interest?

What characteristics of studies will lead to exclusion?


Will decisions be based on: Report tile? Abstracts? Full Reports?

Who will make the relevance decisions? How will reliability of relevance

decisions be assessed?


Examples of studies that would be included and excluded should be given.


C2 reviews can include evidence from studies of implementation. This evidence can derive from a broad range of qualitative and quantitative evidence.

How Should a C2 Protocol Address Questions of Search Strategy?

Reviewers should present details of their search strategy, such as:

Reference databases used Hand searches of specific journals Personal contacts


It should also include a rationale for the search strategy, bearing in mind that the ultimate goal is to reduce the differences between the body of retrieved studies and those that go undiscovered.


Reviewers need to report the keywords, years searched, and search strategy used to guide the search of reference databases and bibliographies.


Reviewers also need to indicate mechanisms they will use to retrieve documents, especially unpublished ones.


As far as possible, evidence considered should not be restricted by nationality of investigators or by language unless there is a good justification to do so.

How Should a C2 Protocol Address Descriptions of Primary Research Methods

The protocol should describe the methods most commonly used in the primary research covered in the review.


This should focus more on designs actually realized than on theory, e.g.

Participant sampling procedures Research designs Measurement methods


Reviewers should identify a few studies that illustrate the methods used in primary research and present the details of these studies.

How Should a C2 Protocol Address Criteria for Determining Independence on Findings?

Reviewers need to describe how they will handle studies that produce multiple findings of effectiveness based on the same data.


This can happen when: Several types of outcome are measured

on the same subjects The same outcome is measured at

several points in time Several treatment groups are compared

to the same control group Several different studies (publications)

use data from the same people


If more than one outcome is used from the same study (or from different studies using the same people) reviewers need to explain how they will assure that the different outcomes will be statistically independent.

How should a C2 Protocol Describe Coding Categories?

Characteristics of studies that will be coded and examined for potential use as moderators of study outcomes should be described.


All retrieved characteristics should be mentioned, even if they are not eventually used.


If some outcomes or moderators are excluded, a rationale should be given for that exclusion.


Information about quality assurance of coding (e.g., coding reliability assessment and monitoring) should be included in this section.

How Should a C2 Protocol Describe Statistical Procedures?

This section should describe the procedures and conventions used to carry out the quantitative analysis of results.


It should explain: The effect size measure used Any adjustments that are made to effect

sizes to reduce bias The techniques that will be used to

combine evidence How missing data will be handled

How Should a C2 Protocol Describe Statistical Procedures? How statistics describing the overall

literature will be presented What techniques will be used to assess

variability of results What techniques will be used to explain

variation in results What sensitivity analyses will be carried

out


A rational for each of the choices above should be given.


If no quantitative synthesis will be carried out, rationale should be given for the alternative technique.

How should a C2 Protocol Address Treatment of Qualitative Research?

Qualitative Studies Can Be Part of C2 Reviews

Qualitative studies can assist in: Defining interventions more precisely The choice of relevant outcome

measure The development of valid research

questions Interpretation heterogeneous results

Qualitative Studies Can Be Part of C2 ReviewsWhen a review contains relevant qualitative

research, reviewers should operationally describe for qualitative studies:

Criteria for inclusion and exclusion of qualitative studies

Methods used in these primary research studies Criteria for determining independent findings Characteristics of included studies in the same

detail as they do for quantitative research.

What Should a C2 Protocol Timeframe Contain?

Examples of some benchmarks to be used in setting targets are the anticipated dates for completion of:

Searches for published and unpublished studies

Pilot testing of inclusion criteria Relevance assessments Pilot testing of study codes and data

collection Extraction of data from research reports Statistical Analysis Preparation of report

What should a C2 Protocol Timeframe Contain?

Reviewers together with the editors of their Collaborative Review Group, must determine and appropriate timeframe for a specific review.

Targets may vary widely from review to review depending on its scope and complexity, as well as the resources available.

What Should a C2 Protocol Plan for Updating the Review Contain?

Updating Plans should include: Specifications for how the review,

once completed, will be updated Information on who will be

responsible for updates The frequency with which updates

can be expected

What Should a C2 Protocol Acknowledgements Section Contain?

Acknowledgment should be made of all individuals contributing to the preparation of the protocol who were not listed on the cover sheet.

What Should a C2 Protocol Statement Concerning Conflict of Interest Contain?

It is a matter of Campbell Collaboration policy that direct funding from a single source with a vested interest in the results of the review is not acceptable.

Reviewers should report any conflict of interest capable of influencing their judgments including:

Personal Political Academic Financial

Campbell Reviews should be free of any real or perceived bias introduced by:

The receipt of any benefit in cash or kind Any hospitality Any subsidy derived from any source that

may have or be perceived to have an interest in the outcome of the review

Financial conflicts of interest cause the most concern. They can and should be avoided, but must be reported if there are any.

Campbell Reviews should be free of any real or perceived bias

It is impossible to abolish conflict of interest, since the only person who does not have some vested interest in a subject is somebody who knows nothing about it.


Disclosing a conflict of interest does not necessarily reduce the worth of a review and it does not imply dishonesty.


Reviewers should include statements in their protocol about potential conflicts even when they are confident that their judgments will not be influenced.


Editors may decide that disclosure is not warranted or they may decide that readers should know about such a conflict of interest so that they can make up their own minds about how important it is.

Decisions about whether or not to publish such information should be made jointly by reviewers and editors.

What Should a C2 Protocol References Section Contain?

The protocol should include complete references of all cited works.References and other stylistic considerations should follow the guidelines of the American Psychological Association:

American Psychological Association. (1994). Publication Manual of the American Psychological Association (4th Ed.). Washington, DC: Author

What Should a C2 Protocol Tables and Figures Contain?

Any tables and figures included in the protocol should appear after the text.

How is a C2 Protocol Registered?

Once a proposed protocol has been completed it should be sent to the appropriate Collaborative Review Group editors.

How is a C2 Protocol Registered?

When the editors are satisfied that the protocol meets the standards of the Campbell Collaboration they will include it in the Review Group’s module for publication in the Campbell Database of Systematic Reviews. Publishing protocols may encourage

interested parties to contact the reviewers and may discourage others from undertaking a review on the same topic

Editors and reviewers should not include a protocol in a module unless there is a firm commitment:

To complete the review within a reasonable timeframe

To keep the review up-to-date once it is completed

What Happens if the Approved C2 Protocol Undergoes Changes? While every effort should be made to

adhere to a predetermined protocol, it is recognized that this is not always possible or appropriate.

Changes in the protocol should not be made on the basis of how they affect the results of the review.

As a rule, when possible analyses should be performed to show the effect of the change on the results of the review.

How should a C2 Protocol be cited?

When the protocol is converted into a full review, the fact that this review was preceded by a published protocol should be noted.

How should a C2 Protocol be cited?

It is Campbell Collaboration policy that protocols that have not been converted into full reviews within:

TWO YEARS

will be withdrawn from the Campbell Database of Systematic Reviews.

What Other Sources of Assistance Can Help Develop a C2 Protocol?

In addition to the guidelines presented above, prospective reviewers will find sound advice for proposing and conducting Campbell Reviews in the following works:

Clark M. & Oxman A. D. (Eds.).(2000). Cochrane reviewers handbook: Version 4.1. In: Review Manager (RevMan) [Computer program]. Version 4.1 Oxford, England: The Cochrane Center. (Available on-line at: http://www.cochrane.org/cochrane/hbook.htm ).


In addition to the guidelines presented above, prospective reviewers will find sound advice for proposing and conducting Campbell Reviews in the following works:

Cooper, H. & Hedges, L. V. (Eds.). (1994). The handbook of research synthesis. New York: Russell Sage Foundation.

What Other Sources of Assistance Can Help Develop a C2 Protocol?In addition to the guidelines presented

above, prospective reviewers will find sound advice for proposing and conducting Campbell Reviews in the following works:

National Health Service Centre for Reviews and Dissemination. (2000). Undertaking systematic reviews of research on effectiveness. York, England: University of York. (Available at: http://www.york.ac.uk/inst/crd/report4.htm ).


Reviewers may contact the Campbell Secretariat for further guidance in preparing Campbell Review protocols.

Documents

What is a C2 Review Protocol? Presented by: Harris Cooper University of Missouri-Columbia Larry V. Hedges University of Chicago