www.kmin-vumc.nl Grading the evidence in systematic reviews of

measurement properties

23 september 2010

Caroline Terwee

Knowledgecenter Measurement InstrumentsDepartment of Epidemiology and Biostatistics

VU University Medical Center

www.kmin-vumc.nl What is a systematic review of measurement properties?

A review of the measurement properties of all available measurement instruments that aim to measure a particular construct in a particular population.

Aim is to select the best instrument for a specific purpose

www.kmin-vumc.nl What is special about a systematic review of measurement properties (SR-MP)?

• A SR-MP has more than one outcome measure, i.e. multiple measurement properties

• Different studies evaluate different measurement properties the number of studies in the analysis is different per measurement property

• The quality of the studies is evaluated per measurement property

• Data synthesis is different per measurement property

• Evidence for one measurement property may come from different studies

Therefore, a SR-MP is actually a collection of separate reviews per measurement property

www.kmin-vumc.nl Methodology of systematic reviews of measurement properties

1. formulating a research question

2. performing a literature search

3. formulating eligibility criteria

4. selecting abstracts and full-text articles

5. evaluating the methodological quality of the included studies

6. data extraction

7. content comparison

8. data synthesis - evaluating of quality of the instrument

9. overall conclusion of the systematic review

10. reporting a systematic review of measurement properties

10 steps

www.kmin-vumc.nl Methodology of systematic reviews of measurement properties

1. formulating a research question

2. performing a literature search

3. formulating eligibility criteria

4. selecting abstracts and full-text articles

5. evaluating the methodological quality of the included studies

6. data extraction

7. content comparison

8. data synthesis - evaluating of quality of the instruments

9. overall conclusion of the systematic review

10. reporting a systematic review of measurement properties

10 steps

www.kmin-vumc.nl Data synthesis

2 steps

1. Decide on combining studies

• Homogeneity of study characteristics study population, setting, (language) version of the instrument, mode of administration, design characterstics (time interval)

• Methodological quality• Consistency of the results of the measurement properties

www.kmin-vumc.nl

2 steps

1. Decide on combining studies

• Homogeneity of study characteristics study population, setting, (language) version of the instrument, mode of administration, design characterstics (time interval)

• Methodological quality• Consistency of the results of the measurement properties

2. Decide on the analysis:

• Quantitative analysis statistical pooling• Qualitative analysis best evidence synthesis

Data synthesis

www.kmin-vumc.nl ExampleReliability of the Quebec Pain Disability Scale (QBPDS)

Nine studies evaluated reliability; 8 included in the analysis

1. Kopec et al. J Clin Epidemiol 1996

2. Schoppink et al. Phys Ther 1996

3. Fritz et al. Phys Ther 2001

4. Davidson et al. Phys Ther 2002

5. Mousavi et al. Spine 2006

6. De Beer et al. S Afr J Physiother 2008

7. Melikoglu et al. Spine 2009

8. Hicks et al. Pain Med 2009

www.kmin-vumc.nl

1 2 3 4 5 6 7 8

Age 40 (10) 44 (12) 79 (5) 40 (12) 42 (9) 55 (17) 37(10) 45 (15)

% Male 70 32 29 45 5 64 48 26

Duration complaints (mo)

12 (0-128)

85%

>12

7 (6) 84 (108)

? 50% >6 <1 51 (50)

Setting General practice

Retirement

community

Orthop/ trauma clinic

Hospital - PT

Hospital Hospital + PT

practice + com.

serv

PT outpatient clinic

Hospital - PT

Country NL Brazil US Iran South Afrika

Australia

US Turkey

Language Dutch Portugese

English Persian Tswana English English Turkish

Study population, setting, country

www.kmin-vumc.nl

1 2 3 4 5 6 7 8

Age 40 (10) 44 (12) 79 (5) 40 (12) 42 (9) 55 (17) 37(10) 45 (15)

% Male 70 32 29 45 5 64 48 26

Duration complaints (mo)

12 (0-128)

85%

>12

7 (6) 84 (108)

? 50% >6 <1 51 (50)

Setting General practice

Orthop/ trauma clinic

Retirement

community

Hospital - PT

Hospital Hospital + PT

practice + com.

serv

PT outpatient clinic

Hospital - PT

Country NL Brazil US Iran South Afrika

Australia

US Turkey

Language Dutch Portugese

English Persian Tswana English English Turkish

Study population, setting, country

www.kmin-vumc.nl Methodological quality

COSMIN checklist

COnsensus-based Standards for theSelection of Health Measurement INstruments

Different boxes for each measurement property, with questions regarding quality aspects

Box reliability: 14 items to evaluate the quality of a reliability study

www.kmin-vumc.nl

1 2 3 4 5 6 7 8

Missing items

20% deleted

? ?% deleted

0 ? ? ? ?

Sample size 89 54 56 31 31 47 23 100

Time interval (d)

7 3-4 11 1 1 42 28 1

Stable patients

Assumable?

Treatment?

Assumable?

Treatment?

Based on

GPC.

Treatment?

Assumable

Assumable

Based on

GPC. PT

treatment

Based on

GPC. PT

treatment

Assumable

Test conditions

2x mail inter/intra-rater

2x mail ? 2x clinic 2x mail 2x clinic 2x clinic

Methodological quality of the studies

www.kmin-vumc.nl

1 2 3 4 5 6 7 8

Missing items

20% deleted

? ?% deleted

0 ? ? ? ?

Sample size 89 54 56 31 31 47 23 100

Time interval (d)

7 3-4 11 1 1 42 28 1

Stable patients

Assumable?

Treatment?

Assumable?

Treatment?

Based on

GPC.

Treatment?

Assumable

Assumable

Based on

GPC. PT

treatment

Based on

GPC. PT

treatment

Assumable

Test conditions

2x mail inter/intra-rater

2x mail ? 2x clinic 2x mail 2x clinic 2x clinic

COSMIN good good good fair fair fair poor fair

Methodological quality of the studies

www.kmin-vumc.nl

1 2 3 4 5 6 7 8

Missing items

20% deleted

? ?% deleted

0 ? ? ? ?

Sample size 89 54 56 31 31 47 23 100

Time interval (d)

7 3-4 11 1 1 42 28 1

Stable patients

Assumable?

Treatment?

Assumable?

Treatment?

Based on

GPC.

Treatment?

Assumable

Assumable

Based on

GPC. PT

treatment

Based on

GPC. PT

treatment

Assumable

Test conditions

2x mail inter/intra-rater

2x mail ? 2x clinic 2x mail 2x clinic 2x clinic

COSMIN good good good fair fair fair poor fair

Methodological quality of the studies

www.kmin-vumc.nl

1 2 3 4 5 6 7 8

ICC 0.90 0.89-0.93

0.94 0.86 0.91 0.84 0.55 0.92

Consistency of results

www.kmin-vumc.nl

1 2 3 4 5 6 7 8

ICC 0.90 0.89-0.93

0.94 0.86 0.91 0.84 0.55 0.92

Results of the studies

COSMIN good good good fair fair fair poor fair

www.kmin-vumc.nl

1 2 3 4 5 6 7 8

ICC 0.90 0.89-0.93

0.94 0.86 0.91 0.84 0.55 0.92

Results of the studies

COSMIN good good good fair fair fair poor fair

Duration complaints (mo)

12 (0-128)

85%

>12

7 (6) 84 (108) ? 50% >6 <1 51 (50)

www.kmin-vumc.nl Step 2: Best evidence synthesis

Data syntheses is different per measurement property

General guideline: levels of evidence, based on Cochrane Back Review group

Level Rating Criteria

strong +++ or ---

Consistent findings in multiple studies of good

methodological quality OR in one study of excellent

methodological quality

moderate ++ or --

Consistent findings in multiple studies of fair

methodological quality OR in one study of good

methodological quality

limited + or - One study of fair methodological quality

conflicting +/- Conflicting findings

unknown ? Only studies of poor methodological quality

www.kmin-vumc.nl ExampleReliability of the Quebec Pain Disability Scale (QBPDS)

Consistent findings of good reliability (ICC>0.70) in three studies of good methodological quality and in four studies of fair methodological quality

Strong evidence for a good reliability (+++)

1 2 3 4 5 6 7 8

ICC 0.90 0.89-0.93

0.94 0.86 0.91 0.84 0.55 0.92

COSMIN good good good fair fair fair poor fair

www.kmin-vumc.nl Other measurement propertiesInternal consistency

Consistent findings in mutiple studies of good methodological quality or one study of excellent methodological quality that (sub)scales are unidimensional

PLUS

Consistent findings in mutiple studies of good methodological quality or one study of excellent methodological quality that Cronbach’s alpha is > 0.70

www.kmin-vumc.nl Content validity

Strong evidence: all items are considered relevant for the construct, purpose, and target population and the instrument is considered comprehensive.

Moderate evidence: the items are considered relevant for the construct or target population and the instrument is considered comprehensive.

Limited evidence: only one aspect of content validity is assessed.

www.kmin-vumc.nl Construct validity and responsiveness

Levels of evidence as described in the Table are applied.

Challenges

• some studies examine more hypotheses than others

• some hypotheses are more challenging than others

• some comparison instruments are better than others

www.kmin-vumc.nl Summary

similarities and dissimilarities with GRADE

Study limitations (methodological quality) are taken into account, but not to downgrade the level of evidence but to exclude studies from the data analysis

Inconsistency is taken into account in applying levels of evidence

Indirectness (generalizibility) is usually not taken into account because there are too many differences in study characteristics and the influence on the measurement properties is unclear

Impresision (only one study) is very common. This is taken into account in applying levels of evidence

Publication bias is not considered, but might be a problem

www.kmin-vumc.nl Systematic reviews of measurement properties

www.kmin-vumc.nl

PhD defence Wieneke Mokkink

COSMIN checklist

COnsensus-based Standards for theSelection of Health Measurement INstruments

TODAY

13.45

AULA VU