Empirical assessment of univariate and bivariate meta-analyses … · 2016-11-15 · bivariate meta- regression models that assume common variances across tests and those which allow

Empirical assessment of univariate and bivariate meta-analyses for comparing the accuracy of diagnostic tests Yemisi Takwoingi, Richard Riley and Jon Deeks

Outline

• Rationale • Methods • Findings • Summary

Motivating example

Gurusamy et al. Endoscopic retrograde cholangiopancreatography versus intraoperative cholangiography for diagnosis of common bile duct stones. Cochrane Database of Syst Rev 2013.

Motivating example 0.

00.

20.

4 0.

60.

81.

0

Sen

sitiv

ity

0.00.20.40.60.81.0

Specificity

Endoscopic retrograde cholangiopancreatographyIntraoperative cholangiography

Bivariate model 0.

00.

20.

4 0.

60.

81.

0

sens

itivi

ty

0.00.20.40.60.81.0

specificity

correlation

sensitivity

specificity

Models both logit(sensitivity) and logit(specificity) and the correlation between them

Combines two random effects meta-analysis of sensitivity and specificity in a single model

logit(sensitivity) and logit(specificity) are specified as random study effects

Bivariate model specification

Number not diseased

Number diseased

),(~ iSpifpitnbinomialitn +

),(~ iSeifnitpbinomialitp +

Level 1 (allows for within-study variability)

Level 2 (allows for between-study variability)

=∑

∑

2

2

,

, ,~BAB

ABA

B

A

iB

iA withNσσσσ

µµ

µµ

iAiSeit ,)(log µ= iBiSpit ,)(log µ=and

Bivariate model for comparative meta-analysis

Assuming a test type covariate Z that may affect both sensitivity and specificity, the model can be extended as

=∑

∑

++

2

2

,

, with,~BAB

ABA

iBB

iAA

iB

iA

ZvZv

Nσσσσ

µµ

µµ

Effect of test type on variance parameters can also be investigated

…results indicate that simpler hierarchical models are valid in situations with few studies or sparse data. For synthesis of sensitivity and specificity, univariate random effects logistic regression models are appropriate when a bivariate model cannot be fitted…

Applies to meta-analysis of a single test

Univariate model for comparative meta-analysis

Assuming a test type covariate Z that may affect both sensitivity and specificity, the model can be expressed as

=∑

∑

++

2

2

,

, with,~BAB

ABA

iBB

iAA

iB

iA

ZvZv

Nσσσσ

µµ

µµ 0

0

Bivariate model simplifies to 2 univariate random effects logistic regression models for sensitivity and specificity

Estimates from bivariate and univariate models comparing ERCP and IOC

Test Logit sensitivity

(SE)

Logit specificity

(SE)

Variance of random

effects for logit

sensitivity (SE)

Variance of random

effects for logit

specificity (SE)

Correlation of the logits

(SE)

Sensitivity (95% CI)

Specificity (95% CI)

Bivariate model

ERCP 1.55 (0.30) 5.35 (2.25) 0.22 (0.27) 2.95 (6.34) 0.41 (1.05) 82.5 (72.3–89.5) 99.5 (71.8–100)

IOC 7.06 (4.53) 4.15 (0.52) 16.7 (26.9) 0.25 (0.54) -0.73 (0.98) 99.9 (14.1–100) 98.5 (95.8–99.4)

Univariate model

ERCP 1.56 (0.30) 5.32 (2.19) 0.22 (0.26) 2.83 (5.87) 0 82.6 (72.6–89.5) 99.5 (73.8–100)

IOC 6.12 (3.28) 4.19 (0.57) 9.74 (12.7) 0.34 (0.67) 0 99.8 (42.3–100) 98.5 (95.6–99.5)

Estimates from bivariate and univariate models comparing ERCP and IOC

Test Variance of random

effects for logit

sensitivity (SE)

Variance of random

effects for logit

specificity (SE)

Correlation of the logits

(SE)

Sensitivity (95% CI)

Specificity (95% CI)

Bivariate model

ERCP 0.22 (0.27) 2.95 (6.34) 0.41 (1.05) 82.5 (72.3–89.5) 99.5 (71.8–100)

IOC 16.7 (26.9) 0.25 (0.54) -0.73 (0.98) 99.9 (14.1–100) 98.5 (95.8–99.4)

Univariate model

ERCP 0.22 (0.26) 2.83 (5.87) 0 82.6 (72.6–89.5) 99.5 (73.8–100)

IOC 9.74 (12.7) 0.34 (0.67) 0 99.8 (42.3–100) 98.5 (95.6–99.5)

Univariate or bivariate comparative meta-analyses: does it matter?

Aim • To investigate validity of assumption of equal variances

when comparing test accuracy in bivariate meta-regression models. – Are there important differences between findings from

bivariate meta-regression models that assume common variances across tests and those which allow variances to differ by test?

• To examine the impact of using univariate random effects logistic regression models. – Are findings from univariate meta-regression models similar to

those from bivariate meta-regression models?

Data source • Reviews identified in DARE from 1994-2012

Eligibility criteria

Included if

1. Diagnostic accuracy of 2 tests compared

2. Meta-analyses were performed

3. Possible to derive 2x2 tables for included studies

• Reviews identified in DARE from 1994-2012

Data analysis A. Preliminary meta-analysis of each test in a test

comparison performed – Bivariate model fitted to assess model stability and

estimation of correlation parameter

B. Comparative meta-analyses of each test comparison

– Bivariate model with and without equal variances – Univariate model with and without equal variances

Criteria for assessment of performance 1. Difference in magnitude of relative test performance

expressed as ratio of relative sensitivities and ratio of relative specificities

2. Difference in precision of measures of relative test performance expressed as a ratio of standard errors

3. Change in statistical significance at the 5% level: do confidence intervals include 1?

4. Change in direction of effect (qualitative change): is the ranking of a pair of tests in terms of superior sensitivity or specificity consistent between the two models?

Findings – cohort of reviews

• 57 reviews and test comparisons included

• Total number of studies in a test comparison ranged between 6 and 103

• At least one study had a zero cell in 53 test comparisons

Preliminary bivariate meta-analyses: Is correlation reliably estimated?

0

-0.5

-1.0

0.5

1.0

Inde

x te

st -

corr

elat

ion

para

met

er

0-0.5 0.5-1.0 1.0Comparator test - correlation parameter

Rarely similar for a pair of tests

Estimated within boundary of parameter space (> –1 < +1) for remaining 80 (70%)

Correlation = +1 for 12/114 (11%) Correlation = –1 for 22/114 (19%)

Comparative bivariate meta-analyses: Is it important for variances to differ by test?

Differences in magnitude

0.6

0.7

0.8

0.9

1.0

1.1

1.2

1.3

Rat

io o

f poi

nt e

stim

ates

Relative sensitivitiesRelative specificities

11 (22%) test comparisons had more than a 10% difference in relative sensitivity and/or relative specificity.

Across 49 test comparisons, median (IQR) ratios of relative sensitivities and relative specificities were 1.00 (0.99 to 1.01) and 1.00 (0.98 to 1.01).


Differences in magnitude Differences in precision

0.2

0.6

1.0

1.4

1.8

2.2

2.6

3.0

3.4

Rat

io o

f sta

ndar

d er

rors


0.6

0.7

0.8

0.9

1.0

1.1

1.2

1.3

Rat

io o

f poi

nt e

stim

ates



Differences in precision

0.2

0.6

1.0

1.4

1.8

2.2

2.6

3.0

3.4

Rat

io o

f sta

ndar

d er

rors


Standard errors were on average higher for models with unequal variances compared to models with equal variances.

Median (IQR) = 1.37 (1.09 to 1.77) for ratios of standard errors of log relative sensitivities and 1.39 (1.15 to 2.05) for those of log relative specificities.

**

***

**************

**

123456789

101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657

Test

com

paris

on ID

0.125 0.25 0.5 1 2 4 8

Relative sensitivity with 95% confidence interval

Equal variances Unequal variances

A

**

***

**************

**

123456789

101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657

Test

com

paris

on ID

0.125 0.25 0.5 1 2 4 8

Relative specificity with 95% confidence interval


B


For 21 (43%) of the 49 test comparisons, likelihood ratio tests indicated statistically significant differences in model fit.

**

***

**************

**

123456789

101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657

Test

com

paris

on ID

0.125 0.25 0.5 1 2 4 8



A

**

***

**************

**

123456789

101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657

Test

com

paris

on ID

0.125 0.25 0.5 1 2 4 8



B

15 (31%) test comparisons had a change in the statistical significance of relative sensitivity or relative specificity while 4 (8%) had a change in both measures.


Qualitative differences were observed for 11 (22%) test comparisons.

Univariate vs bivariate comparative meta-analyses: Are findings similar?

Differences in magnitude

0.900

0.925

0.950

0.975

1.000

1.025

1.050

1.075

1.100

Rat

io o

f poi

nt e

stim

ates


Differences between both models were negligible.

Across 48 test comparisons, median (IQR) ratios of relative sensitivities and relative specificities were 1.00 (1.00 to 1.01) and 1.00 (1.00 to 1.00).


Differences in magnitude Differences in precision

0.900

0.925

0.950

0.975

1.000

1.025

1.050

1.075

1.100

Rat

io o

f poi

nt e

stim

ates


0.8

0.9

1.0

1.1

1.2

1.3

1.4

1.5

1.6

1.7

Rat

io o

f sta

ndar

d er

rors



Differences in precision

0.8

0.9

1.0

1.1

1.2

1.3

1.4

1.5

1.6

1.7

Rat

io o

f sta

ndar

d er

rors


Standard errors tended to be higher for estimates from bivariate models relative to those from univariate models.

Median (IQR) ratios of standard errors for log relative sensitivities and log relative specificities were 1.00 (1.00 to 1.05) and 1.00 (1.00 to 1.01).


**

*

***

*

****

*

****

****

123456789

101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657

Test

com

paris

on ID

0.125 0.25 0.5 1 2 4 8


Univariate model Bivariate model

A

123456789

101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657

Test

com

paris

on ID

0.125 0.25 0.5 1 2 4 8


Univariate model Bivariate model

B

8 test comparisons where likelihood ratio tests indicated statistically significant difference in model fit between univariate and bivariate models.



Change in statistical significance


Qualitative change

Summary and conclusions • Assumption of equal variances in comparative meta-

analyses is not always justified

• Validity of assumptions should be investigated if data permits

• Minimal impact of using a bivariate structure for comparative meta-analyses

• Univariate meta-regression is an alternative when bivariate meta-regression is not feasible – Provides a solution to non-convergence when data are sparse

Documents

Empirical assessment of univariate and bivariate meta-analyses … · 2016-11-15 · bivariate meta- regression models that assume common variances across tests and those which allow