Meta-regression with DisMod-MR: how robust is the model?

Meta-regression with DisMod-MR: how robust is the model?

June 18, 2013

Hannah M Peterson

Post-Bachelor Fellow

2

Global Burden of Disease Study 2010

3

YLDs• Measures morbidity

• Requires age-specific prevalenceo For 291 outcomes

o For 2 sexes

o For 187 countries

o For 3 years

4

Is negative-binomial distribution the best choice?

DisMod-MR

5

Alternative distributions

Distribution Probability Density Function

Normal

Lognormal

Binomial

Negative-binomial

6

Alternative distributions

Distribution Probability Density Function

Normal

Lognormal

Binomial

Negative-binomial

7

Alternative distributions

Distribution Probability Density Function

Normal

Lognormal

Binomial

Negative-binomial

8

Alternative distributions

Distribution Probability Density Function

Normal

Lognormal

Binomial

Negative-binomial

9

Potential experimental frameworks

• Data collectiono Ideal

o Impractical

• Simulationo Impossible to know true data distribution

• Out-of-sample cross validationo Do not have to choose distribution

10

Out-of-sample cross validation

11

Out-of-sample predictive validity

• Randomly select 25% of data to use as “test data”

12

Out-of-sample predictive validity

• Randomly select 25% of data to use as “test data”

13

Out-of-sample predictive validity

• Randomly select 25% of data to use as “test data”

• Fit the remaining 75% of data (“training data”)

14

Out-of-sample predictive validity

• Randomly select 25% of data to use as “test data”

• Fit the remaining 75% of data (“training data”)

• Use fit to calculate statistics for test data

15

Out-of-sample predictive validity

• Randomly select 25% of data to use as “test data”

• Fit the remaining 75% of data (“training data”)

• Use fit to calculate statistics for test data

• For each distribution

• For 1000 test-train splits

• For each disease data set

16

Comparing distributions

How to determine the best distribution?

17

Metrics of evaluation

• Biaso Measures the average difference between observation and estimate

• Median absolute error (MAE)o Measure of overall magnitude of error

• Percent coverage (PC)o Percent of time estimate uncertainty interval contains observation

18

Results

Percent of wins (%)

Distribution Bias MAE PC Total

Normal 22.1 20.6 34.6 25.7

Lognormal 29.7 13.0 36.5 26.4

Binomial 26.3 48.3 1.9 25.5

Negative-binomial 21.9 18.1 27.1 22.4

19

Conclusions

• Choice of distribution doesn’t greatly influence results

• Best overall performance: lognormal distribution

o Contingent on method to adjust data whose value is 0

• Further investigate when each distribution performs best

o Dependent on number of covariates, priors, amount of data?

Thank you

Hannah Peterson

peterhm@uw.edu

www.healthmetricsandevaluation.org