1

SCreening for Occult REnal Disease (SCORED)Simple Algorithms to Predict Kidney Disease: ready to be used in the real world?

Heejung Bang, PhD & Madhu Mazumdar, PhD Division of Biostatistics and Epidemiology

Department of Public Health

Weill Medical College of Cornell University

2

Overview

Background ObjectivesMethods: model development and

validation ResultsDiscussion

3

BackgroundPrevalence of Kidney Disease (1999-2004)

5.75.4 5.4

0.3 0.10

1

2

3

4

5

6

Stage 1 GFR>90Stage 2 90-60Stage 3 59-30Stage 4 29-15Stage 5 <15

Stages 1 and 2 with kidney damage

4

BackgroundEnd-Stage Renal Disease (ESRD) Counts

5

BackgroundTotal Cost of Medicare for ESRD (in billions)

28.3

14.2

0

6

12

18

24

30

1982 1984 1986 1988 1990 1992 1994 1996 1998 2000 2002 2004 2006 2008 2010

95% CL

Projection

Cost

6

Background

Chronic kidney disease (CKD) is a global health problem. Low-awareness and late detection are common problems.

It is progressive disease. Yet, most affected individuals are asymptomatic with known risk factors and are not routinely tested.

Identifying individuals with CKD should be ‘simple’ with serum creatinine concentration that is widely available and inexpensive ($10-20), in combination with urinalysis.

Systematic methods to predict disease in other chronic conditions such as cardiovascular disease (e.g., Framingham, Reynolds scores, stroke instrument), cancer (e.g., Gail model), diabetes exist but not for CKD.

7

Objectives

To develop risk prediction model for prevalent CKD Important prerequisites in our investigation:

Easy to use but accurate Cumulative effects of concurrent risk factors Demographic + medical history + modifiable risk factors

To test the validity of the model internally as well as using independent large databases (i.e., external validation)

To compare the performance of the model with the current clinical practice guidelines

To develop risk prediction model for incident CKD

8

Kidney Early Evaluation Program (KEEP) by the National Kidney Foundation

if a persons is ≥ 18 years old and has one or more of the following:

1. diabetes

2. high blood pressure

3. a family history of diabetes, high blood pressure or kidney disease

http://www.kidney.org/news/keep/

9

SCreening for Occult REnal Disease

(SCORED)Bang et al. (2007)

10

Methods

Cross-sectional analysis of a nationally representative population based survey, the National Health and Nutritional Examination Surveys (NHANES) 1999-2002

Adult subjects only (≥20 years old) Potential risk factors searched from literature Endpoint: CKD stage 3 or higher, i.e., glomerular

filtration rate (GFR) < 60 ml/min/1.73m2 (using the MDRD formula)

11

Methods (Cont’d)

Split-sample method to create a development and validation dataset using a 2:1 ratio.

Standard diagnostic characteristics: # at high risk, sensitivity, specificity, positive & negative predictive values, area under ROC curve

Multiple logistic regression model (with proper weighting and complex survey design)

e.g., proc surveylogistic in SAS.

12

Methods (Cont’d)

‘Categorical scoring system’ derived by assigning an integer for the regression coefficients

‘Continuous probability’ of having CKD from the fitted regression model

External validation using the Atherosclerosis Risk in Communities (ARIC) Study, Cardiovascular Health Study (CHS) and NHANES 2003-2004.

Comparison between SCORED vs. KEEP using standard diagnostic measures

A number of sensitivity analyses (e.g., missing info, different definitions)

--- important to be used in the real world!

13

Results

NHANES 1999-2002 gave 10,291 individuals

After exclusions (based on unmeasured or missing data, etc.), dataset included 8,530 observations

A total of 601 individuals had CKD (5.4% weighted proportion)

14

Final SCORED model in development data (N= 5,666, AUC=0.88)

15

Diagnostic characteristics of SCORED in internal validation dataset (N=2,864) (cutpoint ≥4 to define high risk group)

16

Event rate by risk score

17

Fitting SCORED model to ARIC dataset (N= 12,038, AUC=0.71)

18

Sample questionnaire

19

Advantages of SCORED

Estimate the cumulative likelihood of having disease with multiple risk factors

Accuracy and high sensitivity. Simple to use (implemented by the pen & pencil

method) so foresee a variety of uses e.g., mass screenings public education initiatives, health fair

medical emergency departments web-based medical information sites patient waiting room in clinics

20

Limitations of SCORED

Inability to assess family history of kidney disease -- many large national and community studies do not

enquire about history of kidney disease. For prevalent disease, not incident disease (a new

risk score is needed, later in this talk) Some variables may be commonly missing (e.g.

proteinuria) Low PPV (but prediction is HARD!) Kidney disease: multiple definitions, different stages

21

Diagnostic performance of SCORED vs. KEEP using external validation data (Bang, Mazumdar et al. 2008)

Screening guidelines % high risk Sensitivity Specificity PPV NPV AUC

SCORED

NHANES 40 95 65 20 99 0.88

ARIC/CHS 51 88 52 1498

0.78

ARIC/CHS* 53 89 50 13 98 0.79

ARIC/CHS* 53 90 50 13 98 0.80

KEEP

NHANES 67 90 35 12 97 0.75

NHANES* 69 92 33 12 98 0.77

ARIC 76 88 24 3 98 0.67

ARIC/CHS 77 86 24 9 95 0.65

* some sensitivity analyses

22

A simple algorithm to predict incident kidney disease

(aka, SCORED II)

by Kshirsagar, Bang et al. In Press

23

Prediction is very hard, especially about the future - Yogi Berra

24

Background

Another important issue is to predict a new disease in disease-free population.

In many asymptomatic diseases, both prevalent and incident diseases are important. (in contrast, for hard outcomes such as heart attack, only incident disease makes sense)

Incident disease is less urgent so less user-friendliness is acceptable.

--- 3 different models developed: 1) best-fitting continuous, 2) best-fitting categorical, 3) simplified categorical.

Beyond AUC. We also used AIC/BIC.

25

Background (Conti’)

We need prospective studies to develop the models.

Internal validation only using Split-sample, no external validation.

Same logistic regression --- so observed outcome among survivors.

Cutpoint for high risk group might be less important.

26

Simplified categorical model (AUC=0.69, AIC=6295, BIC=6374)

CovariateBeta coefficient (standard error)

Odds Ratio (95% Cl)

P valueAssigned

score

Age 50-59 0.63 (0.12) 1.9 (1.5, 2.4) <0.0001 1

60-69 1.33 (0.12) 3.8 (3.0, 5.8) <0.0001 2

70 or older 1.46 (0.14) 4.3 (3.3, 5.6) <0.0001 3

Female 0.13 (0.07) 1.1 (1.0, 1.3) 0.05 1

Anemia 0.48 (0.20) 1.6 (1.1, 2.4) 0.02 1

Hypertension 0.55 (0.07) 1.7 (1.5, 2.0) <0.0001 1

Diabetes mellitus 0.33 (0.10) 1.4 (1.2, 1.7) 0.0006 1

History of cardiovascular disease

0.26 (0.10) 1.3 (1.1, 1.6) 0.009 1

History of heart failure 0.50 (0.25) 1.6 (1.0, 2.7) 0.04 1

Peripheral vascular disease 0.41 (0.13) 1.5 (1.2, 1.9) 0.002 1

27

Risk prediction table for up to 10 years

Total score Estimated Risk (%)

≤1 ≤5

2 8

3 13

4 20

5 25

6 30

7 35

≥8 ≥50

28

Discussion

Evidence-based medicine = Science (theory) + Data + Statistics.

Risk score = Statistics + Art + Reality

--- SCORED is a good example.☺ Performed well in a variety of different settings. Seems to provide the enhanced guidelines upon the current

clinical practice guidelines. It started be utilized in the ‘real world’. SCORED II yet to be validated but strong consistency/

similarities observed in SCORED I and II.

29

Discussion (Conti’)

Categorization can be a bad idea (Royston et al. 2005; Greenland 1995) but is crucial for risk scoring algorithms to be useful in the real world.

More than 1 model may be justified and we can let consumers/users to choose because

All models are wrong, but some are useful ---George Box Relying on only 1 measure (e.g., AUC) can be problematic

(Cook et al. 2006; Cook 2007). Trade-offs between accurate vs. easy medical terms. Risk scores for internet vs. physician’s office vs. Walmart can

be different.

30

Current and future research

Evaluation of SCORED in vascular patients because detection of CKD in patients with or at

increased risk of CVD was emphasized by a science advisory from the American Heart Association and National Kidney Foundation (2006).

Relationships SCORED with other risk scores Testing SCORED in community settings

31

References

BangBang, Vupputuri, Shoham et al. (2007). SCreening for Occult REnal Disease (SCORED). A simple prediction model for chronic kidney disease. Archives of Internal Medicine.

BangBang, Mazumdar, Kern et al. (2008). Validation and Comparison of a novel prediction rule for kidney disease: KEEPing SCORED. Arch Int Med.

Kshirsagar, BangBang, Bomback et al . A simple algorithm to predict incident kidney disease. In Press. Arch Int Med.

BangBang, Mazumdar, Newman et al. Screening for kidney disease in vascular patients. Submitted.

Building and Using Disease Prediction Models in the Real World. Roundtable discussion led by H. Bang at JSM, Utah, 2007. Slides at:

http://www.med.cornell.edu/public.health/conference_presentations.htm

32

Exposed to and used by public

Covered by the CBS Early Show (on World Kidney Day 2007)

SCORED questionnaire is posted in some health information websites

Distributed by ESRD network, KidneyTrust, Am Kidney Fund, UK Dept of Health, and UNC Kidney Center for Kidney Education Outreach Program

“Research Highlights” in Nature Clinical Practice Nephrology (2007)

Lead Story in Physician’s Weekly (2007)