Generalizations of the Receiver Operating Characteristic ... · ROC curve overview. De nition The...

ROC curves gROC Real data apps. Future work Conclusions

Generalizations of theReceiver Operating

Characteristic (ROC) Curve

Pablo Martınez-Camblor (et al.)

Hospital Universitario Central de Asturies (HUCA)Oviedo, Asturies (Spain)

pablomc@ficyt.es

Melbourne, 23 July 2015

Talk overview

1 ROC curve overviewDefinitionEmpirical non-parametric estimatorAsymptotic propertiesArea under the ROC curve

2 ROC curve generalizationMotivation and definitionNon-parametric estimatorAsymptotic propertiesArea under the ROC curve

3 Real data applicationsLeukocyte count and mortality risk in critically ill childrenResponse to the OnabotulinumtoxinA treatment for thechronic migraine headaches and the CGRP levels in women

4 Future work

5 Conclusions

ROC curve overview.Definition

The well-known receiver operating characteristic (ROC) curve(Green and Swets, 1966) is a very useful instrument to measurehow well a (bio)marker is able to distinguish two populations(frequently healthy vs. diseased) from each other.

It displays in a plot the sensitivity (SE ) or true-positive rate (TPR)against the false-positive rate (FPR) or 1-specificity (1− SP) foreach possible threshold, x ∈ R.

Hence, the ROC curve is the geometric place of the points{1− SP(x);SE (x)} with x ∈ R.

−2 0 2 4 6

●T Threshold

1 − SE

1 − SP

0.0 0.2 0.4 0.6 0.8 1.0

1−Specificity

Figure: Density functions for the negative and positive subjects (left) andcorresponding ROC curve (right).

Conventionally (wlg), it is assumed that the larger values of themarker indicate the larger confidence that a given subject ispositive. Let χ and ξ be two random variables representing the(bio)marker values for negative and positive subjects, respectively,for a fixed point t ∈ [0, 1], the ROC curve is

Definition

R(t) =1− Fξ(F−1χ (1− t))

=P{ξ > F−1χ (1− t)}=P{1− Fχ(ξ) ≤ t} = F1−Fχ(ξ)(t), (1)

where Fχ and Fξ denote the CDF for the variables χ and ξ.

ROC curve overview. Definition

Note that, under the above assumption, the ROC curves can beread as

R(t) = suput∈Ut

P{ξ ∈ ut} (2)

where Ut = {ut = [a,∞), a ∈ R such that P{χ ∈ ut} ≤ t}.

−2 0 2 4 6

1 − SP=0.1

−4 −2 0 2 4 6

1 − SP=0.1

ROC curve overview.Empirical non-parametric estimator

The direct empirical non-parametric ROC curve estimator is theresulting of replacing, in (1), the (unknown) CDF by the ECDF.Let X and Y be two random samples drawed from ξ and χ, foreach t ∈ [0, 1], the empirical ROC curve estimator is defined by

Definition

R(t) = 1− Fn(X , F−1m (Y , (1− t)), (3)

where Fn(X , ·) is the ECDF referred to the sample X (with size n)and F−1m (Y , ·) = inf{s ∈ R / Fm(Y , s) ≥ ·}.

ROC curve overview. Asymptotic properties

Of course, asymptotic properties have been deeply studied. Hsiehand Turnbull (1995) enunciated the results:

Theorem (consistency)

Under usual (and mild) conditions,

‖R − R‖∞ →n 0 a.s.

Theorem (weak convergence)

√n · [R(t)−R(t)] =λ1/2 · r(t) · B(m)

1 (1− t)

+ B(n)2 (1−R(t)) + o(1) a.s.

where r = R′, λ = lim(m/n) and B(m)1 , B

(n)2 Brownian Bridges.

ROC curve overview.Area under the curve

The area under the ROC curve (AUC), defined as

Definition

∫R(t)dt, (4)

is frequently used for summarizing the diagnostic capacity. In(this) right-side context, it has a direct probabilistic interpretarion,

A =1−∫

Fξ(F−1χ (1− t))dt

=1−∫

Fξ(u)dFχ(u) = P{χ ≤ ξ}.

ROC curve overview. Area under the curve

The direct non-parametric estimator (again, replacing unknownCDF by the respective ECDF) is the well-known Mann-Whitneystatistics,

Definition

n∑i=1

m∑j=1

I{yj < xi}. (5)

From the previous weak convergence is derived (Hsieh andTurnbull; 1995),

√n · [A − A] −→n N(0, σλ).

ROC curve generalization.Motivation and definition

However, sometimes not only the larger (lower) marker values areassociated with disease, but both the lower and larger values arerelated to the presence of the studied feature. For instance, inhaemodialysis population, both the high and low levels of serumiPTH, calcium and phosphate are associated with higher risk ofmortality. Also, in the intensive care units, leukocyte countsgreater than 20,000 (leukocytosis) or below 5,000 (leukopenia) areassociated with bad prognosis in critically ill patients.

ROC curve generalization. Motivation and definition

−4 −2 0 2 4 6

s1 − SP=0.1

−6 −4 −2 0 2 4 6

1 − SP=0.1fχ

0.0 0.2 0.4 0.6 0.8 1.0

1−Specificity (FPR)

0.0 0.2 0.4 0.6 0.8 1.0

From the ROC definition given in (2),

Remember!

R(t) = suput∈Ut

P{ξ ∈ ut},

where Ut = {ut = [a,∞), a ∈ R such that P{χ ∈ ut} ≤ t}.

it can be extended to the two-side situations just considering thefollowing definition

Definition

Rg (t) = supvt∈Vt

P{ξ ∈ vt}, (6)

where Vt = {vt = (−∞, xl ] ∪ [xu,∞), xl < xu ∈ R such thatP{χ ∈ vt} ≤ t}.

−4 −2 0 2 4 6

1 − SP=0.1

−6 −4 −2 0 2 4 6

1 − SP=0.1

0.0 0.2 0.4 0.6 0.8 1.0

If for each t ∈ [0, 1] is defined

Ft = {(xl , xu) ∈ R2 such that (−∞, xl ] ∪ [xu,∞) = vt ∈ Vt},

from (6),

Remember!

Rg (t) = supvt∈Vt

P{ξ ∈ vt},

where Vt = {vt such that P{χ ∈ vt} ≤ t}.

Rg (t) = sup(xl ,xu)∈Ft

{Fξ(xl) + 1− Fξ(xu)},

Besides, given t ∈ [0, 1], for (xl , xu) ∈ Ft (which impliesFχ(xl) + 1− Fχ(xu) ≤ t), there exists γ ∈ [0, 1] (γ = Fχ(xl)/t)such that

Fχ(xl) = γ · t =⇒ xl = F−1χ (γ · t),

and then

Fχ(xu) ≥ 1− [1− γ] · t =⇒ xu ≥ F−1χ (1− [1− γ] · t).

Hence,

Rg (t) = supγ∈(0,1)

{Fξ(F

−1χ (γ · t)) + 1− Fξ(F

−1χ (1− [1− γ] · t))

or, equivalently

Rg (t) = supγ∈(0,1)

{1−R(1− γ · t) +R([1− γ] · t)}

= {1−R(1− γt · t) +R([1− γt ] · t)} ,

where γt = arg supγ∈(0,1) {1−R(1− γ · t) +R([1− γ] · t)}.

The value of γt determines the optimum proportion offalse-positives, γt · t, due to the negative subjects with a markerbelow xl (left tail), and those due to the negative subjects with amarker larger than xu (right tail), [1 − γt] · t.

0.0 0.2 0.4 0.6 0.8 1.0

γt = 0.5

R([1 − γt]·t)

● 1−R(1 − γt·t)

0.0 0.2 0.4 0.6 0.8 1.0

1−R(1−γ·t)+R([1−γ]·t)

ROC curve generalization.Non-parametric estimator

Replacing the unknown (right- and left-side) ROC curves fromtheir empirical estimators (previous and directly defined in (3)) it isobtained the empirical estimator for Rg ,

Definition

Rg (t) = supγ∈[0,1]

{1− R(1− γ · t) + R([1− γ] · t)

1− R(1− γt · t) + R([1− γt ] · t)}. (8)

where γt = arg supγ∈[0,1]

{1− R(1− γ · t) + R([1− γ] · t)

The supremum can be (must be) computed by numerical methods.

ROC curve generalization.Asymptotic properties

Generalized ROC curve inherits most of the usual ROC curveproperties. Particularly, can be easily derived the uniformconsistency:

Theorem (consistency)

‖Rg −Rg‖∞ →n 0 a.s.

For each t, γ ∈ [0, 1], let be

H(t, γ) ={1− R(1− γ · t) + R([1− γ] · t)} and

H(t, γ) ={1−R(1− γ · t) +R([1− γ] · t)}.

From the ROC curve properties, for each γ ∈ [0, 1],

sup0≤t≤1

|H(t, γ)−H(t, γ)| −→n 0 a.s.

and the result is derived from the obvious inequalities,

Rg (t)−Rg (t) ≥ H(t, γt)−H(t, γt),

Rg (t)−Rg (t) ≤ H(t, γt)−H(t, γt).

ROC curve generalization. Asymptotic properties

When the function γt satisfies that γ′t is bounded for eacht ∈ [0, 1], then it holds

√n · [Rg (t)−Rg (t)] =

λ1/2 · [1− γt − γ′t · t] · r([1− γt ] · t) · B(m)1 (1− [1− γt ] · t)

+ B(n)2 (1−R([1− γt ] · t))

− λ1/2 · [−γt − γ′t · t] · r(1− γt · t) · B(m)1 (γt · t)

− B(n)2 (1−R(1− γt · t)) + o(1) a.s.

where r = R′, λ = lim(m/n) and B(m)1 , B

(n)2 Brownian Bridges.

The proof is not difficult but it is really tedious. The standardROC curve properties are applied on

√n · [H(t, γt)−H(t, γt)]

and the two-term Taylor expansion

√n · H(t, γt) =

√n · H(t, γt)−

√n · h(2)(t, γt) · (γt − γt)

2!·√n · h′(2)(t, τt) · (γt − γt)

with τt ∈ (min{γt , γt}, max{γt , γt}). The γt definition and theM-statistics properties lead to the equality

√n · H(t, γt) =

√n · H(t, γt) + o(1) a.s.

ROC curve generalization.Area under the gROC curve

Directly, the area under the generalizated ROC curve is

Definition

∫Rg (t)dt.

Its direct empirical estimator,

Definition

∫Rg (t)dt.

ROC curve generalization.Area under the gROC curve

Asymptotic distribution of Ag can be derived, particularly, it canbe obtain a result in the way,

Theorem (Weak convergence)

Under usual (and mild) conditions, and under regularity on γt .

√n · [Ag −Ag ] −→n N(0, δλ).

Unfortunately, the value of δ2λ is not easy to compute, evenassuming γ′t = 0 (unrealistic). It must approximated viaresampling.

Real data applications.Leukocyte count and mortality risk in critically ill children

Background

Having available tools to determine the risk of mortality atadmission to the Paediatric Intensive Care Unit (PICU), orduring the first 24 hours after admission, is a clinical necessity.

Leukocyte count measurement constitutes a routinelydetermination when a patient is admitted to the PICU.

Classically, low leukocyte or high leukocyte counts aredescribed as one of the criteria for the diagnosis of sepsis.

Design

Prospective observational study set in two PICUs of UniversityHospitals. The study was conducted in a number ofconsecutive patients, age below 18 years, who were admittedto one of these PICUs.

Real data applications. Leukocyte count...

Sample

A total of 188 patients were finally included.

Leukocyte count routine determinations were performedduring the first 12 hours after admission.

Patients were divided in two groups: higher score riskmortality group (high MR) included patients with PIM 2 andPRISM III score greater than the percentile 75 (N = 12);lower score risk mortality group (low MR) included patientswith a PIM 2 and/or PRISM score below the percentile 75(N = 176).

Rey, Garcıa-Hernandez, Concha et al. (2013), for moreinformation about this cohort.

Table: Descriptive statistics for the leukocyte count.

N Mean ± sd P25 P50 P75

Low MR 176 14,290.7 ± 6,588 9,725 13,400 17,875High MR 12 18,325.8 ± 20,343 7,250 11,400 22,850

6 7 8 9 10 11 12 13

log(Leukocyte count)

High MRLow MR

0.0 0.2 0.4 0.6 0.8 1.0

1−R(1−t)

Results

Left-side AUC=0.520 (0.275-0.693)

Ag = 0.74 (0.596-0.884).

If the children with leukocyte count lower than 24,900 (10.12in logarithmic scale) and greater than 10,200 (9.23 inlogarithmic scale) were classified within the low MR group(optimal thresholds in the Youden index sense) the observedTPR and the TNR were 0.750 and 0.636, respectively.

Real data applications.Response to the OnabotulinumtoxinA treatment for thechronic migraine headaches and the CGRP levels in women

Background

The OnabotulinumtoxinA (BoToX) is the first and onlyFDA-approved (United State food and drug administration),preventive treatment for chronic migraine in adults

On the other hand, serum CGRP levels are increased inchronic migraineurs indicating a chronic activation of thetrigemino-vascular system, and it is proposed as the firstbiomarker for this entity (Cernuda-Morollon et al. (2015)).

We explored the relationship between basal levels of CGRPand the response to the BoToX.

Real data applications. CGRP levels...

Sample

Finally, a total of 70 women meeting chronic migraine criteriawere included (all of then treated at the HUCA).

Migraine patients are usually considered as responders whenattack frequency is decreased by 50%, so we adopted thiscriterion and we checked it by the use of monthly headachecalendars in all patients.

Table: Descriptive statistics for the CGRP levels in the response andnon-response groups.

N Mean ± sd P25 P50 P75

No response 15 57.68 ± 22.01 38.71 50.45 66.91Response 55 70.97 ± 33.01 45.01 71.04 88.34

1 2 3 4 5 6 7

log(CGRP levels)

ResponseNo response

●●

Response Non−Response

0.0 0.2 0.4 0.6 0.8 1.0

Results

Right-side AUC=0.619 (0.473-0.765).

Ag = 0.73 (0.579-0.881).

Women with CGRP levels below 36.51 and larger than 66.97are the optimal group. These cut-off points lead to FPR of0.267 and TPR of 0.673.

Different possibilities: placebo effect, other mechanism....

Of course, sample size must be increased...

Future work. Multivariate biomarkers

The introduced definition

same definition, again!

Rg (t) = suput∈Ut

P{ξ ∈ ut},

where Ut = {ut ∈ B(t) such that P{χ ∈ ut} ≤ t},

can be extended to different Ut subsets. We consider amultivariate problems; i.e., we have different variables which canbe used in order to classified subjects.

When we want to study the diagnostic capacity of several(bio)markers, a common procedure is to reduce the providedinformation into a one-dimensional marker (via logistic regression,linear discriminant analysis...).

●●

●● ●

−4 −2 0 2 4 6

X2 ●

●●

● ●

●●

● ●

●●

●● ●

0.0 0.2 0.4 0.6 0.8 1.0

Of course, this is not a new problem and there exist a number ofpapers which deal with,

Richard, Hammit and Tsevat (1996)

Pepe and Thompson (2000)

Pepe, Cai and Longton (2006)

Gao et al. (2008)

Pfeiffer and Bura (2008)

Yu and Park (2015)

Many others...

Most of them study/propose algorithms in order to maximize theAUC or the partial AUC but they are not looking for a realcurve/parameter. They do not have a theoretical framework.

Rg (t) = suput∈Ut

P{ξ ∈ ut},

where Ut = {ut ∈ B(t) such that P{χ ∈ ut} ≤ t},

B = {x ∈ Rp such that: a · X < x}, or

B(t) = {x ∈ Rp such that: at · X < x},

where X = [X1, · · · ,Xp, 1] contains the p-dimensional biomarkerand the constant term.

●●●

●●

●●●

● ●

●●

●●●

●●

●●●

●●

●● ●

●●

● ●

●●

●●● ●

●●

−4 −2 0 2 4 6

X2 ●

●●

● ●

●●

●●●

● ●●

●●

● ●●

●●●

●●

● ●

●●

● ●●

●● ●

●●

● ●●

●●●

●●

●●●

● ●

●●

●●●

●●

●●●

●●

●● ●

●●

● ●

●●

●●● ●

●●

50%95%

●●●

●●

●●●

● ●

●●

●●●

●●

●●●

●●

●● ●

●●

● ●

●●

●●● ●

●●

−4 −2 0 2 4 6−

X2 ●

●●

● ●

●●

●●●

● ●●

●●

● ●●

●●●

●●

● ●

●●

● ●●

●● ●

●●

● ●●

●●●

●●

●●●

● ●

●●

●●●

●●

●●●

●●

●● ●

●●

● ●

●●

●●● ●

●●

Of course, we can consider non-linear regions. In this case, theproblem is similar to this one (Lei, Robins and Wasserman (2013))

Prediction regions and density level sets

we observe iid data Y1, . . . ,Yn ∈ Rp from a distribution P and wewant to construct a prediction region Cn = Cn(Y1, · · · ,Yn) ⊆ Rp

such thatP(Yn+1 ∈ Cn) ≥ 1− α,

where 0 < α < 1 and P = Pn+1 is the product probability measureover the (n + 1)-tuple (Y1, · · · ,Yn+1).

In the ROC curve context, two populations must be handled but,basically, is the same problem (conformal prediction regions).There is a number (big number) of papers.

The proposed generalizated ROC curve provides a theoreticalframework. To take into account,

Regions with biological sense, usefull in practice...

Computational costs

Conclusions (Limitations)

The introduced definition provides a useful theoretical framework.Rg (t) ≥ max{R(t), 1−R(1− t)}, for each t ∈ [0, 1].

R(t) > max{R(t), 1− R(1− t)} ⇒ Ag > 1/2. Usual bootstrapdoes not work in order to check (the null) H0 : Ag = 1/2 neither tobuild confidence intervals when Ag is close to 1/2. Equivalent to

making it for max{R(t), 1− R(1− t)}.In the consider case (Ut sets in the way (−∞, al ] ∪ [au,∞)) theAUC losses its probabilistic interpretation.In this situation, large AUCs must not be expected:

−10 −5 0 5 10

−4 0 2 4 6 8

References

Cernuda-Morollon E, Larrosa D, Ramon C, et al. (2013)

Interictal increase of CGRP levels in peripheral blood as a biomarker forchronic migraine

Neurology, 81(14), 1191-1196.

Green DM, Swets JA, (1966)

Signal detection theory and psychophysics

Wiley, New York.

Hsieh F, Turnbull BW, (1996)

Nonparametric and semiparametric estimation of the receiver operatingcharacteristic curve

Annals of Statistics, 24(1), 25-50.

Lei J, Robins J, Wasserman L, (2013)

Distribution free prediction sets

Journal of the American Statistical Association, 108(501), 278–287.

References

Martınez-Camblor P, Corral N, Rey C, et al. (2014)

Receiver operating characteristic curve generalization for non-monotonerelationships

Statistical Methods in Medical Research, doi:10.1177/0962280214541095.

Pepe MS, Thompson ML, (2000)

Combining diagnostic test results to increase accuracy

Biostatistics, 1(2), 123-140.

Rey C, Garcıa-Hernandez I, Concha A, et al. (2013)

Pro-adrenomedullin, pro-endothelin-1, procalcitonin, C-reactive proteinand mortality risk in critically ill children: a prospective study

Critical Care, 17:R240.

Generalizations of the Receiver Operating Characteristic ... · ROC curve overview. De nition The...

Documents

Receiver operating characteristic (ROC) analysis · The Theory of Receiver Operating Characteristic Curves 1.1 Introduction The Receiver Operating Characteristic (ROC) analysis comes

Receiver Operating Characteristic Curve (ROC) Analysis for Prediction Studies Ruth O’Hara, Helena Kraemer, Jerome Yesavage, Jean Thompson, Art Noda, Joy

Operational measures and accuracies of ROC curve on large ... · A receiver operating characteristic (ROC) curve can be used to evaluate the performances of algorithms in many biometric

THREE ESSAYS ON INCOME AND WEALTH - Summitsummit.sfu.ca/system/files/iritems1/9265/etd4259.pdf · propose using a Receiver Operating Characteristic (ROC) curve to determine the optimum

© 2009, KAISER PERMANENTE CENTER FOR HEALTH RESEARCH Receiver Operating Characteristic (ROC) Curves Assessing the predictive properties of a test statistic

Statistical Approaches to the Analysis of Receiver Operating Characteristic (ROC) Curves · 2004. 2. 25. · 138 McNEIT exo HaNLPY area under the ROC curve. When discrete rating dat-a

Curvas ROC (Receiver-Operating-Characteristic) y sus

Frequency Selectivity and Masking - MIT … various combinations of Hit rates and False Alarm rates for a given sensitivity results in a Receiver Operating Characteristic (ROC). Figure

Model Evaluation - unipi.itdidawiki.cli.di.unipi.it/lib/...evaluation.2017.pdf · ROC (Receiver Operating Characteristic) Developed in 1950s for signal detection theory to analyze

ROC Curves. ROC (Receiver Operating Characteristic) curve ROC curves were developed in the 1950's as a by-product of research into making sense of radio

Receiver Operating Characteristic (ROC) Curveshomepage.divms.uiowa.edu/.../ROC_introduction.pdf · 2019-04-10 · Receiver Operating Characteristic (ROC) Curves Evaluating a classifier

The Meaning and Use of the Area under a Receiver Operating ... · PDF filean ROC curve and, using the links ... under a Receiver Operating Characteristic (ROC) Curvel 29. TABLE I:

Proper Use of ROC Curves in Intrusion/Anomaly Detection · ROC curves (receiver operating characteristic curves) are commonly used to ... 1 Introduction This manual will explain what

The ROC Toolbox: A toolbox for analyzing receiver ... · The ROC Toolbox: A toolbox for analyzing receiver-operating characteristics derived from confidence ratings Joshua D. Koen1

Initial absolute monocyte count as an immune biomarker for ...Fig. 1 Initial AMC cut-off point: receiver operating characteristic curve (ROC) for initial AMC (absolute monocyte count)

COMP 551 -Applied Machine Learningwlh/comp551/slides/06-eval_and_reg.pdf · Receiver operating characteristic (ROC) § Consider simple 2-class classification (blue = negative class;

ROC & AUC, LIFT ד " ר אבי רוזנפלד. Introduction to ROC curves ROC = Receiver Operating Characteristic Started in electronic signal detection theory (1940s

Receiver Operating Characteristic (ROC) Analysis Visual expertise ... the military, radar screens need to be monitored for approaching targets (e.g., missiles) and decisions made as

Practical use of receiver operating characteristic

Instructional Grouping Worksheets DIBELS Next … · The receiver (or relative) operating characteristic (ROC) curve has become the standard for the evaluation of accuracy for screening