Adaptive FDR Estimation for High Dimensional Discrete Data

Adaptive FDR Estimation for High DimensionalDiscrete Data

Naomi Altman1 & Isaac Dialsingh2

ADNAT 2012 - HyderabadFebruary 5, 2013

1. The Pennsylvania State University 2. The University of the West Indiesnaomi@stat.psu.edu Isaac.Dialsingh@sta.uwi.edu

Altman & Dialsingh (Penn State) Discrete FDR February 5, 2013 1 / 34

False Discovery Rate

Controlling error rates is essential for high dimensional “omics” data.

Benjamini & Hochberg (1995) realized that when testing 1000’s ofhypotheses a few errors could be tolerated.

False Discovery Rate (FDR) is the expected percentage of nullhypotheses among the statistically significant tests.

Table : Outcomes of m tests.

Not Significant TotalSignificant

True Null U V m0False Null T S m1

Total W R m

FDR =E(

VR |R > 0

)P(R > 0)

R: number of rejectionsV: number of false

rejections

Total W R m

FDR =E(

VR |R > 0

)P(R > 0)

rejections

Total W R m

FDR =E(

VR |R > 0

)P(R > 0)

rejections

Adaptive FDR

Total W R m

π0 = m0m

m: number of testsm0: number of null tests

Since we are trying to control false discoveries we do not need tocontrol for the truly non-null tests.Adaptive FDR methods use an estimate of π0 to improve thepower of the multiple comparisons adjustments.

Adaptive FDR

Total W R m

π0 = m0m

Adaptive FDR

Total W R m

π0 = m0m

Implementing FDR procedures

Compute a test statistic for each hypothesis.We might use the p-value as the test statistic.

Order the hypotheses from most to least significant, so that H0k hasthe k th significant test statistic.Estimate FDR(k) if we reject H01 · · ·H0k .Either

Pick a level q and reject H01 · · ·H0k if FDR(k)< q ORPick a p-value α and reject H0i if its p-value is less than α. Thenestimate FDR.

0 2000 4000 6000 8000 10000

BH Heuristic

Sorted Hypothesis Number

p−va

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

BH q=0.05Adaptive BH, q=0.05 pi0=.6reject at p<.02, q=0.133

Order the hypotheses from most to least significant, so that H0k hasthe k th significant test statistic.Estimate FDR(k) if we reject H01 · · ·H0k .

EitherPick a level q and reject H01 · · ·H0k if FDR(k)< q ORPick a p-value α and reject H0i if its p-value is less than α. Thenestimate FDR.

0 2000 4000 6000 8000 10000

BH Heuristic

p−va

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

Order the hypotheses from most to least significant, so that H0k hasthe k th significant test statistic.Estimate FDR(k) if we reject H01 · · ·H0k .Either

Pick a level q and reject H01 · · ·H0k if FDR(k)< q ORPick a p-value α and reject H0i if its p-value is less than α. Thenestimate FDR.

0 2000 4000 6000 8000 10000

BH Heuristic

p−va

●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

Estimating FDR

For discrete test statistics we want to:Estimate π0.Estimate FDR.

Discrete test statisticsarise from binary and count data such as

read counts in RNA-seq and ChIP-seqSNP studiesthresholding (above/below)multiple 2-way tables (e.g. surveys)

Estimating FDR

For discrete test statistics we want to:Estimate π0.Estimate FDR.

Discrete test statisticsarise from binary and count data such as

read counts in RNA-seq and ChIP-seqSNP studiesthresholding (above/below)multiple 2-way tables (e.g. surveys)

Why does discreteness matter?

0.0 0.3 0.6 0.9pValues

Nxn−null

Figure : Continuous p-values π0 = 0.8

0.00 0.25 0.50 0.75 1.00pValues

Nxn−null

Figure : Discrete p-values π0 = 0.8

The histogram on the left represents 10000 t-tests.The histogram on the right represents p-values from 10000 Fisher exacttests.

Why does discreteness matter?edgeR 3 samples/trt

p−values

0.0 0.2 0.4 0.6 0.8 1.0

p−values

0.0 0.2 0.4 0.6 0.8 1.0

P-values from an RNA-seq study of2 maize genotypes with 3 biologicalreplicates.

P-values from a microarray study inpoppy tissues with 4 biologicalreplicates.

Why does discreteness matter?

We will use the same heuristics for discrete and continuous tests. BUT

Table : Distribution of p-values.

Continuous Discretenull p-value distribution uniform depends on an ancillary

Prob(p=1) 0 >0percent of support points 0% 100%with positive probability

minimum achievable 0 >0p-value

Estimating π0

Estimating π0 from continuous p-values

Storey (2002) estimates height of flat part of histogram.Nettleton et al (2006) estimate the heights of the bins in excess ofexpected given π̂0.Pounds and Cheng (2004) assume all true non-nulls have p=0, so2 ∗ p̄ ≈ π0.

p−values

0.0 0.2 0.4 0.6 0.8 1.0

Estimating π0

Estimating π0 from discrete p-valuesThese methods seem less plausible since low power non-null testsmay have p-values far from 0.As well, both null and non-null tests have p-values with mass at 1,leading to a peak at p=1.We add 3 new methods.

edgeR 3 samples/trt

p−values

0.0 0.2 0.4 0.6 0.8 1.0

Estimating π0

Mixture distribution of p-valuesWe use the mixture distribution

f (p) = π0f0(p) + (1− π0)fA(p)

wheref is the distribution of the p-values,f0 is the distribution of p-values for the hypotheses that are truly nullandfA is the distribution of p-values for the hypotheses that are truly notnull.

Estimating π0

Estimating π0 from discrete p-values

There is often an ancillary statistic which determines the distributionof the test statistic. e.g. row totals.

If the ancillary statistic is known for each test, the distribution ofp-values under the null f0(p) is known.If there are many tests with the same value of the ancillary, f̂ (p) theempirical distribution of the p-values can be estimated by theobserved frequencies.

Estimating π0

There is often an ancillary statistic which determines the distributionof the test statistic. e.g. row totals.If the ancillary statistic is known for each test, the distribution ofp-values under the null f0(p) is known.

If there are many tests with the same value of the ancillary, f̂ (p) theempirical distribution of the p-values can be estimated by theobserved frequencies.

Estimating π0

There is often an ancillary statistic which determines the distributionof the test statistic. e.g. row totals.If the ancillary statistic is known for each test, the distribution ofp-values under the null f0(p) is known.If there are many tests with the same value of the ancillary, f̂ (p) theempirical distribution of the p-values can be estimated by theobserved frequencies.

Estimating π0 using f (p) = π0f0(p) + (1− π0)fA(p)

Regression Method

Useful when we have many tests with the same ancillary statistic.Regression method - regress empirical frequencies of p-valuesagainst expected frequencies under H0.The slope is approximately π0.

Estimating π0 using the histogram of p-values

0.00 0.25 0.50 0.75 1.00p

expected

observed

Histogram methodUsing the ancillary, we compute theexpected frequency of the p-valuesunder the null.We use the area A between theobserved histogram and the histogramexpected under the null.

A ≤ 2(1− π0)

π̂0 = 1− A2 has expectation at least as

big as π0.The method is sensitive to the choiceof bin boundaries.

0.00 0.25 0.50 0.75 1.00p

expected

observed

Histogram methodUsing the ancillary, we compute theexpected frequency of the p-valuesunder the null.We use the area A between theobserved histogram and the histogramexpected under the null.A ≤ 2(1− π0)

big as π0.

The method is sensitive to the choiceof bin boundaries.

0.00 0.25 0.50 0.75 1.00p

expected

observed

Histogram methodUsing the ancillary, we compute theexpected frequency of the p-valuesunder the null.We use the area A between theobserved histogram and the histogramexpected under the null.A ≤ 2(1− π0)

big as π0.The method is sensitive to the choiceof bin boundaries.

Estimating π0 by removing zero-power tests

Minimum achievable p-valueFor a given test statistic, there is some set ψ1 < · · · < ψk = 1 ofachievable p-values.ψ1 is the minimal achievable p-value for the test.Select a level α the maximum p-value at which to reject the nullhypothesis.If ψ1 > α then the test has zero power.

Estimating π0 by removing zero-power tests

Tarone (1990) noted that we need not consider tests with zeropower.We call a method a “T” method if we remove the zero-power testsand then proceed with a method for continuous data.“T” methods remove some of the excess mass at p = 1 and makethe histogram of p-values more uniform.In this talk, we use the Storey-T method.We remove the tests with zero power at α = 0.01 and then useStorey’s method on the remaining tests (right plot).

p−values pi0=0.9

p−values

0.0 0.2 0.4 0.6 0.8 1.0

p−values pi0=0.9 power>0

p−values

0.0 0.2 0.4 0.6 0.8 1.0

Simulated RNA-seq data

DataWe simulated RNA-seq data assuming:

m (number of tests) =1000 or 10,000.π0 = 0.1,0.2 · · · 0.8,0.9,0.95,1.0Two different discretized log-Normal distributions for totalreads/feature estimated from real data.Features are independent within sample.We used 2 treatments with no replication.The statistic was Fisher’s Exact Test.

0 200 400 600 800 1000

x value

Lognormal Parameters

(3,2)(4,2)

log-Normal distributions

Configuration % 0 or 1 total1 0.9%2 3.2%

Estimated π0 with m=10,000

0.0 0.2 0.4 0.6 0.8 1.0

Scenario 1, m=10000

● pi0HistogramRegressionNettletonPoundStoreyStorey−T

0.0 0.2 0.4 0.6 0.8 1.0

Scenario 2, m=10000

● pi0HistogramRegressionNettletonPoundStoreyStorey−T

logNormal(3,2)

(few small counts)

logNormal(4,2)

(many small counts)

Estimating FDR

Benjamini and Hochberg (1995) suggest an algorithm for controllingFDR at level q:

Find the maximal i such that p(i) ≤ i×qm .

It has been shown when the test statistics are continuous andindependent, then this algorithm controls the FDR at level π0q

The BH method is known to be conservative with discrete tests.

Estimating FDR

Benjamini and Hochberg (1995) suggest an algorithm for controllingFDR at level q:

Find the maximal i such that p(i) ≤ i×qm .

It has been shown when the test statistics are continuous andindependent, then this algorithm controls the FDR at level π0q

The BH method is known to be conservative with discrete tests.

Estimating FDR

Adaptive FDR methods control the FDR at approximate level qusing an estimate of π0.For example, the adaptive Benjamini and Hochberg method is

Find the maximal i such that p(i) ≤ i×qmπ̂0

When the test statistics are continuous and independent, then thisalgorithm controls the FDR at approximately level q

Estimating FDR

Adaptive FDR methods control the FDR at approximate level qusing an estimate of π0.For example, the adaptive Benjamini and Hochberg method is

Find the maximal i such that p(i) ≤ i×qmπ̂0

When the test statistics are continuous and independent, then thisalgorithm controls the FDR at approximately level q

Estimating FDR

Gilbert (2005) uses Tarone’s idea of removing tests which have zeropower to achieve significance at level α.Gilbert filters zero power tests then applies the BH method to theremaining mF tests.

We suggest an adaptive Gilbert method that uses an estimate of π0with Gilbert’s method.

Estimating FDR

Gilbert (2005) uses Tarone’s idea of removing tests which have zeropower to achieve significance at level α.Gilbert filters zero power tests then applies the BH method to theremaining mF tests.We suggest an adaptive Gilbert method that uses an estimate of π0with Gilbert’s method.

Simulation Results

Using the same simulation scenario as before we implementedBenjamini and Hochberg’s 1995 methodGilbert’s (2005) method using α = 0.01Adaptive versions of BH and Gilbert using

true π0estimated π0 using the Storey-T method

We considered error rates forfalse detectionfalse nondetection (including nondetection due to zero power)total errors

Simulation Results

Results m=10,000 few small margins

0.0 0.2 0.4 0.6 0.8 1.0

Scenario 1, m=10000 Total Rejections

● NonNullBHBH−TrueBH−TGilbertGilbert−TrueGilbert−T

0.0 0.2 0.4 0.6 0.8 1.0

Scenario 1, m=10000 False Rejections

BHBH−TrueBH−TGilbertGilbert−TrueGilbert−T

0.0 0.2 0.4 0.6 0.8 1.0

Scenario 1, m=10000 E(V)/E(R)

0.0 0.2 0.4 0.6 0.8 1.0

Scenario 1, m=10000 Total Errors

0.0 0.2 0.4 0.6 0.8 1.0

Results m=10,000 many small margins●

0.0 0.2 0.4 0.6 0.8 1.0

Does it matter?

0.0 0.2 0.4 0.6 0.8 1.0

Scenario 1, m=10000 Difference in Total Errors

BH−TrueBH−TGilbertGilbert−TrueGilbert−T

●●

●●●

●●●●

●●

●●●

●●

BH BHT BHTr G GT GTr

Scenario 1, m=10000, pi0=0.7

0.0 0.2 0.4 0.6 0.8 1.0

Scenario 2, m=10000 Difference in Total Errors

BH−TrueBH−TGilbertGilbert−TrueGilbert−T

●●

●●●

●●

●●●

●●

BH BHT BHTr G GT GTr

Scenario 2, m=10000, pi0=0.7

Blekhman Primate Liver Data

Blekhman, et al, (2010) used RNA-seq to interrogate liver samples inmale and female human, chimpanzee and rhesus monkey.

There were 20689 features but 2803 had no reads, and a further907 had only 1 read across the 18 samples.These 3710 features were removed, leaving 16979 features.There were 3 biological samples for each species by gendercombination.Each sample was divided into 2 sequencing lanes.

The 2 lanes were combined to attain total reads for each feature foreach biological sample.

Blekhman, et al, (2010) used RNA-seq to interrogate liver samples inmale and female human, chimpanzee and rhesus monkey.

There were 20689 features but 2803 had no reads, and a further907 had only 1 read across the 18 samples.These 3710 features were removed, leaving 16979 features.There were 3 biological samples for each species by gendercombination.Each sample was divided into 2 sequencing lanes.The 2 lanes were combined to attain total reads for each feature foreach biological sample.

We look at 2 comparisons:Comparison Test

2 lanes same human Fisher’s exact testmale human versus chimpanzee moderated Negative Binomial test

It is difficult to compute expected counts for the moderated NegativeBinomial test, so we use the T-method.We use Fisher’s exact test to estimate the minimal achievablep-value. It is conservative.Data are normalized using the TMM method.Analysis is done using edgeR in Bioconductor.

We look at 2 comparisons:Comparison Test

2 lanes same human Fisher’s exact testmale human versus chimpanzee moderated Negative Binomial test

It is difficult to compute expected counts for the moderated NegativeBinomial test, so we use the T-method.We use Fisher’s exact test to estimate the minimal achievablep-value. It is conservative.Data are normalized using the TMM method.Analysis is done using edgeR in Bioconductor.

Human Male 1

We compared the two lanes of sequencing data for Human male 1.

13553 features were detected by at least 1 read.10359 features were detected by at least 7 reads (giving minimumachievable p-value>0.01.)We do not expect any differences between the two lanes.

0 5 10 15

Log2(Lane 1+.5)

Lanes 1 and 2 of Human Male 1

124165206247288329370411452493534575616657

Counts

Human Male 1

HS Male 1 p−values

All p−values

0.0 0.2 0.4 0.6 0.8 1.0

HS Male 1 Filtered p−values

Filtered p−values

0.0 0.2 0.4 0.6 0.8 1.0

π̂0 = 1.0 using both Storey’s method and the Storey-T method.Method Number Significant FDR<0.05

Benjamini & Hochberg 3Gilbert 5

Human Males versus Chimpanzee Males

The data were normalized, and the dispersion shrinkage factorswere computed.There are 3 biological replicates of each.16375 features were detected with at least 1 read.13809 features were detected with at least 7 reads.

HS Male Vs Chimp Male

All p−values

0.0 0.2 0.4 0.6 0.8 1.0

HS Male Vs Chimp Male

Filtered p−values

0.0 0.2 0.4 0.6 0.8 1.0

Human Males versus Chimpanzee Males

π̂0 = 1.0 using Storey’s method and 0.87 using the Storey-T method.

Method Number Significant Number SignificantNon-adaptive Adaptive

Benjamini & Hochberg 1166 1239Gilbert 1251 1325

Note: 1166/16375 = 7.1% so π0 = 1 is not reasonable.

Summary:

What did we learn?For tabular count data (e.g. RNA-seq, SNP)

It is important to have an estimate of π0.When most counts are big, remove features with small margins anduse methods for continuous data.When many counts are small, use the regression method.

Adaptive methodsdo not help much for π0 > 0.9.can significantly reduce total errors when π0 < 0.5

Gilbert’s method is preferable to vanilla BH.Happily Gilbert’s method is equivalent to removing features withsmall margins and then using BH.

SummaryFor RNA-seq and SNP data, remove features with small margins andproceed as if p-values were continuous.

Summary:

Gilbert’s method is preferable to vanilla BH.

Happily Gilbert’s method is equivalent to removing features withsmall margins and then using BH.

Summary:

Many thanks

Thanks for your attention

Thanks to NSFNSF DMS 1007801 (Altman, PI)NSF IOS 0820729 (Altman, subcontract from McSteen, PI)

Main Reference:Dialsingh, I (2011) False Discovery Rates when the Statistics areDiscrete. PhD Dissertation, Dept. of Statistics, Penn StateUniversity

Many thanks

Thanks for your attention

Thanks to NSFNSF DMS 1007801 (Altman, PI)NSF IOS 0820729 (Altman, subcontract from McSteen, PI)

Main Reference:Dialsingh, I (2011) False Discovery Rates when the Statistics areDiscrete. PhD Dissertation, Dept. of Statistics, Penn StateUniversity

References

Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: Apractical and powerful approach to multiple testing. Journal of the Royal StatisticalSociety Series B, 57, 289-300.

Benjamini, Y., Hochberg, Y. (2000). On the adaptive control of the false discoveryrate in multiple testing with independent statistics. Journal of BehavioralEducational Statistics, 25, 60-83.

Gilbert,P.B. (2005). A modified false discovery rate multiple comparisonsprocedure for discrete data, applied to human immunodeficiency virus genetics.Journal of Applied Statistics, 54, 143-158.

Nettleton, D., Hwang, J.T.G., Caldo, R.A., Wise, R.P. (2006). Estimating thenumber of true null hypotheses from a histogram of p-values. Journal ofAgricultural, Biological, and Environmental Statistics, 11, 337-356.

Pounds, S. and Cheng, C. (2004). Improving false discovery rate estimation.Bioinformatics, 20, 1737-1745.

Storey, J.D. (2003) The positive false discovery rate: A Bayesian interpretation andthe q-value. Annals of Statistics. 31, 2013-2035.

Tarone,R.E. (1990) A modified Bonferroni method for discrete data. Biometrics.46, 515-522.

Adaptive FDR Estimation for High Dimensional Discrete Data

Documents

Aerodynamics FDR

Adaptive Control of Discrete-Time Systems with Unknown

FDR-1000AWS FDR-2000AWS FDR-3000AWS CR ......FDR-1000AWS, FDR-2000AWS, FDR-3000AWS, CR-IR363AWS (Standard) DICOM Conformance Statement 897N100760V 7 Table 4.2- 64 DICOM Application

An Adaptive Pruning Algorithm for the Discrete L ... - DTU

Optimal Adaptive Control of Nonlinear Discrete-time …ece.mst.edu/media/academic/ece/documents/c/1/CDC_workshop_2011...Optimal Adaptive Control of Nonlinear Discrete-time S stemstime

Single Observation Adaptive Search for Discrete and

DICOM Conformance Statement ***** FDR-1000AWS FDR … · FDR-1000AWS, FDR-2000AWS, FDR-3000AWS, CR-IR363AWS (Standard) DICOM Conformance Statement 897N100760U 7 Table 4.2- 64 DICOM

Optimal Adaptive Control 06 RL short course SEU/RL... · 2019. 6. 21. · 2.5 Optimal adaptive control for discrete-time systems 32 2.5.1 Policy iteration and value iteration for

FDR Report

read.pudn.comread.pudn.com/downloads165/ebook/753063/Adaptive... · Alexander D. Poularikas ADAPTIVE FILTERING PRIMER ... authors introduce discrete-time signal processing, random

Two Dimensional Local Adaptive Discrete Velocity Grids For … · 2017-01-26 · Two Dimensional Local Adaptive Discrete Velocity Grids For Rareﬁed Flow Simulations S. Brull 1,

Multiple testing with discrete data: proportion of true ... · Multiple testing with discrete data: proportion of true null hypotheses and two adaptive FDR procedures Xiongzhi Chen

FDR up to two loops - DESY · 2020. 12. 11. · Roberto Pittau (UniversityofGranada) Desy-Zeuthen, June 16th 2016 RobertoPittau, U.of Granada FDRuptotwo loops. FDR FDR&QCD FDR&IR

An Adaptive Finite Element Eigenvalue Solver of Quasi ... · The corresponding discrete eigenvalue problem reads: ... Adaptive Finite Element Eigenvalue Solver The adaptive nite element

Review: FDR

A Discrete Cosine Adaptive Harmonic Wavelet Packet and Its … · like signal denoising which reduces glitches in the recon-structed signals [11]. In this paper, an adaptive Harmonic

FDR Depression

The Mesh Adaptive Direct Search Algorithm for Discrete Blackbox

EE359 Discussion Session 7 Adaptive Modulation, MIMO systems · Outline 1 Adaptive transmit schemes Variable rate variable power Discrete rate variable power Channel estimation errors

Discrete Model Matching Adaptive Control for Potentially Inversely Non-Stable Continuous- Time Plants by Using Multirate Sampling