Upload
arthur-breitman
View
442
Download
4
Embed Size (px)
Citation preview
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Density estimation Typical approaches Hardy Haar
Hardy HaarNon-linear density estimation using a sparse Haar prior
Arthur Breitman
April 3, 2016
Arthur Breitman
Hardy Haar
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Density estimation Typical approaches Hardy Haar
Table of contents
Density estimationProblem statementApplications
Typical approachesParametric densityKernel density estimation
Hardy HaarPrinciplesRecipeLinear transformsHow to use for data mining
Arthur Breitman
Hardy Haar
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Density estimation Typical approaches Hardy Haar
Problem statement
What is density estimation?
▶ Given an i.i.d sample xn1 from unknown distribution P,estimate P(x) for arbitrary x
▶ For instance P belongs to some parametric family, but nonBayesian treatment possible
▶ Examples:▶ Model P is a multivariate gaussian with unknown mean and
covariance.▶ Kernel density estimation is non paremetric (but morally ∼ to
P as a uniform mixture of n distributions, fit with maximumlikelihood)
Arthur Breitman
Hardy Haar
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Density estimation Typical approaches Hardy Haar
Applications
Why is density estimation useful?
▶ Allows unsupervized learning by recovering the latentparameters of a distribution describing the data
▶ But can also be used for supervized learning.
▶ Learning P(x , y) is more general than learning y = f (x).
▶ For instance, to minimize quadratic error, use
f (x) =
∫y P(x , y) dx∫P(x , y) dx
▶ Knowledge of the full density permits the use of any∗ lossfunction
∗offer void under fat tailsArthur Breitman
Hardy Haar
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Density estimation Typical approaches Hardy Haar
Applications
Mutual information
Of particular is the ability to compute “mutual information”.Mutual information is the principled way to measure what isloosely referred to as “correlation”
I (X ;Y ) =
∫Y
∫Xp(x , y) log
(p(x , y)
p(x) p(y)
)dx dy
Measures the amount of information one variable gives us aboutanother.
Arthur Breitman
Hardy Haar
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Density estimation Typical approaches Hardy Haar
Applications
Correlation doesn’t always capture this relation
In the case of a bivariate gaussian,
I = −1
2log
(1− ρ2
)We can get a correlation equivalent by using
ρ̂ =√
1− e−2I
Arthur Breitman
Hardy Haar
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Density estimation Typical approaches Hardy Haar
Bivariate normal
Assume that the data observed was drawn from a bivariate normaldistribution
▶ Latent parameters: mean and covariance matrix
▶ Unsupervized view: learn the relationship between tworandom variables (mean, variance, correlation)
▶ Supervized view: equivalent to simple linear regression
Arthur Breitman
Hardy Haar
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Density estimation Typical approaches Hardy Haar
Parametric density
Bivariate normal density
Arthur Breitman
Hardy Haar
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Density estimation Typical approaches Hardy Haar
Parametric density
Kernel density estimation
Kernel density estimation is a non-parametric density estimatordefined as
f̂h(x) =1
nh
n∑i=1
K(x − xi
h
)▶ K is a non negative function that integrates to 1 and has
mean 0 (typically gaussian)
▶ h is the scale or bandwith.
Arthur Breitman
Hardy Haar
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Density estimation Typical approaches Hardy Haar
Kernel density estimation
Gaussian kernel density estimation
Arthur Breitman
Hardy Haar
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Density estimation Typical approaches Hardy Haar
Kernel density estimation
Bandwith selection
Picking h can be tricky
▶ h too small =⇒ overfit the data
▶ h too large =⇒ underfit the data
▶ There are rules of thumbs to pick h from variance of data andnumber of points
▶ Can be picked by cross-validation
Arthur Breitman
Hardy Haar
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Density estimation Typical approaches Hardy Haar
Kernel density estimation
Gaussian kernel density estimationUnder and over-fitting with different bandwiths (the correlation ofthe kernel is estimated from the data)
Arthur Breitman
Hardy Haar
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Density estimation Typical approaches Hardy Haar
Kernel density estimation
Issues with kernel density estimation
Naive Kernel density estimation has several drawbacks
▶ Kernel covariance is fixed for the entire space
▶ Does not adjust bandwith to local density
▶ No distributed representation =⇒ poor generalization
▶ Performs poorly in high dimensions
Arthur Breitman
Hardy Haar
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Density estimation Typical approaches Hardy Haar
Kernel density estimation
Adaptive kernel density estimation
One approach is to use a different kernel for every point, varyingthe scale based on local features.Ballon estimators make the kernel width inversely proportional todensity at the test point
h =k
(nP(x))1/D
Pointwise estimator try to vary the kernel at each sample point
Arthur Breitman
Hardy Haar
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Density estimation Typical approaches Hardy Haar
Kernel density estimation
Local bandwith choice
▶ If latent distribution has peaks, tradeoff between accuracyaround the peaks and in regions of low density.
▶ This is reminiscent of time-frequency tradeoffs in fourieranalysis (hint: h is called bandwith)
▶ Suggests using wavelets which have good localization in timeand frequency
Arthur Breitman
Hardy Haar
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Density estimation Typical approaches Hardy Haar
Principles
Introducing Haardy Haar
Hardy Haar attempts to address some of these shortcomings
▶ Full Bayesian treatment of density estimation
▶ (Somewhat) distributed representation
▶ Fast!
Arthur Breitman
Hardy Haar
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Density estimation Typical approaches Hardy Haar
Principles
Examples of Haardy Haar
Arthur Breitman
Hardy Haar
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Density estimation Typical approaches Hardy Haar
Principles
Goals
Coming up with a prior?
▶ In principle, any distribution whose support contains thesample is potential.
▶ Some distributions are more likely than others, but why notjust take the empirical distribution?
▶ May work fine for integration problems for instance▶ Doesn’t help with regression or to understand the data
▶ There should be some sort of spatial coherence to thedistribution
Arthur Breitman
Hardy Haar
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Density estimation Typical approaches Hardy Haar
Principles
Sparse wavelet decomposition prior
To express the spatial coherence constraint, we take a L0 sparsityprior over the coefficient of the decomposition of the PDF in asuitable wavelet basis.
▶ This creates a coefficient ”budget” to describe the distribution
▶ Large scale wavelet describe coarse features of the distribution
▶ Sparse areas can be described with few coefficients
▶ Areas with a lot of sample points are described in more detail
▶ Closely adheres to the minimum description length principle
Arthur Breitman
Hardy Haar
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Density estimation Typical approaches Hardy Haar
Principles
Haar basis
The Haar wavelet is the simplest wavelet
Not very smooth, but no overlap between wavelets at the samescale =⇒ tractability
Arthur Breitman
Hardy Haar
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Density estimation Typical approaches Hardy Haar
Principles
Model
▶ The minimum length principle suggests penalizing the loglikelihood of producing the sample with the number of nonzero coefficients.
▶ We can put a weight on the penalty, which will enforce moreor less sparsity
▶ We can “cheat” with an improper prior: use an infinitenumber of coefficient, but favor models with many zeros
Arthur Breitman
Hardy Haar
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Density estimation Typical approaches Hardy Haar
Recipe
Sampling
To sample from this distribution over distributions, conditional onthe observed sample, we interpret the data as generated by arecursive generative model.As n is held fixed, the number of datapoints in each orthant isdescribed by a multinomial distribution. We put a non-informativedirichlet prior on the probabilities of each quadrant. This processusis repeated for each orthant.
Arthur Breitman
Hardy Haar
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Density estimation Typical approaches Hardy Haar
Recipe
Orthant treePlace the data points in an orthant tree. The structure is built intime O(n log n)
Arthur Breitman
Hardy Haar
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Density estimation Typical approaches Hardy Haar
Recipe
Probabily of each orthant
Conditional on the number of datapoints falling in each orthant,the distribution of the probability mass over each orthant is givenby the Dirichlet distribution:
Γ(∑2d
i=1 ni)∏2d
i=1 Γ(1 + ni )
2d∏i=1
pnii
d is the dimension of the space, thus there are 2d orthants.
Arthur Breitman
Hardy Haar
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Density estimation Typical approaches Hardy Haar
Recipe
What about our prior?Having a zero coefficient in the Haar decomposition translate itselfin certain symmetries. In two dimension there are eight cases
▶ 1 non-zero coeffs: each quadrant has 14 of the mass
▶ 2 non-zero coeff▶ Left vs. right, top and bottom weight independent of side▶ Top vs. bottom▶ Diagonal vs. other diagonal
▶ 3 non-zero coeffs▶ Shared equally between left and right, but each side has its
own distribution between top and bottom▶ Same for top and bottom▶ Same for diagonals
▶ 4 non-zero coeffs: each quadrant is independent
N.B. probabilities must sum to 1Arthur Breitman
Hardy Haar
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Density estimation Typical approaches Hardy Haar
Recipe
Single point example
Arthur Breitman
Hardy Haar
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Density estimation Typical approaches Hardy Haar
Recipe
Marginalizing
▶ The distribution of weight over orthants is independentbetween the “levels” of the tree
▶ We can marginalize to efficiently compute the mean density ateach point.
▶ The cost is then O(2dn log n)
Arthur Breitman
Hardy Haar
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Density estimation Typical approaches Hardy Haar
Linear transforms
Orthant artifacts
▶ The model converges to the true distribution
▶ But the choice of orthants is arbitrary
▶ Introduces unnecessary variance
▶ We’d like to remove that sensitivity
Solution: integrate over all affine transforms of the data
Arthur Breitman
Hardy Haar
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Density estimation Typical approaches Hardy Haar
Linear transforms
Integrating over linear transforms
Fortunately, the Haar model gives us the evidence
▶ Assume the data comes from a gaussian Copula
▶ Adjust by the Jacobian of the transform▶ To sample from linear distributions
▶ perform PCA▶ translate by ( ux√
n, uy√
n)
▶ rotate randomly▶ scale variances by 1√
2n
▶ ... then weight by evidence from the model
Arthur Breitman
Hardy Haar
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Density estimation Typical approaches Hardy Haar
How to use for data mining
Application to data-mining
▶ select relevant variables to use as regressors
▶ evaluate the quality of hand-crafted features
▶ explore unknown relationships in the data
▶ in time series, mutual information between time and datadetects non stationarity
Arthur Breitman
Hardy Haar