?W H A T I S . . .

Data MiningMauro Maggioni

Data collected from a variety of sources has beenaccumulating rapidly. Many fields of science havegone from being data-starved to being data-richand needing to learn how to cope with large datasets. The rising tide of data also directly affectsour daily lives, in which computers surrounding ususe data-crunching algorithms to help us in tasksranging from finding the quickest route to ourdestination considering current traffic conditionsto automatically tagging our faces in pictures; fromupdating in near real time the prices of sale itemsto suggesting the next movie we might want towatch.

The general aim of data mining is to find usefuland interpretable patterns in data. The term canencompass many diverse methods and thereforemeans different things to different people. Here wediscuss some aspects of data mining potentially ofinterest to a broad audience of mathematicians.

Assume a sample data point xi (e.g., a picture)may be cast in the form of a long vector of numbers(e.g., the pixel intensities in an image): we representit as a point in RD. Two types of related goalsexist. One is to detect patterns in this set of points,and the other is to predict a function on the data:given a training set (xi , f (xi))i , we want to predictf at points outside the training set. In the case oftext documents or webpages, we might want toautomatically label each document as belongingto an area of research; in the case of pictures, wemight want to recognize faces; when suggesting thenext movie to watch given past ratings of moviesby a viewer, f consists of ratings of unseen movies.

Mauro Maggioni is assistant professor of mathematics andcomputer science at Duke University. His email address ismauro.maggioni@duke.edu.

DOI: http://dx.doi.org/10.1090/noti831

Typically, xi is noisy (e.g., noisy pixel values), andso is f (xi) (e.g., mislabeled samples in the trainingset).

Of course mathematicians have long concernedthemselves with high-dimensional problems. Oneexample is studying solutions of PDEs as func-tions in infinite-dimensional function spaces andperforming efficient computations by projectingthe problem onto low-dimensional subspaces (viadiscretizations, finite elements, or operator com-pression) so that the reduced problem may benumerically solved on a computer. In the caseof solutions of a PDE, the model for the datais specified: a lot of information about the PDEis known, and that information is exploited topredict the properties of the data and to constructlow-dimensional projections. For the digital datadiscussed above, however, typically we have littleinformation and poor models. We may start withcrude models, measure their fitness to the data andpredictive ability, and, those being not satisfactory,improve the models. This is one of the key pro-cesses in statistical modeling and data mining. Itis not unlike what an applied mathematician doeswhen modeling a complex physical system: he maystart with simplifying assumptions to construct a“tractable” model, derive consequences of such amodel (e.g., properties of the solutions) analyticallyand/or with simulations, and compare the resultsto the properties exhibited by the real-world phys-ical system. New measurements and real-worldsimulations may be performed, and the fitnessof the model reassessed and improved as neededfor the next round of validation. While physicsdrives the modeling in applied mathematics, anew type of intuition, built on experiences in theworld of high-dimensional data sets rather thanin the world of physics, drives the intuition of the

532 Notices of the AMS Volume 59, Number 4

mathematician set to analyze high-dimensionaldata sets, where “tractable” models are geomet-ric or statistical models with a small number ofparameters.

One of the reasons for focusing on reduc-ing the dimension is to enable computations,but a fundamental motivation is the so-calledcurse of dimensionality. One of its manifestationsarises in the approximation of a 1-Lipschitz func-tion on the unit cube, f : [0,1]D → R satisfying|f (x) − f (y)| ≤ ||x − y|| for x, y ∈ [0,1]D. Toachieve uniform error ε, given samples (xi , f (xi)),in general one needs at least one sample in eachcube of side ε, for a total of ε−D samples, which istoo large even for, say, ε = 10−1 and D = 100 (arather small dimension in applications). A commonassumption is that either the samples xi lie ona low-dimensional subset of [0,1]D and/or f isnot simply Lipschitz but has a smoothness thatis suitably large, depending on D (see referencesin [3]). Taking the former route, one assumesthat the data lies on a low-dimensional subset inthe high-dimensional ambient space, such as alow-dimensional hyperplane or unions thereof, orlow-dimensional manifolds or rougher sets. Re-search problems require ideas from different areasof mathematics, including geometry, geometricmeasure theory, topology, and graph theory, withtheir tools for studying manifolds or rougher sets;probability and geometric functional analysis forstudying random samples and measures in highdimensions; harmonic analysis and approximationtheory, with their ideas of multiscale analysis andfunction approximation; and numerical analysis,because we need efficient algorithms to analyzereal-world data.

As a concrete example, consider the followingconstruction. Given n points {xi}ni=1 ⊂ RD and ε >0, construct Wij = exp(− ||xi−xj ||

2

2ε ), Dii =∑j Wij ,

and the Laplacian matrix L = I − D− 12WD−

12 on

the weighted graph G with vertices {xi} and edgesweighted by W . When xi is sampled from a man-ifold M and n tends to infinity, L approximates(in a suitable sense) the Laplace-Beltrami operatoron M [2], which is a completely intrinsic object.The random walk on G, with transition matrixP = D−1W , approximates Brownian motion onM.Consider, for a time t > 0, the so-called diffusiondistance dt(x, y) := ||P t(x, ·)− P t(y, ·)||L2(G) (see[2]). This distance is particularly useful for cap-turing clusters/groupings in the data, which areregions of fast diffusion connected by bottlenecksthat slow diffusion. Let 1 = λ0 ≥ λ1 ≥ · · · bethe eigenvalues of P and ϕi be the correspond-ing eigenvectors (ϕ0, when G is a web graph, isrelated to Google’s pagerank). Consider a diffu-sion map Φtd that embeds the graph in Euclidean

−0.05

0

0.05

−0.03−0.02

−0.010

0.010.02

−0.04

−0.02

0

0.02

0.04

−3

−2

−1

0

1

2

3

φ

ψ φ

−0.15 −0.1 −0.05 0 0.05

−0.50

0.5

−0.1

−0.05

0

0.05

0.1

0.15

45

6AnthropologyAstronomySocial SciencesEarth SciencesBiologyMathematicsMedicinePhysics

Figure 1. Top: Diffusion map embedding of theset of configurations of a small biomolecule(alanine dipeptide) from its 36-dimensional statespace. The color is one of the dihedral anglesϕ,ψϕ,ψϕ,ψ of the molecule, known to be essential tothe dynamics [4]. This is a physical systemwhere (approximate) equations of motion areknown, but their structure is too complicatedand the state space too high-dimensional to beamenable to analysis. Bottom: Diffusion map of adata set consisting of 1161 Science Newsarticles, each modeled by a 1153-dimensionalvector of word frequencies, embedded in alow-dimensional space with diffusion maps, asdescribed in the text and in [2].

space, where Φtd(x) := (√λt1ϕ1(x), . . . ,

√λtdϕd(x)),

for some t > 0 [2]. One can show that the Euclideandistance between Φtd(x) and Φtd(y) approximatesdt(x, y), the diffusion distance at time scale tbetween x and y on the graph G.

April 2012 Notices of the AMS 533

David Blackwell

Memorial Conference

Join us on April 19–20, 2012, at Howard University in Washington, DC for a special conference honoring David Blackwell (1919–2010), former President of the Institute of Mathematical Statistics and the fi rst African-American to be inducted into the National Academy of Sciences.

Th is conference will bring together a diverse group of leading theoretical and applied statisticians and mathematicians to discuss advances in mathematics and statistics that are related to, and in many cases have grown out of, the work of David Blackwell. Th ese include developments in dynamic programming, information theory, game theory, design of experiments, renewal theory, and other fi elds. Other speakers will discuss Blackwell’s legacy for the community of African-American researchers in the mathematical sciences.

Th e conference is being organized by the Department of Mathematics at Howard University in collaboration with the University of California, Berkeley, Carnegie Mellon University, and the American Statistical Association. Funding is being provided by the National Science Foundation and the Army Research Offi ce. To learn more about the program, invited speakers, and registration, visit: https://sites.google.com/site/conferenceblackwell.

April 19–20, 2012Howard University Washington, DC

Phot

ogra

ph b

y Lo

tfi A

. Zad

eh.

In Figure 1 we apply this technique to twocompletely different data sets. The first one is a setof configurations of a small peptide, obtained by amolecular dynamics simulation: a point xi ∈ R12×3

contains the coordinates in R3 of the 12 atomsin the alanine dipeptide molecule (representedas an inset in Figure 1). The forces between theatoms in the molecule constrain the trajectoriesto lie close to low-dimensional sets in the 36-dimensional state space. In Figure 1 we apply theconstruction above1 and represent the diffusionmap embedding of the configurations collected [4].The second one is a set of text documents (articlesfrom Science News), each represented as a R1153

vector whose kth coordinate is the frequency of thekth word in a 1153-word dictionary. The diffusionembedding in low dimensions reveals even lower-dimensional geometric structures, which turn outto be useful for understanding the dynamics ofthe peptide considered in the first data set and forautomatically clustering documents by topic in thecase of the second data set. Ideas from probability(random samples), harmonic analysis (Laplacian),and geometry (manifolds) come together in thesetypes of constructions.

This is only the beginning of one of many re-search avenues explored in the last few years. Manyother exciting opportunities exist, for example thestudy of stochastic dynamic networks, where asample is a network and multiple samples arecollected in time: quantifying and modeling changerequires introducing sensible and robust metricsbetween graphs.

Further reading: [5, 3, 1] and the referencestherein.

References1. Science: Special issue: Dealing with data, February 2011,

pp. 639–806.2. R. R. Coifman, S. Lafon, A. B. Lee, M. Maggioni,

B. Nadler, F. Warner, and S. W. Zucker, Geometricdiffusions as a tool for harmonic analysis and structuredefinition of data: Diffusion maps, Proc. Natl. Acad. Sci.USA 102 (2005), no. 21, 7426–7431.

3. D. Donoho, High-dimensional data analysis: The cursesand blessings of dimensionality, “Math Challenges of the21st Century”, AMS, 2000.

4. M. A. Rohrdanz, W. Zheng, M. Maggioni, andC. Clementi, Determination of reaction coordinates vialocally scaled diffusion map, J. Chem. Phys. 134 (2011),124116.

5. J. W. Tukey, The Future of Data Analysis, Ann. Math.Statist. 33, Number 1 (1962), 1–67.

1We use here a slightly different definition of the weightmatrix W , which uses distances between molecular con-figurations up to rigid affine transformations, instead ofEuclidean distances.

534 Notices of the AMS Volume 59, Number 4