Workflows for visualizing data mining results

Finn Arup Nielsen

Center for Integrated Molecular Brain Imaging

DTU Informatics, Technical University of Denmark

Neurobiology Research Unit, Copenhagen University Hospital, Rigshospitalet

April 1, 2009

Abstract

I review methods and tools for visualizations in neu-roimaging as well as point to some of the problemswith interpreting such visualizations. I furthermoredescribe Internet-based visualization and informa-tion visualization. I focus on workflows associatedwith the Brede Toolbox for display of data miningresults. A few neuroinformatics databases now ex-ist that record results from neuroimaging studies inthe form of brain coordinates in stereotaxic space.The Brede Toolbox was originally developed to ex-tract, analyze and visualize data from one of them— the BrainMap database. Since then the BredeToolbox has expanded and now includes its owndatabase with coordinates along with ontologies forbrain regions and functions: The Brede Database.With Brede Toolbox and Database combined wesetup automated workflows for extraction of data,mass meta-analytic data mining and visualizations.Most of the Web presence of the Brede Database isestablished by a single script executing a workflowinvolving these steps together with a final gener-ation of Web pages with embedded visualizationsand links to interactive three-dimensional modelsin the Virtual Reality Modeling Language.

1 Introduction

In a narrow sense neuroimaging workflows involveneuroimaging image processing and analysis. In amore broader sense the workflow in a neuroimagingstudy involves a number of other processes: gath-ering information, designing the experiment, brainscanning, interpretation of the study, relating it toother studies and communicating the study. Datamining in neuroimaging may not only be applied asthe standard neuroimaging analysis but also set towork on other components in workflow, and visual-ization of the data mining results may help the in-dividual researcher in understanding his or her data

as well as in communication with other researchers.

A number of tools exists for visualizing neu-roimaging data mining results when the result is avolumetric neuroimage. There are, however, alsovisualization tools for other aspects of the neu-roimaging process, and one example is our BredeToolbox (Nielsen and Hansen, 2000a). Starting outas a program for handling and visualization of datafrom the BrainMap database of Fox et al. (1994)the Brede Toolbox now includes its own databaseof results from neuroimaging—the Brede Database(Nielsen, 2003)—as well as analysis and visualiza-tion functions for a range of tasks. Below I willdescribe some of the tools for neuroimaging visual-ization, and as our experience has been mostly withthe Brede Database and Toolbox I will take a par-ticular focus on workflows and visualizations withthese tools. We have setup an automated work-flow involving a few non-interactive batch scriptsthat construct practically the entire Web presenceof the Brede Database with static Web pages andvisualizations. Furthermore, automated workflowsusing the ontologies of the Brede Database can per-form mass meta-analysis across brain functions orbrain regions (Nielsen, 2005; Nielsen et al., 2006a).

2 Visualization tools

The abundance of tools for visualization as wellas for other aspects of the neuroimaging processhas spawned an interest in generating overviewsfor these tools, and now there exist several Web-based directories: Neuroscience Database Gate-way (NDG) (Gardner and Shepherd, 2004), Neu-roscience Information Framework (NIF) (Gard-ner et al., 2008), Neuroimaging Informatics Toolsand Resources Clearinghouse (NITRC) (Bucci-grossi et al., 2008), I Do Imaging and InternetAnalysis Tools Registry (IATR), see also (Dinovet al., 2008). Some of these have an API sothat HTML or XML for a specific tool can be re-

1

quested. The NIF resource may be downloaded asan XML file. NITRC, IATR and I Do Imaginghave Web 2.0 components with user-provided toolratings and NITRC has an associated wiki. Since2001 I have updated the Bibliography on Neuroin-formatics which also lists numerous tools. RecentlyI began the Brede Wiki with structured informationabout neuroscience including neuroimaging visual-ization tools. Anyone can ‘micro-publish’ relevantinformation, and the structured content allows foroff-wiki database queries (Nielsen, 2009).

Among the visualization tools are, e.g., MRIcron(Rorden and Brett, 2000), MindSeer (Moore et al.,2007), iiV (Lee et al., 2008), DataViewer3D (Gouwset al., 2009), MIPAV (Bazin et al., 2007). Sev-eral workflow management systems exists (Fissell,2007) and they often have associated visualizationprograms for the neuroimaging data. Presently NI-TRC has 35 tools listed under visualization, andIATR returns 48 on a query on visualization.

3 Visualizing neuroimages

In the published literature the most used visualiza-tion is the display of summary statistics. For ex-ample, in the NeuroImage issue volume 21 number1 of May 2006, practically all articles display func-tional neuroimaging data mining results as sliceswith a thresholded and color-coded summary statis-tics image (e.g., the t-value map from statisticalparametric mapping) on top of a grey-scale back-ground of an anatomical magnetic resonance im-age. There are a few other types of visualizationpresent: Projection of the activations on the corti-cal surface visualized in three dimensions (3D), ei-ther with the method implemented in the SPM tool(Frackowiak et al., 2003) or with the more elabo-rate approach involving cortical surface extraction,that may be further turned into two dimensional(2D) flat-maps where the sulcal and gyral curva-ture is indicated with a gray-scale, for a recent ex-ample see Van Essen and Dierker (2007). This typeof visualization is also used for results on corticalthickness analysis. Other types are ‘glass-brain’ vi-sualizations with maximum intensity projection onthree orthogonal planes. The few EEG studies inthis NeuroImage issue use their specialized visual-ization, and a meta-analysis plot activation coor-dinates from the literature in 2D and 3D as wellas plot showing the ranges of x, y and z coordi-nates. Other forms of visualization that appear inthe literature are, e.g., volume rendering and cor-ner cube visualization. Corner cube visualizationcombines a 3D rendering with 2D tri-planar plots,where, e.g., thresholded summary statistics appear

as 3D objects with their projection on surrounding2D ‘walls’ (Rehm et al., 1998). Other more special-ized renderings are associated with diffusion tensionimaging and tractography and visualization for es-timated EEG dipoles that require glyphs with di-rectionality. Results reported with respect to brainregions—not volumes or coordinates—may be dis-played by color-coding a segmented brain.

The typical visualization displays areas that sur-vive a threshold and usually color-codes the areaabove threshold according to t-value or F -value.There are, however, other elements from the datamining result that may be visualized. What theyare can be revealed by what has been termed in the‘only’ statistical formula by Sackett (2001):

Confidence =Signal

Noise×

√

Sample size (1)

Thresholding at a certain t-value will, e.g., be basedon the ‘confidence’ part of the equation, and thatis depended on the signal, the noise and the samplesize. In certain situations a researcher is actuallyinterested in the ‘signal’ or the signal to noise ra-tio. It is possible to plot estimates of these terms.In the General Linear Model the so-called contrastimage relates to the signal, while the residual is thenoise. The correlation coefficient is a kind of signalto noise ratio. That the issue of what is plotted isnot entirely unimportant is seen with the Reimold-effect (Reimold et al., 2005): The combination ofspatially varying signal and noise patterns togetherwith spatial smoothing as a step in the image pro-cessing may conspire to make it seem as if the ‘ac-tivations’ in the t-map area moved to areas of nosignal, e.g., white matter. Reimold et al. (2005)suggest viewing the contrast image instead of thet-map.

When viewing the thresholded t-maps or corre-lation visualizations one should remember that thethresholded voxels are selected in a multiple com-parison procedure and the values are no longer un-biased estimates, — unless corrected for. If a scat-ter plot is made with values of a thresholded voxelacross time or subject as a function of a design vari-able of interest, then the scatter plot and its associ-ated correlation may give the naive reader the im-pression of a stronger association than actual. Thiswill also be the case for, e.g., a bar plot of percentsignal changes for a peak voxel.

The biased correlation has recently been referredto as ‘voodoo correlations’ by Vul et al. (2009) orless colorful ‘biased post hoc correlations’ by Lieber-mann et al. (2009). Methods exist that deliversan unbiased scatter plot. In machine learning it iscommon to split the data into two sets to get an un-biased estimate of the model performance (Mørch

2

et al., 1997). This may also be done in more stan-dard neuroimaging analysis with one part of thedata acting as a hypothesis-generating data set todetermine the region above threshold and the otherpart acting as the hypothesis-testing data set thatgives unbiased estimates within the thresholded re-gion. This procedure is not widespread, but havebeen applied in a few PET studies, see, e.g., Fiezet al. (1996) and Law et al. (1998). Though thesetwo studies do not have unbiased statistical valuesfor other reasons: the split does not occur acrosssubjects or subjects are excluded in the hypothesis-testing group if they do not survive the threshold.

Interpretation of a visualization with a thresholdshould be done with caution. Such a visualizationmay hide an area that did not survive the thres-hold, and should, say, an activation appear in theright hemisphere and not in the left hemisphere,one is (usually) not allowed to conclude that thereis a statistical significant difference between the leftand right hemispheres: If the threshold is selectedon, say, t = 4 the right activation may be 4.1 whilethe left 3.9, and given variation in the data the 0.2difference may not be significant, see also the dis-cussion by Jernigan et al. (2003). The phenomenonis not limited to neuroimaging and statisticians hasdiscussed it in a paper with the telling title TheDifference Between “Significant” and “Not Signifi-cant” is not Itself Statistically Significant (Gelmanand Stern, 2006), which also lists the standard is-sues with thresholding, e.g., the arbitrariness withthe threshold on, e.g., 0.05.

There is a range of non-spatial visualization forneuroimaging data. Some of these pertains to qual-ity assurance, e.g., Keator et al. (2008) report thecomputation of values such as center-of-mass, meanintensity, per-slice spikiness and have displays of‘mean of running difference’ and ‘per-slice varia-tion’, and Luo and Nichols (2003) present plots withseveral different statistical measures to characterizesignal and noise, see also Duff et al., this issue. Im-age processing programs could generate these kindsof visualization as part of a workflow.

Large-scale studies may process individual sub-jects as they are scanned, and plots can render theevolution of the statistical tests on the hypothesisof interest as more and more subjects are added, seeFigure 1 and Frokjaer et al. (2008). In the BredeToolbox bootstrap resampling gives an indication ofthe variation of the estimate as more subjects areadded. If the scanning of subjects is stopped basedon the statistical test it should be noted that mul-tiple dependent comparisons have been performedand the statistical threshold should be corrected.The statistical area of sequential analysis deals withthis problem.

10 20 30 40 50 60 70 80−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1

Data points

Cor

rela

tion

coef

ficie

nt

1000 bootstraps

Median

Maximum

Minimum

97.5%

2.5%

Figure 1: Plot of the evolution of the partial cor-relation coefficient as the number of subjects in-cluded in the analysis is increased from 1 to 83. Thelines indicate maximum, 97.5% fraction, median,2.5% fraction and minimum correlation among 1000bootstrap samples. Original data from Frokjaeret al. (2008).

Temporal signals, such as the fMRI time courses,are often visualized with line plots. Duann et al.(2002) suggest another way of revealing the timecourse of fMRI data with the ‘BOLD image’ thatplots multiple time-series as an image with the am-plitude color-coded. In the Brede Toolbox we alsouse ‘image’ plots to visualize the data and resultsin partial correlation analysis, see Figure 2. Kherifet al. (2003) describe one example of visualiza-tion with multidimensional scaling (MDS) plots ofspatial and temporal similarities between subjects.Similarities are computed as model-based RV coef-ficients between all pairs of subjects. The resultingMDS shows subjects as individual points in the 2Dplots and makes it relatively easy to visually spottemporal or spatial outliers. The ‘receptor finger-prints’ of Zilles and Palomero-Gallagher (2001) dis-play brain region results of multiple dimensions ina polar plot.

4 Visualizing predictors

Standard neuroimaging data mining results in avolume, — often termed a statistical parametricmap. In this mode of analysis design variablescan be regarded as predictors and the neuroimagingdata as response variables. Another mode of datamining uses the neuroimage data to predict the ex-perimental variables (Lautrup et al., 1995). Stan-

3

First matrix

10 20 30

10

20

30

40

50

60

70

80

Second matrix

5 10 15

10

20

30

40

50

60

70

80

Nuisance matrix

0.5 1 1.5 2 2.5

10

20

30

40

50

60

70

80

Partial correlation coefficient

5 10 15

5

10

15

20

25

30

35

Uncorrected 2−tailed P−values

5 10 15

5

10

15

20

25

30

35

Corrected 2−tailed P−values

5 10 15

5

10

15

20

25

30

35

Figure 2: Partial correlation analysis data and re-sults presented in the Brede Toolbox. Multiple per-sonality trait scores, multiple brain regional valuesfrom positron emission tomography and nuisancevariables (sex and age) for 83 subjects are displayedin the upper row. The lower row presents the re-sults of partial correlation analysis: the correlationmatrix and matrices of corrected and uncorrectedP -values. Data from Frokjaer et al. (2008)

dard statistics, such as multiple regression, cannotdo statistical inference in this case as the predic-tors form a massively ill-posed problem, and theproblem is often recast as a machine learning task.As such the constructed machine learning algo-rithms can classify patients from healthy controls(Lautrup et al., 1995) or classify fMRI and PETscans with brain activation induced by tasks andstimuli (Mørch et al., 1995, 1997). The approachusually splits the data into a training and a testset. First the model parameters of the machinelearning algorithm are estimated on the trainingset, and then the classification or regression per-formance of the model is evaluated on the test set.The primary result is not an image that may bevisualized, but ‘just’ the prediction model and itsperformance. However with further computation,images can be formed that tell how much voxelsare involved in the prediction. For linear mod-els a weight vector is usually associated with theprediction, and this can be visualized, for a recentexample see, e.g., (Mourao-Miranda et al., 2007).For non-linear models, such as multilayer sigmoidalneural networks and non-linear support vector ma-chines, the visualization is not straightforward, butsome approaches have been suggested. The saliencymap technique, inspired from the method of Opti-mal Brain Damage of Le Cun et al. (1990), esti-

Figure 3: Meta-analytic forest plot as a Web ser-vice with studies on personality genetics. Compo-nents in the Scalable Vector Graphics image file arehyperlinked and the content may be controlled in-teractively through a HTML form.

mates how much the elimination of a voxel affectsthe prediction performance. A closed-form approx-imation for the effect exists for a trained multilayerneural network and the resulting map can be visu-alized in 3D brain space (Mørch et al., 1995). Thesensitivity map asks a related question: What isthe effect of a small perturbation of a voxel (Kjemset al., 2002). Both the saliency and sensitivity mapcan also be applied with linear models. Althoughlinear weight vectors, saliency and sensitivity mapsmay be compared to results from standard statis-tical analysis, e.g., by Receiver Operating Charac-teristics curves and histogram equalization (Langeet al., 1999; Hansen et al., 2001), the values oftheir individual elements are not directly ‘statis-tically’ interpretable. For gaining a further statis-tical interpretation of the magnitude of the individ-ual elements of a linear weight vector, a saliency orsensitivity map these multidimensional results maybe compared to results from resampling. One ap-proach, the NPAIRS framework of Strother et al.(2002), uses split-half resampling. Such methodslead to statistically justifiable thresholds for use invisualization of neuroimaging predictors.

5 Meta-analytic visualization

Many meta-analyses use so-called forest plots andfunnel plots, where scatter plots with whiskers dis-play effect sizes and estimators of their variations intwo dimensions (Lewis and Clark, 2001), see Fig-ure 3. These meta-analyses typically investigate

4

a single variable—continuous or dichotomous—andits relation to another variable, e.g., a personal-ity trait and its association with a genetic poly-morphism. In neuroimaging meta-analysis we havea quite different situation: The neuroimage resultcontains not just one variable but many variables,i.e., voxels. One would need thousands of standardmeta-analysis plots to capture the result acrossstudies. Another much more fundamental problemstems from the fact that neuroimaging researcherstypically only report the positive results, e.g., ar-eas with activation to a given task, — not signalchanges for brain regions that did not survive thestatistical threshold selected. Meta-analysts usu-ally regard the discarding of negative results as aheresy, referring to it as the file drawer problemor with the term publication bias. All the standardstatistical meta-analysis technique require that alsonegative results are reported, — at least to some ex-tent (Hedges and Olkin, 1985). So we may ask if itis at all possible to make appropriate analyses andvisualizations across studies in neuroimaging?

One simple visualization simply plots the posi-tive results—the reported coordinates—in stereo-taxic space. The program associated with the orig-inal BrainMap database displayed coordinates in2D tri-planar plot (Fox et al., 1994). This typeof visualization is maintained in a newer versionof the database with the program Sleuth (Lairdet al., 2005). WebCaret may display coordinatesin 3D as colored spheres together with an inflatedcortical surface (Van Essen and Dierker, 2007), seeFigure 4. The Brede Toolbox can generate 3D vi-sualizations in the corner cube style of Rehm et al.(1998). Plotting points in 3D is not straightfor-ward, — simple ‘zero’ dimensional graphics do notgive an important perception of depth, thereforewe use 3D glyphs of different color and shape. Tohelp the viewer in spatial localizing the coordinateswe can add components in a configurable workflowsuch as AC/PC axes, stalks for the glyphs, glyphshadows on the tri-planar walls, contour and cere-bral cortex outlines from the atlas of Talairach andTournoux (1988). Figure 5 shows two visualiza-tions of this kind with Figure 5(a) displaying allcoordinates in the Brede Database from papers au-thored by Edward T. Bullmore and Figure 5(b) dis-playing cingulate coordinates colored according re-sults from a text mining of the associated abstracts(Nielsen et al., 2005, 2006a). The batch script setupfor the Brede Database will automatically generatea plot like Figure 5(a) for each author mentioned inthe author ontology. Sometimes these simple plotsreveal interesting features: The Bullmore coordi-nates appear somewhat limited to the middle of theinferior-superior axis perhaps reflecting a restricted

Figure 4: WebCaret server-side display of returnedcoordinates from the Surface Management Sys-tem Database (SumsDB) with a query on ’middlefrontal gyrus’. The right window offers some con-trol over the rendering and the buttons in the leftwindow may rotate the cortical surface. SumsDBallows the query on a neuroanatomical label to beinvoked from another program or Web site by sim-ple Web linking, and the Brede Wiki automaticallyconstructs such links.

field of view selected for some of the studies. Theelaborate and automated workflow for generating aplot like Figure 5(b) involves

1. Select a brain region and from the BredeDatabase brain region ontology get all nam-ing variation of the brain regions and its sub-areas. With these names extract coordinatesfrom papers recorded in the database, modeltheir spatial distribution and include extranon-matched coordinates that lies within theregion.

2. Get abstracts from the Brede Database that—for the brain region in question—have one ormore coordinates and perform text mining,which results in clusters of themes, such as‘pain’ and ‘memory’ and documents belongingto these clusters.

3. Perform statistical tests on the spatial distri-bution of the coordinates grouped accordingto the text mining clusters to determine if thetext mining has discovered functions that aresegregated in the region.

The procedure is done for all brain regions in theBrede Database brain region ontology and Fig-ure 5(b) shows one of the regions that listed highafter sorting brain regions according to statisticalsignificance in the spatial distribution test.

Data mining directly with the coordinateshas been termed coordinate-based meta-analysis

5

(a) Coordinates from the five studies in the Brede Database authored by Edward T. Bullmore.The 3D glyphs have type and color according to paper.

(b) Cingulate coordinates colored according to the clustering results after a text mining ofabstracts in the Brede Database. Dark magenta glyphs are from the ‘memory’ cluster whilethe light yellow are from the ‘pain’ cluster. From Nielsen et al. (2006a).

Figure 5: Two examples of coordinates in a 3D corner cube visualization.

6

(a) Gaussian mixture model of the three main behavioraldomains in BrainMap: Perception (red wireframe), cog-nition (green surface) and motion (‘M’-textured surface).From Nielsen and Hansen (1999).

(b) Kernel density modeling of auditory (red wireframe)and vision (green) studies. From Nielsen and Hansen(2000b).

Figure 6: VRML visualizations with coordinate-based meta-analysis of data from BrainMap with iso-surfaces in conditional probability densities.

(CBMA) and several methods exists (Wager et al.,2009), see also Laird et al., this issue. For themost part they involve a form of estimation of aconditional probability density p(v|c) in stereotaxicspace v. The conditioning, c, may be, e.g., for aspecific brain function or a specific anatomical la-bel. Once the probability density is estimated it canbe converted to a volume by sampling the proba-bility density in voxels and visualized in the sameway as standard neuroimages, or the density canbe used to color-code the cortical surfaces in a 3Dvisualization, see Wager et al. (2009).

Fox et al. (1997) introduced the method to modelthe probability density: A single confined area—theprimary motor area for the mouth—were examinedso only a model with mean and standard devia-tion was devised, i.e., a simple Gaussian model.As more complex brain functions are distributed inbrain space more flexible models are needed. Ourfirst effort in modeling the probability density wasby Gaussian mixture models (Nielsen and Hansen,1999)

p(v|c) =

Kc∑

k

p(v|k)P (k|c), (2)

where each p(v|k) estimates a 3D Gaussian prob-ability density. Figure 6(a) shows the isosurfacesin model of this type where the parameters have

been fitted to data from the BrainMap database.Here each ellipsoids corresponds to a single Gaus-sian p(v|k) and c corresponds to three differentlabels of ‘behavioral domain’ from the BrainMapdatabase that are associated with each coordinate.Although the Gaussian mixture model may gen-eralize, the ellipsoids do not look neuroanatomicalplausible and call for yet more flexible models. Fig-ure 6(b) is generated with kernel density estima-tion using a Gaussian kernel (Nielsen and Hansen,2000b). Such models seems to generate probabili-ties that are somewhat more neuroanatomical plau-sible than the Gaussian mixture model.

The isosurfaces in the probability densities inboth subplots of Figure 6 has been set for displaypurpose. More statistically grounded values can beobtained with the methods by Turkeltaub et al.(2002); Nielsen (2005); Costafreda et al. (2009).The methods for probability density estimation ofcoordinates are not limited to activations but maybe applied to any kind of coordinates in stereo-taxic space from ‘deactivations’, cortical stimula-tions, lesions or structural changes, e.g,. obtainedwith voxel-based morphometry.

When a probability density estimate is con-structed for a set of coordinates and it is convertedto a voxel-volume, then the volumes across mul-tiple sets of coordinates may be aggregated into

7

a single data matrix X(sets × voxels). This datamatrix may then be decomposed with multivariateanalysis in a number of ways, e.g., with singularvalue decomposition for principal component anal-ysis, ULV

′ = X, where the left factorization matrixU(sets× components) contains loading over sets ofcoordinates for each principal component and theright factorization matrix V(voxel × components)contains loadings over voxels. Other types of de-composition for this matrix is independent compo-nent analysis (MS = X, with M the mixing ma-trix and S the source matrix), non-negative matrixfactorization (WH = X) and K-means clustering(CA = X, with C a centroid matrix and A anassignment matrix). The right decomposition ma-trices, V, S, H and A all contain vectors that eachrepresents a volume. As part of the workflow forpresenting the information in the Brede Databaseon the Web the decompositions work on data ma-trices formed from sets of papers and sets of experi-ments, and corner cube visualizations are automat-ically constructed with isosurfaces in the volumescontained in the right decomposition matrices. Fig-ure 7 shows such a visualization for a componentfrom non-negative matrix factorization, i.e., a rowin the H matrix. Such visualizations may be usefulfor navigating among the studies in the database,and to a certain extent they reveal spatial distri-butions of the ‘cognitive components’ of the brain.Together with the visualization on the Web pageis listed the experiments that have high associationwith the component, i.e., experiments associatedwith large elements in a column of the left matrixW. For the component in Figure 7 they are exper-iments described as, e.g., ‘Visual object decision’,‘Buildings visual objects’, ‘Color perception duringfree viewing’ and ‘Passively viewed scenes’.

Before putting too much trust in visualizationsand analysis across studies one needs to remem-ber that the study results may have arisen in quitedifferent ways. In standard meta-analysis the onlyvariations between studies that are usually mod-eled is the number of subjects and the standarddeviation of the data in the individual studies. Inneuroimaging meta-analysis and visualization thesevariables are not usually modeled, for exceptionssee Fox et al. (1997) and Laird et al., this issue. Be-sides there are several other variables that neitherare considered: The varying thresholds applied,e.g., corrected and uncorrected P -values (Nielsenet al., 2006b), the difference in field of view be-tween studies, the reporting style of coordinates(e.g., ‘extent threshold’, ‘number of maxima percluster’) as well as the variation from the differentpre-processing and analysis choices that have beenmade. Furthermore, the different CBMA model

Figure 7: Corner cube visualization on the Webpage of the Brede Database with results from a non-negative matrix factorization of experiments in thedatabase.

may produce different results on the same material.Salimi-Khorshidi et al. (2009) compared differentCBMA models and their application of a thresholdmakes a ‘blob’ appear and disappear depending onthe type of CBMA.

6 Internet-based visualization

Quite a few tools exist for interactive neuroimagingvisualization across the Internet. Often these toolsare based on a client-server model with the clientimplementing the visualization and graphical userinterface in Java. Among these tools are JIV thatrenders multiple volume data by orthogonal sliceviews implemented as a Java applet (Cocosco andEvans, 2001). iiV implements a similar functional-ity (Lee et al., 2008), and MindSeer can also renderin 3D remotely (Moore et al., 2007). NeuroTer-rain implements 3D visualization and has demon-strated its use in connection with a Mouse atlas(Gustafson et al., 2007). The Talairach Applet ren-ders a digital representation of the Talairach Atlasand combines it with neuroanatomical labeling ofcoordinates via the Talairach Daemon described byLancaster et al. (2000). Also in connection with theBrainMap database the Java client-program Sleuthplots 3D points in orthogonal 2D slices based onuser query to the BrainMap server (Laird et al.,2005).

The Internet Brain Volume Database (IBVD)records published values for brain region volumesacross variables such as gender and diagnosis(Kennedy et al., 2003). Since the neuroimagingdata analysis arrives at one single value—the brain

8

Figure 8: Screenshot of the Web page for an ex-periment in Brede Database with a corner cube vi-sualization of the coordinates in a experiment to-gether with a wireframe indicating an isosurface ofthe kernel density estimate with the coordinates.An interactive rendering is provided with the linkto a generated corner cube visualization in a VRMLfile.

volume in cubic centimeters—the visualization ofthe data is relatively simple compared to other neu-roinformatics visualizations: From Web-based userqueries IBVD generates on-the-fly PNG image-fileswith the brain volumes from the different studiesplotted as a function of age with color-coding andthe variability indicated. Interactive visualizationsystems for neuroimages with server-side 3D ren-dering have been described by Poliakov et al. (2005)and a public system is available with the WebCaretWeb service, see Figure 4.

With the Brede Toolbox we construct 3D visual-izations browsable on the Web by using the VirtualReality Modeling Language (VRML) (ISO/IEC,1997; Nielsen and Hansen, 2000a), see the VRMLexamples in Figure 6. When defined in the middleof 1990s VRML held great promise to get wide-spread use for 3D interactive and hyperlinked visu-alizations, but since then it has had limited growth:VRML lacks good browser implementations andthere has been erratic adoption of a scripting lan-guage. Nevertheless, it is one of the few means forWeb distribution of 3D content in free standard-ized form. An alternative format is the Univer-sal 3D File Format (U3D) that can be embeddedin newer versions of the PDF format. Apart fromthe Brede Toolbox ImageSurfer described by Fenget al. (2007) implements VRML export.

For the Web presentation of the Brede Databasewe generate 3D corner cube visualizations of the

Figure 9: The Web-based INC Interactive TalairachAtlas queried with a coordinate from the BredeWiki.

coordinates in the database with an offline Mat-lab batch script, — both as image files embeddedon the Web page as well as VRML files, see Fig-ure 8. Matlab is not well suited to work as a Webscript, and for the interactive Web scripts associ-ated with the Brede Database there are presentlyno visualization implemented. The INC InteractiveTalairach Atlas renders 2D orthogonal slices fromthe Talairach and the MNI single subject atlases.This Web service can merge a user-given coordi-nate with the visualization, and as such we use itfor visualization of individual coordinates from theBrede Database and the Brede Wiki.

Besides Java, VRML and standard image filessuch as PNG the Scalable Vector Graphics (SVG)format may prove useful for Internet-based visual-izations, see Figure 3 for an example. These filesmay contain hyperlinks and JavaScript. However,Web browsers do not yet consistently implementthe standard.

7 Information visualization

Data mining results from neuroimaging analysis arenot the only type of information for visualization.Information about the background, design, scan-ning, analysis procedure, and interpretation sur-rounds the data mining results of a typical neu-roimaging study. In scientific articles the body text

9

1 2 3 4 5

1

2

3

4

5

Component

Num

ber

of c

ompo

nent

sCluster bush

memoryepisodicpainretrieval

memoryepisodicretrievalwords

painpainfulheatnoxious

memoryepisodicretrievalwords

facialexpressionsfacesemotion

painpainfulheatnoxious

memoryepisodicretrievalwords

facialexpressionsfacesemotion

visualeyemotionmotor

painpainfulheatnoxious

memoryepisodicretrievalwords

associativeencodingverbalnovelty

facialexpressionsfacesemotion

visualeyemotionmotor

painpainfulheatnoxious

Figure 10: A so-called ‘cluster bush’ visualizationof the text mining results of the abstracts in theBrede Database. Each yellow dot is a cluster ofarticles and words in the article. The four wordswith highest load on each cluster are listed.

mostly carries this ‘context’ information, thoughsometimes authors also use tables to describe, e.g.,subject information. Authors rarely apply visual-izations for this kind of information except in sit-uations with explanation of the experimental de-sign and scanning. The experimental design hasa natural temporal evolution and as such the vi-sualization often displays the design as a functionof time. Users of the behavioral experiment soft-ware from Psychology Software Tools is familiarwith the graphical programming environment of E-Prime which has this kind of visualization as anintegral part of the development of the experiment.Other parts of the neuroimaging study may be vi-sualized with what is usually referred to as infor-mation visualization.

In a demonstration visualization we employed atorus topology for the an entire neuroimaging studyprocess constructing 3D icons for ‘funding’, the ex-perimental design, authors, experimental subjectsetc. (Nielsen and Hansen, 1997). The usefulness ofsuch a visualization depends on how effective it con-veys information compared to standard text, andif the visualization format requires specialized andlimited distributed programs for rendering and in-teraction the impact may be small. Manual cre-ation of these visualizations is infeasible, — thevisualization should be constructed automaticallyfrom description of the study, e.g., the so-called‘provenance’ (Fissell, 2007). In related visualiza-tions some workflow management systems display

Figure 11: Corner cube visualization of a ‘func-tional’ atlas generated from a combined data min-ing of text and brain coordinates.

the processing flow graphically (Dinov et al., 2008).

When neuroimaging studies get reported in ar-ticles the relationships between the articles can beturned in to visualizations. Many types of visual-izations exist and many relationships may be re-vealed: Between terms, concepts, citations to andfrom articles as well as between authors, cited au-thors and cited journals. The visualizations are ofcourse not limited to articles only in neuroimag-ing, see, e.g., Card et al. (1999); Chen (1999). Foran example in neuroscience Naud et al. (2007) usea spherical embedding algorithm to display a bi-partite graph in 3D space with two spheres. Oneof their illustrations visualized the relationship be-tween poster sessions in the Society for Neuro-science 2006 meeting together with words from theabstracts in the sessions. Another example of textmining result visualization is what we termed a‘cluster bush’, that describe the clusters in a hier-archical multivariate analysis (Nielsen et al., 2005):Clusters are indicated with dots and thick linesindicate a large similarity between two clusters.Given a set of abstracts the automated workflowfor generating a plot like Figure 10 involves theconversion of the texts to a bag-of-words matrix,the exclusion of a large number of words (stopwords), hierarchical non-negative matrix factoriza-tion and lastly the ’cluster bush’ visualization allimplemented with the functions of the Brede Tool-box.

Coordinate-based meta-analysis and text miningcan be combined to form visualizations, see Fig-ure 11 and Nielsen et al. (2004). The workflowfor constructing the visualization in the figure in-volves the setup of a matrix describing the words in

10

neuroimage

hum bra map

pro nat aca sci usa

j neurosci

nature

sciencebrain

cer cor

j cer blo flo met

j neurophyneuroreport

j com ass tom

j cog neurosci

mag res med

neuropsychologia

neurology

exp bra res

ann neurol

bra resj com neurolele cli neurophyeur j neurosci

tre neurosci

arc neurol

neurosci letann rev neurosci

j nuc med

soc neurosci absneuron

iee tra med ima

mag res ima

cog bra res

arc gen psy

cur opi neurobio

radiology

cerebral, pet, mapping, hum

an, regionalfm

ri, fu

nctio

nal,

mem

ory,

eve

nt−

rela

ted,

wor

king

human, cortex, areas, somatosensory, role

fmri, pet, bold, analysis, parametric

Figure 12: Visualization of data mining result of journal co-citation analysis with singular value decom-position on citation data from NeuroImage.

the abstract of papers and the construction of an-other matrix from kernel density estimation withthe coordinates in each paper. After non-negativematrix factorization each individual factor may berendered in 3D and associated with words from theabstract, e.g., the blue area in Figure 11 in the oc-cipital lobe is associated with words such as ‘visual’and ‘eye’.

Based on a corpus of articles published between1997 and 2000 in the journal NeuroImage we couldplot cited authors and cited journals in 2D. Thedata mining with visualization would for exam-ple reveal a dichotomy between PET and fMRI(Nielsen, 2002), see Figure 12. Here the workflowinvolves specialized algorithms that extract cita-tions and the use of matrix computations, particu-larly singular value decomposition, for multidimen-sional scaling-like projection of the data onto 2D.For the Brede Database we automatically constructwhat we have termed ‘bullseye plots’ to display thenetwork of coauthors for each recorded author. Fig-ure 13 shows a larger bullseye plot on coauthors

in the NeuroImage corpus. Authors near the cen-ter, such as Friston and Dolan, have high networkdegrees, which here corresponds to the number ofauthored articles (Nielsen, 2002).

The well-tested and widely used GraphViz pack-age provides spatial graph layout for a given net-work (Gansner and North, 2000). At one point thePubGene Web service used GraphViz in a large-scale application for displaying relations betweengenes based on literature in PubMed (Jenssen et al.,2001). GraphViz layouts graphs for the Web pre-sentation of the Brede Database. These graphs dis-play the brain function and brain region ontolo-gies, e.g., indicating that ‘vision’ has ‘perception’as taxonomic parent or that the cingulate area isa parent for the posterior cingulate, see Figure 14.Our workflow with the Brede Toolbox involves ex-traction of the ontology from Brede Database XMLfiles, construction of a file with the graph thatGraphViz reads, invoking GraphViz for generationof an image file, and then finally construction of theWeb page with the image file embedded. GraphViz

11

−10 −8 −6 −4 −2 0 2 4 6 8 10−10

−8

−6

−4

−2

0

2

4

6

8

10

Friston K

Dolan RZarahn EAguirre G

Ashburner J

Esposito M

Zilles K

Holmes A

Evans A

Hari R

Kawashima R

Noll D Price C

Rosen BFletcher P

Frith C

Glover G

Josephs O

Poline J

Turner R

Buckner R

Büchel C

Dale A

Frackowiak R

Fukuda H

Howseman A

Mazoyer B

McIntosh A

Alpert N

Andreasen N

Cohen J

Cramon D

Fazio F

Frank J

Geyer S

Gitelman D

Gjedde A

Haxby J

Johannsen P

Mattay V

Mellet E

Mesulam M

Petersson K

Rugg M

Sadato N

Schleicher A Schormann TSugiura M

Worsley KApkarian A

Baron J

Bettinardi V

Boulanouar K

Buechel C

Chua P

Decety J

Desmond J

Detre J

Doyon B

Duyn J

Fabiani M

Farde L

Franck G

Gabrieli J

Gore J

Goto R

Grassi F

Gratton G

Gunn R

Gärtner H

Halldin C

Hansen L

Horwitz B

Ingvar M

Kapur S

Kato T

Kennedy D

Kiebel S

Koutstaal W

LaBar K

Leary DMagnotta V

Meyer J

Michel C

Minoshima S

Moseley M

Nakamura A

Nielsen F

Okada K

Orban G

Parrish TPaulson O

Perani D

Petit L

Ranjeva J

Rees G

Rostrup E

Ruttimann USalenius S

Salmelin R

Schacter D

Senda M

Smith A

Strother S

Toga A

Tzourio N

Weinberger DYang Y

Ye F

Figure 13: Coauthor bullseye plot (target diagram) with data from NeuroImage 1997–2000. A linebetween two authors indicates that they co-wrote a paper. The concentric circles indicate the numberof articles written by the author in the corpus.

can construct HTML image maps so the nodes inthe graph image are associated with clickable hy-perlinks. On the final Web page a reader may nav-igate the brain region and brain function ontolo-gies by clicking on the nodes in the graph. TheBrede Toolbox can also use GraphViz for layoutof other types of data that can be described as anetwork, e.g., from structural equation modelingof regional neuroimaging data. A number of jour-nal Web sites use plots called Citation map in thestyle of GraphViz for visualizing in- and out-goingcitations of each article, see, e.g., BMJ and TheJournal of Neuroscience Web sites.

Another type of graph visualization within neu-roimaging is the interactive graph visualizationwith a hyperbolic browser that features in toolsfrom the Laboratory of Neuro Imaging (LONI):LOVE and iTools (Dinov et al., 2006, 2008). ISIWeb of Knowledge provides a Java applet to render

their citation information with a similar topology.

8 Conclusion

Numerous visualization tools exist for display ofneuroimages, and several Web directories recordthese tools. There are different types of results fromneuroimaging analysis that can be visualized, andone should be aware of the problems that a naivereader faces when viewing such results.

With the Brede Toolbox we are able to build aworkflow with extraction of data from the BredeDatabase, automated data mining and visualiza-tions. The automated procedures generate pub-licly accessible Web pages with interactive visual-izations. An advantage of the automated procedureis that little human intervention is required to up-date the visualizations as new data is added to the

12

Figure 14: For presenting the Brede Database brainregion ontology on the Web the workflow withthe Brede Toolbox invokes the GraphViz programwhich generates hyperlinked plot of the brain regionhierarchy, here for the ’posterior cingulate gyrus’.

database. The visualizations can display not onlyspatial neuroimages, but for example also resultsfrom text mining, and visualization can take placeacross the Internet with data originating on oneserver and displayed on another.

Acknowledgment

Many thanks to Kristoffer Hougaard Madsen,Daniela Balslev and Lars Kai Hansen for com-ments on an earlier version of the manuscript. Thiswork was supported by the Lundbeck Foundationthrough the Center for Integrated Molecular BrainImaging.

References

Bazin, P.-L., Cuzzocreo, J. L., Yassa, M. A.,Gandler, W., McAuliffe, M. J., Bassett, S. S.,and Pham, D. L. (2007). Volumetric neu-roimage analysis extension for the MIPAV soft-ware package. Journal fo Neuroscience Methods,165(1):111–121.

Buccigrossi, R., Ellisman, M., Grethe, J., Hasel-grove, C., Kennedy, D., Martone, M., Preuss, N.,Sullivan, M., and Wagner, K. (2008). The neu-roimaging informatics tools and resources clear-inghouse (NITRC). In 14th Annual Meetingof the Organization for Human Brain Mapping,pages 319 T–AM. Organization for Human BrainMapping.

Card, S. K., MacKinlay, J. D., and Shneiderman,B., editors (1999). Readings in Information Vi-

sualization. Using Vision to think. The Mor-gan Kaufmann Series in Interactive Technolo-gies. Morgan Kaufmann Publishers, San Fran-cisco, California, USA.

Chen, C. (1999). Visualising semantic spacesand author co-citation networks in digital li-braries. Information Processing and Manage-ment, 35(3):401–420.

Cocosco, C. A. and Evans, A. C. (2001). Javainternet viewer: A WWW tool for remote 3Dmedical image data visualization and compar-ison. In Proceedings of the 4th InternationalConference on Medical Image Computing andComputer-Assisted Intervention, volume 2208 ofLecture Notes In Computer Science, pages 1415–1416, London, UK. Springer.

Costafreda, S., David, A. S., and Bram-mer, M. J. (2009). A parametric approachto voxel-based meta-analysis. NeuroImage,doi:10.1016/j.neuroimage.2009.01.031.

Dinov, I. D., Rubin, D., Lorensen, W., Dugan, J.,Ma, J., Murphy, S., Kirschner, B., Bug, W.,Sherman, M., Floratos, A., Kennedy, D., Ja-gadish, H. V., Schmidt, J., Athey, B., Califano,A., Musen, M., Altman, R., Kikinis, R., Kohane,I., Delp, S., Parker, D. S., and Toga, A. W.(2008). iTools : a framework for classification,categorization and integration of computationalbiology resources. PLoS ONE, 3(5):e2265.

Dinov, I. D., Valentino, D., Shin, B. C., Konstan-tinidis, F., Hu, G., MacKenzie-Graham, A., Lee,E.-F., Shattuck, D., Ma, J., Schwartz, C., andToga, A. W. (2006). LONI visualization environ-ment. Journal of Digital Imagin, 19(2):148–158.

Duann, J.-R., Jung, T.-P., Kuo, W.-J., Yeh, T.-C., Makeig, S., Hsieh, J.-C., and Sejnowski, T.(2002). Single-trial variability in event-relatedBOLD signals. NeuroImage, 15(4):823–835.

Feng, D., Marchburn, D., Jen, D., II, R. J. W. R.M. T., and Burette, A. (2007). Stepping into thethird dimension. The Journal of Neuroscience,27(47):12757–12760.

Fiez, J. A., Raife, E. A., Balota, D. A., Scwarz,J. P., Raichle, M. E., and Petersen, S. E. (1996).A positron emission tomography study of theshort-term maintenance of verbal information.The Journal of Neuroscience, 16(2):808–822.

Fissell, K. (2007). Workflow-based approaches toneuroimaging analysis. Methods in Molecular Bi-ology, 401:235–266.

13

Fox, P. T., Lancaster, J. L., Parsons, L. M., Xiong,J.-H., and Zamarripa, F. (1997). Functional vol-umes modeling: Theory and preliminary assess-ment. Human Brain Mapping, 5(4):306–311.

Fox, P. T., Mikiten, S., Davis, G., and Lancaster,J. L. (1994). BrainMap: A database of humanfunction brain mapping. In Thatcher, R. W.,Hallett, M., Zeffiro, T., John, E. R., and Huerta,M., editors, Functional Neuroimaging: TechnicalFoundations, chapter 9, pages 95–105. AcademicPress, San Diego, California.

Frackowiak, R. S. J., Friston, K. J., Dolan, R., andPrice, C. (2003). Human Brain Function. Aca-demic Press, second edition.

Frokjaer, V., Mortensen, E. L., Nielsen, F. A.,Haugbøl, S., Pinborg, L. H., Adams, K. H.,Svarer, C., Hasselbalch, S. G., Holm, S., Paul-son, O. B., and Knudsen, G. M. (2008). Fron-tolimbic serotonin 2A receptor binding in healthysubjects is associated with personality risk fac-tors for affective disorder. Biological Psychiatry,63(6):569–576.

Gansner, E. R. and North, S. C. (2000). An opengraph visualization system and its applicationsto software engineering. Software — Practice andExperience, 30(11):1203–1234.

Gardner, D., Akil, H., Ascoli, G., Bowden, D. M.,Bug, W., duncan E. Donohoe, Goldberg, D. H.,Grafstein, B., Grethe, J. S., Gupta, A., Halavi,M., Kennedy, D. N., Marenco, L., Martone,M. E., Miller, P. L., Muller, H.-M., Robert, A.,Shepherd, G. M., Sternberg, P. W., Van Essen,D. C., and Williams, R. W. (2008). The Neu-roscience Information Framework: A data andknowledge environment for neuroscience. Neu-roinformatics, 6(3):149–160.

Gardner, D. and Shepherd, G. M. (2004). A gate-way to the future of neuroinformatics. Neuroin-formatics, 2:271–274.

Gelman, A. and Stern, H. (2006). The difference be-tween “significant” and “not significant” is not it-self statistically significant. The American Statis-tician, 60(4):1–4.

Gouws, A., Woods, W., Millman, R., Morland, A.,and Green, G. (2009). DataViewer3D: An open-source, cross-platform multi-modal neuroimagingdata visualization tool. Frontiers in Neuroinfor-matics, 3:9.

Gustafson, C., Bug, W. J., and Nissanov, J. (2007).NeuroTerrain — a client-server system for brows-ing 3D biomedical image data. BMC Bioinfor-matics, 8:40.

Hansen, L. K., Nielsen, F. A., Strother, S. C., andLange, N. (2001). Consensus inference in neuro-imaging. NeuroImage, 13(6):1212–1218.

Hedges, L. V. and Olkin, I. (1985). Statistical Meth-ods for Meta-Analysis. Academic Press, Orlando,Florida.

ISO/IEC (1997). Information technology — Com-puter graphics and image processing — The Vir-tual Reality Modeling Language (VRML) — Part1: Functional specification and UTF-8 encod-ing. International Organization for Standardiza-tion/International Electrotechnical Commission.International Standard ISO/IEC 14772-1:1997.

Jenssen, T.-K., Læreid, A., Komorowski, J., andHovig, E. (2001). A literature network of hu-man genes for high-throughput analysis of geneexpression. Nature Genetics, 28(1):21–28.

Jernigan, T. L., Gamst, A. C., Fennema-Notestine,C., and Ostergaard, A. L. (2003). More “map-ping” in brain mapping: Statistical comparisonof effects. Human Brain Mapping, 19(2):90–95.

Keator, D. B., Grethe, J. S., Marcus, D., Ozyurt,B., Gadde, S., Murphy, S., Pieper, S., Greve,D., Notestine, R., Bockholt, H. J., Papadopou-los, P., BIRN Function, BIRN Morphometry,and BIRN-Coordinating (2008). A national hu-man neuroimaging collaboratory enabled by thebiomedical informatics research network (BIRN).IEEE Transactions on Information Technologyin Biomedicine, 12(2):162–172.

Kennedy, D. N., Haselgrove, C., and McInerney,S. (2003). MRI-based morphometric analysis oftypical and atypical brain development. MentalRetardation and Developmental Disabilities Re-search Reviews, 9(3):155–160.

Kherif, F., Poline, J.-B., Meriaux, S., Benali, H.,Flandin, G., and Brett, M. (2003). Group analy-sis in functional neuroimaging: selecting subjectsusing similarity measures. NeuroImage, 20:2197–2208.

Kjems, U., Hansen, L. K., Anderson, J., Frutiger,S., Muley, S., Sidtis, J., Rottenberg, D., andStrother, S. C. (2002). The quantitative eval-uation of functional neuroimaging experiments:mutual information learning curves. NeuroIm-age, 15(4):772–786.

14

Laird, A. R., Lancaster, J. L., and Fox, P. T.(2005). BrainMap: the social evolution of a hu-man brain mapping database. Neuroinformatics,3(1):65–78.

Lancaster, J. L., Woldorff, M. G., Liotti, M.,Freitas, C. S., Rainey, L., Kochunov, P. V.,Nickerson, D., Mikiten, S. A., and Fox, P. T.(2000). Automated Talairach atlas labels forfunctional brain mapping. Human Brain Map-ping, 10(3):120–131.

Lange, N., Strother, S. C., Anderson, J. R., Nielsen,F. A., Holmes, A. P., Kolenda, T., Savoy, R., andHansen, L. K. (1999). Plurality and resemblancein fMRI data analysis. NeuroImage, 10(3):282–303.

Lautrup, B., Hansen, L. K., Law, I., Mørch, N.J. S., Svarer, C., and Strother, S. C. (1995).Massive weight sharing: A cure for extremelyill-posed problems. In Herman, H. J., Wolf,D. E., and Poppel, E. P., editors, Proceedings ofthe Workshop on Supercomputing in Brain Re-search: From Tomography to Neural Networks,pages 137–148, Singapore. World Scientific.

Law, I., Svarer, C., Rostrup, E., and Paulson, O. B.(1998). Parieto-occipital cortex activation duringself-generated eye movements in the dark. Brain,121(11):2189–2200.

Le Cun, Y., Denker, J. S., and Solla, S. A. (1990).Optimal brain damage. In Touretzky, D. S., ed-itor, Advances in Neural Information Process-ing Systems: Proceedings of the 1989 Confer-ence, pages 598–605, San Mateo, CA. Morgan-Kaufmann. NIPS-2.

Lee, J. T., Munch, K. R., Carlis, J. V., and Pardo,J. V. (2008). Internet image viewer (iiV). BMCMedical Imaging, 8:10.

Lewis, S. and Clark, M. (2001). Forest plots:trying to see the wood and the trees. BMJ,322(7300):1479–1480.

Liebermann, M. D., Berkman, E. T., and Wager,T. D. (2009). Correlations in social neurosciencearen’t voodoo: A reply to Vul et al. Perspectiveson Psychological Science. In press.

Luo, W.-L. and Nichols, T. E. (2003). Diag-nosis and exploration of massively univariateneuroimaging models. NeuroImage, 19(3):1014–1032.

Moore, E. B., Poliakov, A. V., Lincoln, P., andBrinkley, J. F. (2007). MindSeer: a protable and

extensible tool for visualization of structural andfunctional neuroimaging data. BMC Bioinfor-matics, 8:389.

Mørch, N. J. S., Hansen, L. K., Strother, S. C.,Svarer, C., Rottenberg, D. A., Lautrup, B.,Savoy, R., and Paulson, O. B. (1997). Nonlinearversus linear models in functional neuroimaging:Learning curves and generalization crossover. InDuncan, J. and Gindi, G., editors, InformationProcessing in Medical Imaging, volume 1230 ofLecture Notes in Computer Science, pages 259–270, Berlin/Heidelberg. Springer.

Mørch, N. J. S., Kjems, U., Hansen, L. K., Svarer,C., Law, I., Lautrup, B., Strother, S. C., andRehm, K. (1995). Visualization of neural net-works using saliency maps. In Proceedings of1995 IEEE International Conference on NeuralNetworks, volume 4, pages 2085–2090.

Mourao-Miranda, J., Friston, K. J., and Brammer,M. (2007). Dynamic discrimination analysis: Aspatial-temporal SVM. NeuroImage, 36:88–89.

Naud, A., Usui, S., Ueda, N., and Taniguchi, T.(2007). Visualization of documents and conceptsin neuroinformatics with the 3D-SE viewer. Fron-tiers in Neuroinformatics, 1:7.

Nielsen, F. A. (2002). Neuroinformatics in Func-tional Neuroimaging. PhD thesis, Informaticsand Mathematical Modelling, Technical Univer-sity of Denmark, Richard Petersens Plads, Build-ing 321, DK-2800 Kgs. Lyngby. IMM-PHD-2001-87.

Nielsen, F. A. (2003). The Brede database: a smalldatabase for functional neuroimaging. NeuroIm-age, 19(2). Presented at the 9th InternationalConference on Functional Mapping of the Hu-man Brain, June 19–22, 2003, New York, NY.Available on CD-Rom.

Nielsen, F. A. (2005). Mass meta-analysis in Ta-lairach space. In Saul, L. K., Weiss, Y., andBottou, L., editors, Advances in Neural Informa-tion Processing Systems 17, pages 985–992, Cam-bridge, MA. MIT Press.

Nielsen, F. A. (2009). Brede Wiki: Neuro-science data structured in a wiki. Submitted toSemWiki2009.

Nielsen, F. A., Balslev, D., and Hansen, L. K.(2005). Mining the posterior cingulate: Segrega-tion between memory and pain component. Neu-roImage, 27(3):520–532.

15

Nielsen, F. A., Balslev, D., and Hansen, L. K.(2006a). Data mining a functional neuroimag-ing database for functional segregation in brainregions. In Olsen, S. I., editor, Den 15. DanskeKonference i Mønstergenkendelse og Billedanal-yse, Copenhagen, Denmark. The Department ofComputer Science, University of Copenhagen.

Nielsen, F. A., Christensen, M. S., Madsen, K. H.,Lund, T. E., and Hansen, L. K. (2006b). fMRIneuroinformatics. IEEE Engineering in Medicineand Biology Magazine, 25(2):112–119.

Nielsen, F. A. and Hansen, L. K. (1997). Interac-tive information visualization in neuroimaging.In Ebert, D. S. and Nicholas, C. K., editors, Pro-ceedings of the 1997 workshop on New paradigmsin information visualization and manipulation,pages 62–65, New York, NY, USA. ACM.

Nielsen, F. A. and Hansen, L. K. (1999).Modeling of BrainMap data. Online athttp://isp.imm.dtu.dk/publications/1999-/nielsen.nips99.ps.gz.

Nielsen, F. A. and Hansen, L. K. (2000a). Ex-periences with Matlab and VRML in functionalneuroimaging visualizations. In Klasky, S. andThorpe, S., editors, VDE2000 - VisualizationDevelopment Environments, Workshop Proceed-ings, Princeton, New Jersey, USA, April 27–28, 2000, pages 76–81, Princeton, New Jersey.Princeton Plasma Physics Laboratory.

Nielsen, F. A. and Hansen, L. K. (2000b). Func-tional volumes modeling using kernel densityestimation. Online at http://www.imm.dtu.dk-/pubdb/views/edoc download.php/4688/pdf-/imm4688.pdf.

Nielsen, F. A., Hansen, L. K., and Balslev, D.(2004). Mining for associations between textand brain activation in a functional neuroimag-ing database. Neuroinformatics, 2(4):369–380.

Poliakov, A. V., Albright, E., Hinshaw, K. P., Co-rina, D. P., Ojemann, G., Martin, R. F., andBrinkley, J. F. (2005). Server-based approach toweb visualization of integrated three-dimensionalbrain imaging data. Journal of the AmericanMedical Informatics Association, 12:140–151.

Rehm, K., Lakshminarayan, K., Frutiger, S. A.,Schaper, K. A., Sumners, D. L., Strother, S. C.,Anderson, J. R., and Rottenberg, D. A. (1998).A symbolic environment for visualizing activatedfoci in functional neuroimaging datasets. MedicalImage Analysis, 2(3):215–226.

Reimold, M., Slifstein, M., Heinz, A., Mueller-Schauenburg, W., and Bares, R. (2005). Effect ofspatial smoothing on t-maps: arguments for go-ing back from t-maps to masked contrast images.Journal of Cerebral Blood Flow & Metabolism,26(6):751–759.

Rorden, C. and Brett, M. (2000). Stereotaxic dis-play of brain lesions. Behavioural Neurology,12(4):191–200.

Sackett, D. L. (2001). Why randomized con-trolled trials fail but needn’t: 2. failure to em-ploy physiological statistics, or the only formulaa clinician-trialist is ever likely to need (or under-stand). Canadian Medical Association Journal,165(9):1226–1237.

Salimi-Khorshidi, G., Smith, S. M., Keltner, J. R.,Wager, T. D., and Nichols, T. E. (2009). Meta-analysis of neuroimaging data: A comparisonof image-based and coordinate-based pooling ofstudies. NeuroImage, 45:810–823.

Strother, S. C., Anderson, J., Hansen, L. K.,Kjems, U., Kustra, R., Sidtis, J., Frutiger, S.,Muley, S., LaConte, S., and Rottenberg, D.(2002). The quantitative evaluation of functionalneuroimaging experiments: the NPAIRS dataanalysis framework. NeuroImage, 15(4):747–771.

Talairach, J. and Tournoux, P. (1988). Co-planarStereotaxic Atlas of the Human Brain. ThiemeMedical Publisher Inc, New York.

Turkeltaub, P. E., Eden, G. F., Jones, K. M.,and Zeffiro, T. A. (2002). Meta-analysis of thefunctional neuroanatomy of single-word reading:method and validation. NeuroImage, 16(3 part1):765–780.

Van Essen, D. C. and Dierker, D. L. (2007).Surface-based and probabilistic atlases of pri-mate cerebral cortex. Neuron, 56:209–225.

Vul, E., Harris, C., Winkielman, P., and Pash-ler, H. (2009). Puzzlingly high correlations infMRI studies of emotion, personality, and socialcognition. Perspectives on Psychological Science.In press. The paper formerly known as “VoodooCorrelations in Social Neuroscience”.

Wager, T. D., Lindquist, M. A., Nichols, T. E.,Kober, H., and Snellenberg, J. X. V. (2009).Evaluating the consistency and specificity of neu-roimaging data using meta-analysis. NeuroIm-age, 1(1, supplement 1):S210–S221.

16

Zilles, K. and Palomero-Gallagher, N. (2001). Cyto-, myelo-, and receptor architectonics of the hu-man parietal cortex. NeuroImage, 14(1, part2):S8–S20.

17