Chinese Journal of ElectronicsVol.27, No.5, Sept. 2018

A Survey of Macroscopic Brain Network

Visualization Technology∗

CHEN Weifeng1, SHI Lei2 and CHEN Wei3

(1. School of Information, Zhejiang University of Finance & Economics, Hangzhou 310018, China)

(2. State Key Lab of Computer Science, Institute of Software, Chinese Academy of Sciences, Beijing 100190, China)

(3. State Key Lab of CAD&CG, Zhejiang University, Hangzhou 310007, China)

Abstract — Brain science, as an important branch ofNeuroscience, is a discipline that studies the structure and

function of brain nervous system of human and other mam-mals. With the invention of new technologies such as brain

imaging, light microscopes and brain electromagnetics,

brain science is gradually unraveling the mysteries of hu-man emotion, intelligence and behavior. In this wave, data

visualization punctuates the landmark advances in brainscience since its beginning. This survey reviews the recent

literature on brain network visualization (aka connectome)

from the fields of both connectomics and visualization. Inparticular, we focus on the macroscopic-level brain net-

work visualization techniques, that reveal the structural

and functional connectivity of the whole brain, in compar-ison to microsopic-level neuronal connectivities. We also

discuss the interactive visualization tools currently avail-able for viewing the brain networks. Finally, we conclude

with a number of ongoing challenges in macroscopic brain

network visualization.

Key words — Brain network, Connectomics, Visualiza-

tion.

I. Introduction

Connectomics is a division of neuroscience that stud-ies connectome, i.e., the map of the neuronal system inthe brain and all its connections among neuronal struc-tures. The goal of connectomics is to create a completerepresentation of such map by understanding the brain’swiring mechanism[1]. It is believed that the representa-tion of connectome can help to increase our knowledge ofhow functional brain states emerge from their underlyinganatomical structure[2].

Studies of the connectome, also know as the brain net-work, are becoming increasingly popular in the fields ofneuroscience, computer science and neurology. Modern

brain connectivity and brain network data have typicalbig data properties, because of the large variety of fea-tures recorded and represented. According to the taxon-omy of brain networks, three types of connectivity can bedistinguished. First, structural or anatomical connectivityusually represents the physical connections between neu-ral elements by a “wiring diagram”. Second, Functionalconnectivity concentrates on “the temporal correlationbetween spatially remote neurophysiological events”[3].Finally, effective connectivity concerns the causal inter-action between distinct units within the brain nervoussystem[3].

In Ref.[2], Sporns et al. differentiated among macro-,meso- and micro-scale connectomes. Several image modal-ities have been invented and employed in connec-tomics since 1920s, among which Electroencephalography(EEG)[4] is the oldest noninvasive functional neuroimag-ing technique. While EEG records electrical brain activ-ity from a bundle of electrodes on the scalp, Magnetoen-cephalography (MEG) measures magnetic fields outsidethe head which are induced by electrical brain activity[5].EEG and MEG provide functional connectivity informa-tion on the macroscale, as well as some newer imagemodalities such as functional Magnetic resonance image(fMRI)[6] and Positron emission tomography (PET)[7].The structural information on the macroscale can be ob-tained by image modalities such as Single-photon emis-sion computed tomography (SPECT) and Magnetic reso-nance imaging (MRI)[8]. At the macroscale, the spatialresolution is typically in the range of millimeters andthe temporal resolution ranges from milliseconds to min-utes (see Fig.1). The mesoscale connectome describes con-

∗Manuscript Received Feb. 24, 2018; Accepted Mar. 21, 2018. This work is supported by the National Key Basic Research andDevelopment Program of China (973 Program) (No.2014CB340301) and National Natural Science Foundation of China (No.61772504,

No.61772456, No.61761136020).c© 2018 Chinese Institute of Electronics. DOI:10.1049/cje.2018.04.007

890 Chinese Journal of Electronics 2018

nectivity in the range of micrometers, and lighting mi-croscopy (LM) techniques are the typical image modalitiesfor imaging of living cells. Most of LM techniques focuson structural information of single neurons, with suffi-cient resolution for the identification of major cell parts,such as dendrites, somas, axons, and also boutons as pos-sible locations for synaptic connections[1]. At the finestmicroscale, the connectome involves mapping single neu-ronal cells and their connectivity patterns. Electron mi-croscopy (EM) techniques enable imaging of neuronal tis-sue at the nanometer scale. However, the sample prepa-ration and image acquisition in EM is labor-intensive andtime-consuming, and the size of the acquired dataset isup to multiple terabytes.

With the invention of new imaging technologies, brainscience is gradually unraveling the mysteries of humanemotion, intelligence and behavior. Modern brain scienceresearch has generated massive data sets and many braincomputation models, closely linked with the progress ofinformation science, especially the big data analysis andartificial intelligence. As our data grows in complexity,visualization of brain imaging data can support the anal-ysis of brain structures and their structural and functionalconnectivity in a new manner. The visualization of brain

network connectivity punctuates the landmark advancesin the human connectome research since its beginnings[9].The study of the human brain using both scientific and in-formation visualization techniques has become indispens-able and important means of modern brain science.

The left part of Fig.2 summarizes the categories ofbrain networks according to different spatial resolutions.Such big number of varieties, classifications and imagingmodalities leads to the complexity in designing the visual-ization and visual analysis techniques for brain networks.Corresponding to the study of visualization, we summa-rize five levels of research topics (right part of Fig.2) fromneuron level to individual level, in defining the brain net-work visualization research. In the finest level, neuron-level brain connectivity visualization mainly presents thedatasets on the mesoscale and microscale. The structuralinformation of a single neuron, such as microscopic soma,dendrite, and axon captured by LM techniques, is visual-ized, as well as the synapse connections captured by EMtechniques. Voxel level brain data visualization presentsa coarser granularity of brain connectivity informationby rendering and interacting with the three-dimensionalbrain images. The main image modality used in this vi-sualization level is the MRI.

Fig. 1. Different brain imaging modalities andtheir spatial and temporal resolutions[1]

Fig. 2. Left part: macro-, meso- and micro-scale connectomes that are differen-tiated according to the spatial resolution of imaging modalities[2]. Right

part: five visualization levels that are defined according to the targetedsubject visualized in the brain network visualization technique. The top

three levels are categorized as the macroscopic visualizations according

to their application scope for the whole brain network

To visualize the global connectivity structure of thewhole brain, researchers often use the neural fiber tracingtechniques (aka tractography) on brain imaging data tosynthesize the connectivity information of individual vox-els into continuous streamlines at the nerve fiber level.The direction of each streamline is tangent to the pri-mary eigenvector of the voxel tensor at the voxel location,through which the main direction of the correspondingfiber is represented. Most vector field visualization tech-niques serve at this level.

Connectivity level brain network visualization dis-cretizes neural fibers into intra-brain connections and fur-ther map them to the network structure between the

brain regions. These network structures are mainly di-vided into two types, namely the structural brain networkand the functional brain network, which correspond to theanatomical connectivity and functional connectivity, re-spectively. Different from the three aforementioned levelswhich mainly rely on the theory and methodology of thescientific visualization, network connectivity level visual-ization often depends on methods of information visual-ization. Finally, individual level brain network visualiza-tion focuses on the grouping and inter-group comparisonsof the human subjects’ brain network. The main researchat this level focuses on the visualization and visual com-parison among the brain networks of different classes of

A Survey of Macroscopic Brain Network Visualization Technology 891

subjects, and the brain networks of the same individualin different mental states and life stages.

According to the scope of the brain visualized, we di-vide the above five levels of brain network visualizationinto two major categories, i.e., macroscopic and micro-scopic brain network visualization. Macroscopic visualiza-tion mainly focuses on the visualization of high-level brainnetworks, while microscopic visualization mainly focuseson the visualization of the structure of neuron, such assoma, dendrite, axon and synapse. Although the voxellevel visualization also employs the dataset produced bythe MRI modality, which is classified to the macroscaleconnectome by Sporns et al.[2], the main goal of thesestudies is to develop novel visual metaphors to visually en-code the complex information bounding to a single voxel.Therefore, we categorize the nerve fiber level, connectiv-ity level and individual level visualization as the macro-scopic brain network visualization studied in this survey.To this end, we review the state-of-the-art macroscopicbrain network visualization techniques in the field of bothconnectomics and visualization. We describe the currentbrain network visualization techniques to visualize boththe anatomical and functional connectivity from differentimaging modalities. In Section III, we will further presentand discuss the interactive visualization tools currentlyavailable for use in research and engineering domains. Wefinally conclude with a number of challenges in the ongo-ing macroscopic brain network visualization research.

II. Macroscopic Brain NetworkVisualization

1. Nerve fiber levelThe voxel level visualization (categoried in micro-

scopic visualization) mainly shows the local distribution ofthe connection in the whole brain. At the nerve fiber level,nerve fiber tractography is often used to connect the con-nectivity information of individual voxels into continuousstreamlines to visualize the global connection structure ofthe brain. The direction of each streamline is tangent tothe primary eigenvector of the voxcel tensor at the voxellocation, as a result the streamline can represent the cor-responding nerve fiber. A three-dimensional vector field isformed by collection all the brain nerve fibers. Therefore,the visualization methods of vector fields attracted muchattention at this level.

Firstly, traditional nerve fiber tractography techniquesgenerate definitive streamlines, also known as determin-istic tractography, which can be displayed using a varietyof visual glyphs. The mostly fundamental method uses athree-dimensional linear representation[10]. The later ad-vanced visualization methods introduced the models ofcomputer graphics and GPU accelerating algorithm torepresent the nerve fibers. Petrovic et al. invented tuboid,

a fully-shaded streamtube impostor constructed entirelyon the GPU from streamline vertices to represent theconnectivity between different brain regions[11]. Tuboidsrequire little to no preprocessing or extra space overthe original streamline data. Text labeling technique fortuboids that appear attached to the surface of the tubesprovides adaptive, aesthetically pleasing labels (Fig.3(a)).Merhof et al. used textured triangle strips and pointsprites to obtain a tube-like appearance of streamlinerepresentation[12]. By employing GPU programming, theyachieved real-time rendering of dense bundles encompass-ing a high number of fibers with high visualization qual-ity (Fig.3(b)). Peeters et al. presented a curled hair modelto show extra tensor properties along the fibers in DTIdataset[13]. They also used line illumination and shadow-ing of fibers in order to improve the perception of theirstructure (Fig.3(c)). Also, Zhang et al. employed stream-tubes and streamsurfaces to represent structures with lin-ear diffusion and planar diffusion, respectively[14]. Theresearch on traditional nerve fiber tractography mainlyfocuses on three-dimensional rendering models and com-putational methods that generate interactive and appeal-ing visualization for further surgery planning. Visual ef-ficiency and interactivity are the two main goals to beaddressed by these proposed methods.

Fig. 3. Deterministic tractography of diffusion tensor imag-

ing dataset. The streamlines are represented using

different computer graphic models. (a) The tuboidand its text labeling[11]; (b) A hybrid strategy to

obtain a tube-like streamline appearance that com-bines textured triangle strips and point sprites[12];

(c) A curled hair model to show extra tensor

properties[13]; (d) Line-based ambient occlusion withPhong illumination[19]

An important consideration of traditional nerve fibertractography is the depth perception of fiber tracts. It canbe difficult to perceive the spatial structure and the spa-tial relationship between fiber tracts, because of the twodimensional projection for display and the denseness ofthe fiber tracts itself. Illuminated stream lines/tubes can

892 Chinese Journal of Electronics 2018

alleviate the problem somewhat by employing a realisticshading model, which also increases quality and realismof the resulting images (Fig.3(a–c)). Klein et al.[15] pro-posed distance-encoded contours and shadows for tubesas visual cues for depth ordering. Everts et al.[16] thenpresented illustrative line rendering with depth-dependenthalos based on the observation that line-based techniquesoutperform tube-based renderings in showing more detailsof fibrous structures (Fig.4(c)). Inspired by global illumi-nation models in computer graphics, specifically ambientocclusion[17], Dıaz Garcıa and Vazquez[18] proposed am-bient occlusion halos around the fiber tracts to enhancethe illustrative results by removing clutter and revealingfibers shapes and orientations. Eichelbaum et al.[19] im-proved upon these approaches with line-based ambientocclusion technique that improves spatial and structuralperception of line renderings (Fig.3(d)).

Secondly, high-dimensional multivariate data are oftenattached to the nerve fiber streamline data. One simpleexample is for the fiber bundle names, see Fig.3(a). Thevisualization of these additional data became a hot is-sue in recently. A large number of studies have discussedhow to model and visualize the uncertainty of nerve fibertracking results. Probabilistic tractography yields a prob-ability flow field by sampling the same nerve fiber multipletimes using Monte Carlo method and counting the ratioof streamlines running through individual voxels. The un-certainty visualization of probability flow field is a hottopic in recent years. The main approaches include thevisual encoding of the flow field probability using colorchannels, such as different color scale[20], transparency[21],and saturation[22]. Besides, Rick et al. present probabilis-tic flow field information through enhanced 3D depth per-ception by using virtual reality[23].

Thirdly, bundling and abstraction are another re-search topics for nerve fiber visualization due to the dif-ficulty of real-time and accurate visualization of the mas-sive amounts of nerve fibers in human brain. Unlike tra-ditional deterministic tractography that emphasizes onphoto-realistic rendering of streamline, bundling and ab-straction methods often employ a hierarchical method toaggregate nerve fibers into nerve fiber bundles and drawabstractly according to the connection and geometric ori-entation. Enders et al [24] presented wrapped streamlinesto intuitively represent white matter tracts (Fig.4(a)).Rottger et al.[25] presented the BundleExplorer as a GPU-based focus and context rendering framework for diffusiondata. They proposed a combination of a fiber encompass-ing hull and line rendering to provide insight into inner-bundle fiber configurations as well as to enable bundlecrossing exploration (Fig.4(b)). The aforementioned twomethods employ direct metaphors (wrapping surfaces andfiber encompassing hulls) that represent the bundle ofnerve fibers to highlight their independent characteristics,

on the other hand, indirect bundle mapping can also beemployed to represent dense nerve fibers with a typicalmethod of depth-dependent halo mapping (Fig.4(c))[17].The dense nerve fibers can be clustered into multiplenerve fiber bundles according to their geometric orien-tation. Then the nerve fibers within the same bundle canbe drawn in the same color using just a few hint lines,and silhouette and contours are used to improve the defi-nition of the cluster borders (Fig.4(d))[26]. Everts et al.[27]

achieved fiber abstraction by analyzing the local similar-ity of tract segment directions at different scales usinga stepwise increase of the search range, which results ina better understanding of the brain’s three-dimensionalfiber tract structure.

Fig. 4. The visualization of nerve fiber bundle. (a) Wrapped

streamlines[24]; (b) A fiber encompassing hull and line

rendering from the BundleExplorer[25]; (c) Depth-dependent halo mapping[17]; (d) 49 illustrative clus-

ters using hint lines, silhouette and contours[26]

Finally, interaction design dedicating to lower visualcomplexity and high performance in exploring massiveneural fibers also attracts some interest. Chen et al.[28]

introduced a novel interaction method that augments the3D visualization with a 2D representation containing alow-dimensional embedding of the DTI fibers. The pro-posed embedding preserves the relationship between thefibers and removes the visual clutter that is inherent in3D renderings of the fibers. Blaas et al.[29] presented aneffective approach that offers facilities to aid the userin selecting fiber bundles of interest using multiple con-vex objects. Schultz et al.[30] suggested a novel visualiza-tion metaphor that supports the visual analysis of classi-cal streamlines from fiber tracking by integrating contextfrom anatomical data. The whole brain nerve fiber con-nectivity map typically uses a Focus + Context visualiza-tion method that highlights selected nerve fiber bundles inthe foreground and coarser granularity in the backgroundto present other nerve fibers and brain structure.

2. Connectivity levelThe research at connectivity level focuses on brain net-

work structures between the regions of interest inside the

A Survey of Macroscopic Brain Network Visualization Technology 893

whole brain. The nerve fibers will be firstly discretizedinto inter-brain connections and further be constructed toform the brain network structure. As mentioned before,structural brain network and functional brain networkare the main categories of the brain network structures(Fig.5)[31]. According to its definition, effective brain net-work can be understood as an extend of functional brainnetwork, therefore, we treat the visualization methods asthe same category. In a brain network, the nodes representthe structural Regions of interest (sROI) or the functionalRegions of interest (fROI), which are generally extractedaccording to a certain physical map or a functional map,respectively. The connection between nodes is the inter-region nerve fiber connection for structural brain network,while it is the inter-region activation timing dependencyfor functional brain network. The brain network usuallyis a undirected weighted network, since the connectionsare of weight but no direction.

Fig. 5. Graph theoretical analysis of structural and functional

networks in human neuroimaging data[31]

The connectivity level visualization of brain networkusually employs the methods of information visualization,while the nerve fiber level visualization, as well as the neu-ron level and voxel level visualization, mainly relies onscientific visualization. In particular, the visualization atthis level employs a lot of the classical metaphors and lay-out algorithms in the field of graph visualization[32], sincethe brain networks are usually equivalent to weightednetworks. Several results that employ a classical node-edge visual metaphor are shown in Fig.6. Xia et al.[33]

implemented a software tool called BrainNet Viewer toillustrate human connectomes as ball-and-stick models.BrainNet Viewer also draws the brain surface, nodes andedges in sequence and displays brain networks in multipleviews (Fig.6(a)). Worsley et al.[34] compared two commonmethods (thresholding correlations and singular value de-composition) for detecting functional connectivity using athree-dimensional interactive exploring space (Fig.6(b)).

For functional brain network visualization, due to thecomplexity of potential connections between fROIs andunconstrained mappings between real and logical brainspace, graph layout algorithms are often introduced tomap functional brain network to logical space (usually in2D space) to optimize the aesthetics of layout. Salvador etal.[35] depicted a neurophysiological architecture of func-tional magnetic resonance images of human brain by us-ing a dimensionality reduction method (such as MDS) tomap the real brain space into a two-dimensional logicalspace. It makes the visualization more clear to be un-derstood by the compromise between the authenticity ofthe structural layout and the legibility of the functionallayout (Fig.6(c)). Analogously, Achard et al.[36] createdundirected graphs of functional brain networks directlyafter thresholding the wavelet correlation matrices. Theyconstructed a more clear topological map of a small-worldhuman brain functional network (Fig.6(d)).

Fig. 6. Connectivity level visualization with node-edgemetaphor. (a) Ball-and-stick models used in BrainNet

Viewer[33]; (b) The three-dimensional interactive ex-

ploring space used by Worsley et al.[34]; (c) Multidi-mensional scaling (MDS) solution for healthy group

mean partial correlation matrix[35]; (d) Topologicalmap of a small-world human brain functional network

created by thresholding the scale 4 wavelet correlation

matrix representing functional connectivity in the fre-quency interval 0.03–0.06Hz[36]

New visual metaphors except the classical node-edgethat represent the brain network connectivity are inventedand introduced in brain network visualization. The adja-cency matrix is another common visual metaphor usedin functional brain network visualization. As shown inFig.7(a)[37], each row/column of the matrix can repre-sent a functional region of interest in brain space, whilethe filled color of each cell can represent the strength ofthe neural connectivity between two regions of interest.

894 Chinese Journal of Electronics 2018

The detailed visual design of the matrix cell can also beused to compare the differences between individual struc-tural/functional brain networks (see Section II.3). Theadjacency matrix metaphor is more suitable for the func-tional brain network visualization because of the highercomplexity of the functional connectivity than structuralconnectivity. Sanz-Arigita et al.[38] used a matrix-basedgraph analysis on resting-state condition fMRI data tostudy connectivity changes in early Alzheimer’s disease(AD) (Fig.7(b)). Besides, ChordMap (Fig.7(c))[39] andNodeTrix (Fig.7(d))[40] metaphors were introduced byMcGonigle et al. and Yang et al., respectively, to simul-taneously represent and visualize the brain network con-nectivity and the hierarchical structures of the brain.

Fig. 7. Connectivity level visualization with new visualmetaphors. (a) Adjacency matrix outperform node-

edge in supporting weighted graph comparison

tasks[37]; (b) Synchronization matrix of AD group[38];(c) ChordMap[39]; (d) NodeTrix[40]

In particular, the structure of functional brain networkis usually time-varying. Inspired by the way people com-prehend and manipulate physical cubes, Bach et al.[41]

introduced matrix cube metaphor (Fig.8(a)) for dynamicnetworks. The third dimension of time is added to the two-dimensional adjacency matrix, and specifically designedinteractions are used to guide the user to discover theimportant two-dimensional visualizations that emphasizespecific aspects of the dynamic network suited to par-ticular analysis tasks. They also proposed small multip-iles (Fig.8(b))[42] and time curves (Fig.8(c))[43] metaphorsconsecutively in their subsequent research work. The mul-tipiles metaphor is based on the physical analogy of pil-ing adjacency matrices, each one representing a singletemporal snapshot, while the proposed ‘piling’ metaphorpresents a hybrid interface to help neuroscientists investi-gate changes in brain connectivity networks over severalhundreds of snapshots[42]. Time curves is a general ap-

proach for visualizing patterns of evolution in temporaldata. It pays more attention to show the time-varyingcharacteristics of dynamic network structure. By map-ping a set of timing networks into curves that connectmultiple nodes, the time-varying similarities between dy-namic functional brain networks or individual differencesare highlighted[43].

Fig. 8. New visual metaphors for dynamic functional brainnetworks. (a) The matrix cube metaphor[41] ; (b) The

small multipiles[42]; (c) The time curves[43]

3. Individual levelFinally, the individual level visualization of brain net-

work mainly focuses on the research of grouping and inter-group comparison of human brain networks, with the ma-jor aim of assisting neuroscientists on the mechanism ofsome certain diseases. In contrast to the aforementionedfour levels, the main work at this level is about visual com-parison of brain networks among different individuals, oramong different states and phases of the same individ-ual. In other words, researchers paid more attentions onfunctional brain network at this level.

Fair et al.[44] reported their research findings that, overdevelopment, the organization of multiple functional net-works shifts from a local anatomical emphasis in childrento a more distributed architecture in young adults. Asshown in Fig.9(a), regions in children are largely orga-nized by their anatomical location, but over age anatomi-cally clustered regions segregate (top row), while the moredistributed adult functional networks in children are inmany ways disconnected and they integrate over develop-ment (bottom row).

Grouping according to the individual attributes is usu-ally employed before visualization and visual analysisof functional brain networks. Daianu et al.[45] analyzedanatomical connectivity based on 3-T diffusion-weightedimages from 111 subjects, which are categorized in 4groups, to better understand how AD affects brain con-

A Survey of Macroscopic Brain Network Visualization Technology 895

nectivity. They performed whole brain tractography basedon the orientation distribution functions, and compiledconnectivity matrices showing the proportions of detectedfibers interconnecting 68 cortical regions. Fig.9(b) showsthe asymmetries in anatomical connectivity in healthy el-derly (top left), early (top right) and late mild cognitiveimpairment (bottom left), and AD (bottom right), wherethe colors encode p-values from their regression modelcomparing the left and right hemisphere connectivity ma-trices.

The main difficulty in the visualization and groupcomparison of individual-level brain networks is the align-ment of network nodes in different individuals. So far, themeasurement of the brain network at the microscale (typ-ically for EM) and mesoscale (typically for LM) is stillconfined to a very small sub-region of the whole brain,therefore, the individual level brain network visualizationand analysis are performed mainly based on the datasetacquired at the macroscale. The same set of brain net-work nodes of the whole brain atlas for all individuals isclassified to reduce the complexity and difficulty of com-paring macroscopic brain networks. Zalesky et al.[46] pro-posed a Network-based statistic (NBS) to identify differ-ences in brain networks. Fig.9(c) shows their visualiza-tion to present the comparison between the results of thenetwork-based statistic approach (bottom row) and theresults of link-based FWE control method (top row). Theorange nodes correspond to connections that are part ofthe contrast (which is absent in Fig.9(c), refer Ref.[46] formore details) and correctly identified as true positives,while the red nodes correspond to connections that werenot part of the contrast but incorrectly identified.

Recently, the research of individual level brain net-work visualization mainly focus on two issues: 1) how tovisualize the grouping of brain networks; and 2) how to vi-sually compare the brain networks of different groups. Asshow in Fig.10(a)[47], the strength of selected connectionsbetween network nodes of the same group of individuals isaveraged and then mapped to the visual channels, such ascolor and width, of the edges of the brain network. How-ever, this method does not show the distribution of theconnection strength within the group. Wang et al.[48] pro-posed a solution inspired by scientific visualization thatsuperimposes individual brain networks in the same groupto form a volumetric dataset, and then visualize suchdataset using volume rendering to show the connectionstrength distribution (Fig.10(b)). Visual clutter may occurdue to the high density of connections of human brain net-works when we employ the node-edge metaphors. Alperset al.[37] systematically studied the user analysis abilitiesof node-edge and adjacency matrix metaphors in a varietyof visual comparison designs, and reported that the use of

Fig. 9. Individual level visualization of brain networks. (a)Over age the graph architecture matures from a local

organization to a distributed organization[44] ; (b) Theasymmetries in anatomical connectivity in four groups

of subjects[45]; (c) The group comparison between the

results of the network-based statistic approach and theresults of link-based FWE control method[46]

adjacency matrix metaphor and their superimposed de-sign can significantly improve the user analysis ability.Shi et al.[49] also analyzed the visual comparison perfor-mance of graph-based brain network visualization, andproposed an integrated framework to orchestrate compu-tational models with comprehensive data visualizationson the human brain network, which greatly improves theperformance of visual comparison of the brain networks(Fig.10(c)).

III. Softwares in Brain NetworkVisualization Community

There have been multiple applications and softwarepackages that are designed to process, visualize and visu-ally analyse brain network datasets. Some lists of them arefound at wikipedia.org∗∗,∗∗∗ . In this section, we enumer-ate some typical and popular tools used in brain networkvisualization community.

An early and probably the most popular softwarepackage is Diffusion Toolkit (DTK, http://trackvis.org),which is designed for diffusion imaging data processingand tractography[50]. DTK is implemented in C++, usingQt (https://www.qt.io/ ) for the graphical user interfaceand Visualization Toolkit (VTK, https://www.vtk.org/ )for the visualization core. It can handle large whole braintractography datasets including Diffusion tensor imag-ing (DTI) data, High angular resolution diffusion imag-ing (HARDI) data, Diffusion spectrum imaging (DSI)data and Q-ball imaging data. Two components, diffusion

∗∗ https://en.wikipedia.org/wiki/List of neuroimaging software∗∗∗ https://en.wikipedia.org/wiki/List of functional connectivity software

896 Chinese Journal of Electronics 2018

toolkit itself and the visualization program called Track-Vis (Fig.11), are comprised in the software package andare cooperating in reconstruction, fiber tracking, analysisand visualization of brain tractography data. Some of themost striking examples of tractography can be found intheir subsequent articles, such as Ref.[51].

Fig. 10. New approaches to visual analytics of individual levelbrain networks. (a) Visual representation of the av-

erage strength of connections[47] ; (b) Visual represen-tation of the strength distribution of connections by

superimposing individual brain networks in the same

group[48]; (c) The framework for the visual explorationof multi-label brain networks and the visual compar-

ison among brain networks across different subjectgroups[49]

Fig. 11. A snapshot of TrackVis

Chen et al.[28] developed an effective interaction mode(Fig.12) that combines 3D, 2D, and statistical views tobroaden the user’s exploratory space, which was designedto enable efficient browsing, manipulation, and quanti-tative analysis of DTI fiber tracts. Intuitive interactionssuch as rotation, lens viewing, coloring, slicing, and se-lection are provided in a three-dimensional view (left sideof Fig.12), as well as a traditional fiber tracts visualiza-tion. Specially, their two-dimensional point embedding

(right side of Fig.12) of the DTI fibers preserves therelationship between the fibers and removes the visualclutter that is inherent in 3D renderings of the fibers.Later, Jianu et al.[52] proposed two-dimensional neuralmaps (Fig.13) for exploring connectivity. Rather than us-ing two-dimensional point embedding as aforementioned,the planar path representation (bottom panels in Fig.14)may be more intuitive and easier to use and learn.

Fig. 12. A snapshot of DTI fiber exploration system from chen

et al.[28]

Fig. 13. An interactive analysis system using linked views andplanar tract-bundle projections[52]

Fig. 14. The graphical user interface of OpenWalnut

Most of softwares in research community may largelyaim either at visualization researher or the neuro-scientist,who own expert knowledge. Such softwares are usuallydesigned to answer a certain set of questions and han-dle certain data sets from different imaging modalities,with accordingly designed graphical user interfaces. Inopen source community, softwares may be designed tobe used with different signals, modalities and data types,aiming at a general purpose of data processing workflow

A Survey of Macroscopic Brain Network Visualization Technology 897

and interactive rendering and explorability user interface.3D Slicer is a free open source software application formedical image computing, which not only supports versa-tile visualizations but also provides advanced functional-ity such as automated segmentation and registration for avariety of application domains[53]. OpenWalnut is anoteropen source software package in brain network visualiza-tion community[54]. It has a modular design and a built-in pipelining engine with drag-and-drop graphical userinterface (Fig.14). OpenWalnut integrates several well-know toolkits, such as Eigen (http://eigen.tuxfamily.org),OpenSceneGraph (http://www.openscenegraph.org) andBioSig (http://biosig.sourceforge.net), etc., to utilize theirstrengths and make itself more extensible and flexi-ble. Due to the large scale of the amount of con-nectivity data, Eklund et al.[55] implemented a GPUaccelerated interactive interface for exploratory func-tional connectivity analysis. The connectivity analysiswas written in Open computing language (OpenCL,https://www.khronos.org/opencl/ ). The correlation mapof all voxels in fMRI data can be updated in real-time,which greatly facilitates the analysis of the connectiv-ity patterns in the brain. The interactive tool was im-plemented in the MeVisLab (https://www.mevislab.de)[56]

software environment, which also supports an intuitivepipeline editing with abundant data processing, interac-tion and visualization components.

Previously described softwares in this section mainlyfocus on the visualization of structural connectivity, how-ever, the inherent multi-modality of connectome datasetsposes new challenges for data organization, integrationand sharing, as well as knowledge based data processing,such as data mining, exploration and comparison. Ger-hard et al.[57] designed and implemented a set of extensi-ble and also open source neuroimaging tools, called Con-nectome Viewer Toolkit. The plugin architecture of itselfsupports extensions with network analysis packages. Thetoolkit is written in Python, which also make it much eas-ier to integrate and utilize the numerous existing powerfullibraries, such as graph theory based analysis and otherdata mining algorithms, from scientific python commu-nity. The popular brain network visualization tool men-tioned in Section II.2, called BrainNet Viewer, employa graph-based network visualization to illustrate humanconnectomes as ball-and-stick models[33]. The toolbox isnotable for its diversity with both network mapping andsurface-based data presentation. BrainNet Viewer is im-plemented using MATLAB with a friendly graphical userinterface, helps researchers to visualize brain networks inan easy, flexible and quick manner.

Although impressive improvements in software distri-bution has been made in recent decades, there is still awide gap between software developers and users. Issues of

installation, configuration and running of softwares maybe prone to occur due to complexity of operation sys-tem environments, including but not limited to softwaredependency, hardware driver version, etc. However, re-cent advancements in web technologies give promise of achange for the better. The improvements of JavaScriptand WebGL make the developing of full-fledged three di-mensional graphics applications that can run in a generalweb browser (e.g. Google Chrome) possible. Nowadays, aweb browser based application can offer a friendly graph-ical experience comparable to most of traditional stand-alone desktop client. Meanwhile, the issues of burdensomeconfiguration of runtime environment are also overcomebecause the applications will run in a web browser as aweb service and some native JavaScript programs whichexecute on a built-in JavaScript engine. Haehn et al.[58]

contributed “The X Toolkit”(XTK), the first JavaScript-based framework for visualizing and interacting with med-ical imaging data using WebGL. The toolkit is geared to-wards powerful scientific visualization and provides a sim-ple Application programming interface (API) which hideslow-level elements of WebGL from users.

IV. Conclusions

In this survey, we first categorize the visualizationtechniques on interpreting brain networks into five lev-els, according to the diverging research targets rangingfrom a single neuron up to fibers, networks, and groupsof brain networks. We focus on the macroscopic brainnetwork visualization techniques where the whole brainnetwork, including its structural connectivity and func-tional connectivity, attracts the most attention. As themassive brain imaging data nowadays is captured by awide variety of modalities, it is intuitive to depend onthe visualization to arrive at a thorough understandingof such digital data. In other words, learning how thehuman brain works is an important neuroscience goal,which leads to the emerging discipline of connectomicsthat analyzes the neuronal connectivity. However, visual-izing the whole brain network is a challenging task dueto the complexity of the human brain neuronal system,which has an estimated number of 1011 neurons and 1015

connections[2]. Visual clutter may be prone to exist whenwe visualize such a whole brain network. Abstraction, pro-jection, bundling, and many other techniques are widelyemployed to make the visualization result more readable.These challenges demand attentions on many researchcompromise, such as thoroughness/readability, aestheticappeal/information content, and faithfulness/abstraction,etc. In addition, there also are ethical concerns if we diveinto the brain connectivity research due to the uniquenessof the human brain compared with the other body blocks.

There are several software from the brain network vi-sualization community offered as powerful tools for the

898 Chinese Journal of Electronics 2018

interactive exploration and analysis of neuronal connec-tivity. Advanced neuroinformatics tools are also requiredfor the full-scale connectome mapping, analysis, and vi-sualization, with intuitive interaction design on selection,rotation, moving and labeling, etc. The inherent multi-modality, as well as the huge data volume, poses manynew challenges on the data cleaning, storage, integration,registration, and organization. In the early stage of thebrain network visualization research, scientific visualiza-tion methods, in combination with the image analysistechniques, were mostly used to emphasize the renderingand interactive exploration of faithful structural connec-tivity. More recently, the latest research has focused onintroducing information visualization methods to visuallydisplay and analyze brain networks through novel visualmetaphors, effective data mining algorithms, and accu-rate machine learning models. The construction of a gen-eral platform for the visual analysis of macroscopic brainnetwork in multiple levels is also a promising research di-rection.

References

[1] H. Pfister, V. Kaynig, C.P. Botha, et al., “Visualization in

connectomics”, Scientific Visualization, Springer, pp.221–245,2014.

[2] O. Sporns and J.D. Zwi, “The small world of the cerebral cor-tex”, Neuroinformatics, Vol.2, No.2, pp.145–162, 2004.

[3] K.J. Friston, “Functional and effective connectivity in neu-roimaging: A synthesis”, Human Brain Mapping, Vol.2, No.1–2,

pp.56–78, 1994.

[4] D.L. Schomer and F.L. Da Silva, Niedermeyer’s Electroen-

cephalography: Basic Principles, Clinical Applications, and Re-lated Fields, Lippincott Williams & Wilkins, 2012.

[5] M. Hamalainen, R. Hari, R. J. Ilmoniemi, et al., “Magneto

encephalography–theory, instrumentation, and applications to

noninvasive studies of the working human brain”, Reviews ofModern Physics, Vol.65, No.2, p.413, 1993.

[6] N.K. Logothetis, J. Pauls, M. Augath, et al., “Neurophysio-

logical investigation of the basis of the fMRI signal”, Nature,

Vol.412, No.6843, p.150, 2001.

[7] D.L. Bailey, D.W. Townsend, P.E. Valk, et al., Positron Emis-sion Tomography, Springer, 2005.

[8] D.W. McRobbie, E.A. Moore and M.J. Graves, MRI from Pic-ture to Proton, Cambridge University Press, 2017.

[9] D.S. Margulies, J. Bottger, A. Watanabe, et al., “Visualizing thehuman connectome”, NeuroImage, Vol.80, pp.445–461, 2013.

[10] M. Zockler, D. Stalling and H.-C. Hege, “Interactive visualiza-

tion of 3D-vector fields using illuminated stream lines”, Proc.

IEEE Visualization, pp.107–113, 1996.

[11] V. Petrovic, J. Fallon and F. Kuester, “Visualizing whole-brain DTI tractography with GPU-based tuboids and LoD man-

agement”, IEEE Transactions on Visualization and Computer

Graphics, Vol.13, No.6, pp.1488–1495, 2007.

[12] D. Merhof, M. Sonntag, F. Enders, et al., “Hybrid visualizationfor white matter tracts using triangle strips and point sprites”,

IEEE Transactions on Visualization and Computer Graphics,

Vol.12, No.5, pp.1181–1188, 2006.

[13] T. Peeters, A. Vilanova and R. ter Haar Romeny, “Visualiza-tion of DTI fibers using hair-rendering techniques”, Proc. ASCI,

pp.66–73, 2006.

[14] S. Zhang, C. Demiralp and D.H. Laidlaw, “Visualizing diffu-sion tensor MR images using streamtubes and streamsurfaces”,

IEEE Transactions on Visualization and Computer Graphics,

Vol.9, No.4, pp.454–462, 2003.

[15] J. Klein, F. Ritter, H.K. Hahn, et al., “Brain structure visual-

ization using spectral fiber clustering”, SIGGRAPH ResearchPosters, Article No.168, 2006.

[16] M.H. Everts, H. Bekker, J.B. Roerdink, et al., “Depth-

dependent halos: Illustrative rendering of dense line data”,IEEE Transactions on Visualization and Computer Graphics,

Vol.15, No.6, pp.1299–1306, 2009.

[17] S. Zhukov, A. Iones and G. Kronin, “An ambient light illu-

mination model”, G. Drettakis and N. Max Eds., Rendering

Techniques’98, Vienna: Springer Vienna, pp.45–55, 1998.

[18] J. Dıaz Garcıa and P.-P. Vazquez, “Fast illustrative visualiza-

tion of fiber tracts”, G. Bebis, R. Boyle, B. Parvin, et al., Eds.,

Advances in Visual Computing, Berlin, Heidelberg: SpringerBerlin Heidelberg, pp.698–707, 2012.

[19] S. Eichelbaum, M. Hlawitschka and G. Scheuermann, “Lineao–improved three-dimensional line rendering”, IEEE Transactions

on Visualization and Computer Graphics, Vol.19, No.3, pp.433–

445, 2013.

[20] G.J. Parker, H.A. Haroon and C.A. Wheeler-Kingshott, “A

framework for a streamline-based probabilistic index of connec-tivity (PICo) using a structural interpretation of MRI diffu-

sion measurements”, Journal of Magnetic Resonance Imaging,

Vol.18, No.2, pp.242–254, 2003.

[21] A. von Kapri, T. Rick, S. Caspers, et al., “Evaluating a vi-

sualization of uncertainty in probabilistic tractography”, Med-ical Imaging 2010: Visualization, Image-Guided Procedures,

and Modeling, International Society for Optics and Photonics,

Vol.7625, p.762534, 2010.

[22] P. Hermosilla, R. Brecheisen, P.-P. Vazquez, et al., “Uncertainty

visualization of brain fibers”, I. Navazo and G. Patow, Eds.,

Spanish Computer Graphics Conference, the Eurographics As-sociation, 2012.

[23] T. Rick, A. von Kapri, S. Caspers, et al., “Visualization of prob-abilistic fiber tracts in virtual reality”, Studies in Health Tech-

nology & Informatics, Vol.163, pp.486–492, 2011.

[24] F. Enders, N. Sauber, D. Merhof, et al., “Visualization of whitematter tracts with wrapped streamlines”, Visualization VIS 05.

IEEE, pp.51–58, 2005.

[25] D. Rottger, D. Merhof and S. Muller, “The BundleExplorer:

A focus and context rendering framework for complex fiber

distributions”, T. Ropinski, A. Ynnerman, C. Botha, et al.,Eurographics Workshop on Visual Computing for Biology and

Medicine, the Eurographics Association, pp.1–8, 2012.

[26] R. Otten, A. Vilanova and H. Van De Wetering, “Illustrative

white matter fiber bundles”, Computer Graphics Forum, Wiley

Online Library, Vol.29, No.3, pp.1013–1022, 2010.

[27] M.H. Everts, E. Begue, H. Bekker, et al., “Exploration of the

brain’s white matter structure through visual abstraction andmulti-scale local fiber tract contraction”, IEEE Transactions on

Visualization and Computer Graphics, Vol.21, No.7, pp.808–

821, 2015.

[28] W. Chen, Z. Ding, S. Zhang, et al., “A novel interface for inter-

active exploration of DTI fibers”, IEEE Trans. on Visualization

and Computer Graphics, Vol.15, No.6, pp.1433–1440, 2009.

[29] J. Blaas, C.P. Botha, B. Peters, et al., “Fast and reproducible

fiber bundle selection in DTI visualization”, Visualization (VIS05), IEEE, pp.59–64, 2005.

[30] T. Schultz, N. Sauber, A. Anwander, et al., “Virtual klingler

dissection: Putting fibers into context”, Computer Graphics Fo-rum, Wiley Online Library, Vol.27, No.3, pp.1063–1070, 2008.

[31] E. Bullmore and O. Sporns, “Complex brain networks: Graphtheoretical analysis of structural and functional systems”, Na-

ture Reviews Neuroscience, Vol.10, No.3, p.186, 2009.

A Survey of Macroscopic Brain Network Visualization Technology 899

[32] G.D. Battista, P. Eades, R. Tamassia, et al., Graph Drawing:Algorithms for the Visualization of Graphs, Prentice Hall PTR,

1998.

[33] M. Xia, J. Wang and Y. He, “Brainnet viewer: Anetwork visu-alization tool for human brain connectomics”, PloS One, Vol.8,

No.7, doi: 10.1371/journal.pone.0068910, 2013.[34] K.J. Worsley, J.-I. Chen, J. Lerch, et al., “Comparing functional

connectivity via thresholding correlations and singular value de-

composition”, Philosophical Trans. of the Royal Society of Lon-don B: Biological Sciences, Vol.360, No.1457, pp.913–920, 2005.

[35] R. Salvador, J. Suckling, M.R. Coleman, et al., “Neurophys-iological architecture of functional magnetic resonance images

of human brain”, Cerebral Cortex, Vol.15, No.9, pp.1332–1342,

2005.[36] S. Achard, R. Salvador, B. Whitcher, et al., “A resilient, low-

frequency, small-world human brain functional network withhighly connected association cortical hubs”, Journal of Neu-

roscience, Vol.26, No.1, pp.63–72, 2006.

[37] B. Alper, B. Bach, N. Henry Riche, et al., “Weighted graph com-parison techniques for brain connectivity analysis”, Proceedings

of the SIGCHI Conference on Human Factors in ComputingSystems, ACM, pp.483–492, 2013.

[38] E.J. Sanz-Arigita, M.M. Schoonheim, J.S. Damoiseaux, et al.,

“Loss of ‘small-world’ networks in Alzheimer’s disease: Graphanalysis of FMRI resting-state functional connectivity”, PloS

One, Vol.5, No.11, p.e13788, 2010.[39] J. McGonigle, A.L. Malizia and M. Mirmehdi, “Visualizing

functional connectivity in fMRI using hierarchical edge bun-

dles”, New Phytologist, Vol.206, No.3, pp.1038–1050, 2011.[40] X. Yang, L. Shi, M. Daianu, et al., “Blockwise human brain net-

work visual comparison using nodetrix representation”, IEEETransactions on Visualization and Computer Graphics, Vol.23,

No.1, pp.181–190, 2017.

[41] B. Bach, E. Pietriga and J.-D. Fekete, “Visualizing dynamic net-works with matrix cubes”, Proc. of the SIGCHI Conf. on Hu-

man Factors in Computing Systems, ACM, pp.877–886, 2014.[42] B. Bach, N. Henry-Riche, T. Dwyer, et al., “Small multipiles:

Piling time to explore temporal patterns in dynamic networks”,

Computer Graphics Forum, Wiley Online Library, Vol.34, No.3,pp.31–40, 2015.

[43] B. Bach, C. Shi, N. Heulot, et al., “Time curves: Folding time tovisualize patterns of temporal evolution in data”, IEEE Trans-

actions on Visualization and Computer Graphics, Vol.22, No.1,

pp.559–568, 2016.[44] D.A. Fair, A.L. Cohen, J.D. Power, et al., “Functional brain

networks develop from a “local to distributed” organization”,

PLoS Computational Biology, Vol.5, No.5, p.e1000381, 2009.[45] M. Daianu, N. Jahanshad, T.M. Nir, et al., For the Alzheimer’s

Disease Neuroimaging Initiative, “Breakdown of brain connec-tivity between normal aging and alzheimer’s disease: A struc-

tural k-core network analysis”, Brain Connectivity, Vol.3, No.4,

pp.407–422, 2013.[46] A. Zalesky, A. Fornito and E.T. Bullmore, “Network-based

statistic: Identifying differences in brain networks”, Neuroim-age, Vol.53, No.4, pp.1197–1207, 2010.

[47] L. Shi, H. Tong and X. Mu, “Brainquest: Perception-guided

brain network comparison”, 2015 IEEE International Confer-ence on Data Mining (ICDM), IEEE, pp.379–388, 2015.

[48] J. Wang, S. Fang, H. Li, et al., “Multigraph visualization forfeature classification of brain network data”, Proceedings of the

EuroVis Workshop on Visual Analytics, Eurographics Associa-

tion, pp.61–65, 2016.[49] L. Shi, H. Tong, M. Daianu, et al., “Visual analysis of brain

networks using sparse regression models”, ACM Trans. Knowl.Discov. Data, Vol.12, No.1, pp.5:1–5:30, 2018.

[50] R. Wang, T. Benner, A.G. Sorensen, et al., “Diffusion toolkit:A software package for diffusion imaging data processing and

tractography”, Proc. of the 15th Annual Meeting of ISMRM,

Vol.15, p.3720, 2007.

[51] V.J. Wedeen, D.L. Rosene, R. Wang, et al., “The geomet-ric structure of the brain fiber pathways”, Science, Vol.335,

No.6076, pp.1628–1634, 2012.

[52] R. Jianu, C. Demiralp and D.H. Laidlaw, “Exploring brain con-nectivity with two-dimensional neural maps”, IEEE Transac-

tions on Visualization and Computer Graphics, Vol.18, No.6,

pp.978–987, 2012.

[53] A. Fedorov, R. Beichel, J. Kalpathy-Cramer, et al., “3D sliceras an image computing platform for the quantitative imaging

network”, Magnetic Resonance Imaging, Vol.30, No.9, pp.1323–1341, 2012.

[54] S. Eichelbaum, M. Hlawitschka and G. Scheuermann, “Open-

walnut: An open-source tool for visualization of medical and

bio-signal data”, Biomedizinische Technik. Biomedical Engi-neering, Vol.58, No.5, doi: 10.1515/bmt-2013-4183, 2013.

[55] A. Eklund, O. Friman, M. Andersson, et al., “A GPU acceler-

ated interactive interface for exploratory functional connectivityanalysis of fMRI data”, 2011 18th IEEE International Confer-

ence on Image Processing (ICIP), IEEE, pp.1589–1592, 2011.

[56] F. Heckel, M. Schwier and H.-O. Peitgen, “Object-oriented ap-

plication development with mevislab and python”, GI Jahresta-gung, Vol.154, pp.1338–51, 2009.

[57] S. Gerhard, A. Daducci, A. Lemkaddem, et al., “The connec-

tome viewer toolkit: An open source framework to manage,analyze, and visualize connectomes”, Frontiers in Neuroinfor-

matics, Vol.5, No.3, doi: 10.3389/fninf.2011.00003, 2011.

[58] D. Haehn, N. Rannou, B. Ahtam, et al., “Neuroimaging in the

browser using the x toolkit”, Frontiers in Neuroinformatics,doi: 10.3389/conf.fninf.2014.08.00101, 2014.

CHEN Weifeng received the B.E.

degree in science from Zhejiang Universityof Technology and the Ph.D. degree in com-

puter science and technology from ZhejiangUniversity. He is currently an assistant pro-

fessor of Zhejiang University of Finance &

Economics. His research interests includescientific visualization and information vi-

sualization. (Email: cwf818@zufe.edu.cn)

SHI Lei (corresponding author) re-ceived the B.S., M.S., and Ph.D. degrees

from Tsinghua University, in 2003, 2006,

and 2008, respectively. He is a professorin SKLCS, Institute of Software, Chinese

Academy of Sciences. His research interestsinclude visual analytics and data mining.

He is the recipient of the IBM Research

Accomplishment Award on Visual Analyt-ics and the VAST Challenge Award twice

in 2010 and 2012. (Email: shil@ios.ac.cn)

CHEN Wei is a professor inthe State Key Lab of CAD&CG, Zhe-

jiang University. His research interestsinclude visualization and visual analy-

sis, and has published more than 30

IEEE/ACM Transactions and IEEE VISpapers. For more information, please refer

to http://www.cad.zju.edu.cn/home/chenwei. (Email: chenwei@cad.zju.edu.cn)