Visibility-Aware Direct Volume Rendering - ycwu

Mak WH, Wu Y, Chan MY et al. Visibility-aware direct volume rendering. JOURNAL OF COMPUTER SCIENCE AND

TECHNOLOGY 26(2): 217–228 Mar. 2011. DOI 10.1007/s11390-011-1124-9

Visibility-Aware Direct Volume Rendering

Wai-Ho Mak1 (麦伟豪), Yingcai Wu (巫英才)2,∗, Ming-Yuen Chan1 (陈明远)and Huamin Qu1 (屈华民), Member, IEEE

1Department of Computer Science and Engineering, The Hong Kong University of Science and TechnologyHong Kong, China

2Department of Computer Science, The University of California, Davis, CA, U.S.A.

E-mail: {wallacem, pazuchan, huamin}@cse.ust.hk; [email protected]

Received July 24, 2009; revised November 28, 2010.

Abstract Direct volume rendering (DVR) is a powerful visualization technique which allows users to effectively exploreand study volumetric datasets. Different transparency settings can be flexibly assigned to different structures such thatsome valuable information can be revealed in direct volume rendered images (DVRIs). However, end-users often feel thatsome risks are always associated with DVR because they do not know whether any important information is missing fromthe transparent regions of DVRIs. In this paper, we investigate how to semi-automatically generate a set of DVRIs and alsoan animation which can reveal information missed in the original DVRIs and meanwhile satisfy some image quality criteriasuch as coherence. A complete framework is developed to tackle various problems related to the generation and qualityevaluation of visibility-aware DVRIs and animations. Our technique can reduce the risk of using direct volume renderingand thus boost the confidence of users in volume rendering systems.

Keywords visualization systems and software, volume visualization

1 Introduction

Volume visualization enables people to explore andstudy volumetric data by using interactive computergraphics techniques. Among the many volume visua-lization techniques, direct volume rendering (DVR)is best known for its flexibility. With the control-lable transparency and color settings of voxels, DVRprovides an informative 2D view of volumetric data re-vealing the relationship among different 3D structures,and also helping users gain insights into the data. Themapping of density values to transparency and color isdetermined by transfer functions which are core partsof DVR.

Over the last two decades, many researchers haveworked on various related problems of direct volumerendering. In particular, much work has been con-ducted on transfer function design. Also, tremendousefforts have been put into improving efficiency andinterface. Most research work focuses on detection offeatures in the data. In most commercial systems, somepreset transfer functions are used to ease the difficultiesof using DVR for non-expert users. The time perfor-mance of DVR has also been dramatically improved and

there already exist some intuitive interfaces for interac-tive volume exploration.

While many technical issues of direct volume ren-dering have been studied extensively, this sophisticatedtechnique is not yet being widely used in practice. Us-ability is one main issue which has already attractedmuch attention from the visualization community. An-other major obstacle, which currently has not drawnadequate attention, is that users are unsure of the safetyaspect of using this flexible technique in volume explo-ration. Because there are often multiple structures andtransparency is used in direct volume rendered images,some structures or parts of the structures may becomeover transparent or hidden, users may not be aware ofthe existence of those structures or parts in the renderedimages. During the whole visualization process, thereis no guarantee that all or most of the information hasbeen effectively revealed. Some important informationmay be missing as some voxels can be assigned withvery low opacity during the entire volume explorationprocess. Even if some voxels are given high opacityvalues, they may not be well revealed due to the oc-clusion. As a result, a certain degree of risk of missingimportant information is always associated with DVR.

Regular PaperThe work is supported in part by Hong Kong RGC CERG under Grant No. 618705.∗Corresponding Author©2011 Springer Science +Business Media, LLC & Science Press, China

218 J. Comput. Sci. & Technol., Mar. 2011, Vol.26, No.2

The safety issue is critical for a wider adoption of directvolume rendering in practice. However, to the best ofour knowledge, it has not been well addressed before.

There are some trivial approaches for this problem.For example, we can directly give all the invisible voxelshigh opacity values, but obviously it cannot really solvethe problem because some voxels may still be invisibledue to occlusion. Another possibility is that we gradu-ally remove voxels with high visibility, then the hiddenvoxels may be eventually revealed. But since too muchcontext is removed, users may only see a set of voxelsfrom which no structure-level information can be iden-tified. All these trivial approaches would only generatemessy direct volume rendered images that are not in-formative and cannot reveal missing information well.

In this paper we investigate a systematic way topresent missing information in a region of interest(ROI) to users. We assume that some preset, user-tuned, or automatically generated transfer functionshave been used to reveal some information in the data.Viewpoint, lighting, and structure colors are assumedto be already carefully chosen. Users can use our sys-tem to generate a set of DVRIs, which can then be usedto compose a comprehensive animation, that effectivelyreveal the missing information in the ROI.

The generated DVRIs should meet the following re-quirements:• Guaranteed Visibility. Interesting structures are

guaranteed to have certain level of visibility in the ori-ginal DVRIs given by users or the DVRIs generated byour method.• Coherent Information. The presentation of the

structures is coherent with the actual structures so thatthe shapes of the interesting structures can be clearlyand correctly perceived.• Familiar Context. The DVRIs and the original

DVRIs share some common content so that users areoriented about the spatial relation among the structurespresented in them.

The major contributions of this paper are:• We investigate one weakness of DVR and propose

a semi-automatic solution as a means to increase thesafety of volume visualization with direct volume ren-dering. We introduce a complete framework that iscapable of detecting and revealing the missing informa-tion.• We present visibility metric, coherence metric,

and context metric to access the quality of a semi-automatically generated DVRI using our framework.•We systematically attack the optimization problem

involved in generating DVRIs according to the afore-mentioned quality metrics.• We demonstrate how our framework can be

extended as a volume exploration method. Our methodcan gradually generate new DVRIs based on what havealready been revealed.

This paper is organized as follows. After introducingrelated work in Section 2, we describe our framework inSection 3. The preprocessing procedures of our systemare then described in Section 4. Section 5 introducesthe quality metrics, followed by Section 6, in which weexplain the rest of the semi-automatic DVRI genera-tion. Production of animation based on the generatedDVRIs is introduced in Section 7. In Section 8, wepresent experimental results to demonstrate our pro-posed framework. Finally, discussion is given in Section9 followed by conclusion in Section 10.

2 Related Work

Direct volume rendering has been thoroughly inves-tigated in the visualization field and many papers havebeen published on transfer function design and efficientrendering. Some recent results have been summarizedin several books and a few survey papers. In this sec-tion, we only review some closely related work.

2.1 Transfer Function Design

Searching for a proper transfer function is a challeng-ing problem. He et al.[1] used a stochastic search tech-nique to explore the transfer function parameter space.A semi-automatic transfer function generation methodbased on the histogram volume is proposed in [2]. Knisset al.[3] proposed several useful manipulation widgetsfor transfer function design. A set of tools for interac-tive specification of parameters for volume visualizationsystem built on VolumePro technology was introducedby Konig et al.[4] Tzeng et al.[5] proposed a classifica-tion method based on user painting. Very recently, visi-bility becomes a major component in transfer functiondesign[6], as proposed by Correa and Ma. In our frame-work, we use a divide-and-conquer approach for trans-fer function design to achieve our various purposes.We also introduce local depth opacity-modulation asa voxel-level visibility boosting technique.

2.2 Genetic Algorithm

Genetic algorithm (GA) is a widely-used techniquefor searching solutions to optimization problems[7]. Heet al.[1] first employed GA for transfer function de-sign. GA was also exploited to optimize visualiza-tion quality through automatic visualization parame-ters selection[8]. Wu and Qu[9] developed a frameworkfor editing DVRIs using GA with the energy functionbased on image similarity. In this paper, we also use GAfor generating DVRIs but we have different purposes.

Wai-Ho Mak et al.: Visibility-Aware Direct Volume Rendering 219

Also, our energy function is formulated particularly forthe image quality and visibility guarantee.

2.3 Quality Metrics

Quality measurement is a fundamental problem incomputer vision. Image quality can be measured by ob-jective metrics and subjective metrics. The subjectivemetrics, such as mean option score (MOS) which haslong been considered as the best image quality mea-surement, are too expensive to be useful in practice asthey require a number of human observers. The ob-jective image quality measures can be generally classi-fied into full-reference[10], no-reference[11], and reduced-reference[12] measures. For details, interested read-ers can refer to [13]. Wang et al.[14] proposed animage-based quality metric which measures the contri-bution of the multiresolution data blocks to the result-ing image for their multiresolution level-of-detail selec-tion and rendering framework. They further developeda reduced-reference method for volumetric data qualitymeasurement[15]. However, the study of quality met-ric for DVRIs is scarce. Our framework needs to esti-mate the image quality, as we require that our resultingDVRIs should not only reveal the missing informationbut also be coherent and have enough contextual infor-mation. Consequently, our image quality metrics arebuilt from the perspective of the coherence and contextcriteria, which are different from previous work.

2.4 Coherence

Coherence often indicates the logical and orderlyconsistent relationship of items. Groller and Purga-thofer provided an excellent survey on coherence incomputer graphics[16]. Among various types of coher-ence discussed in their survey, object coherence and im-age coherence are most relevant to our work. Objectcoherence states that local close regions in 3D spatialdomain are likely to be occupied by similar entities,which is also employed by Lundstrom et al.[17] for fea-ture detection. Image coherence is similarly defined inthe 2D image domain. In addition, 3D objects andtheir corresponding 2D projections should have similardegrees of connectedness and smoothness.

2.5 Feature Revealing Techniques

GPU-accelerated interactive clipping methods pro-posed by Weiskopf et al.[18] are useful for data explo-ration. A novel importance-based approach was devel-oped by Viola et al.[19] for showing the underlying fea-tures in a “focus + context” manner. Kruger et al.[20]

and Bruckner et al.[21] introduced new context preserv-ing visualization techniques for interactive visualization

of volumetric data. Rezk-Salama and Kolb[22] pro-posed a technique called opacity peeling to reveal hid-den structures by removing visible voxels layer by layer.Our approach also reveals underlying features, yet inaddition, it ensures that all voxels of features becomevisible in a set of DVRIs.

3 System Overview

We assume that the volumetric dataset is alreadysegmented and a set of DVRIs are already generated byusers. In the preprocessing stage, the segmented struc-tures are decomposed into components called atomicstructures. Their properties and the relation amongthem are stored. After the volume decomposition, thesystem analyzes and records the visibility and visua-lization quality of each atomic structure in all thegiven DVRIs. This information is tracked and updatedthroughout the whole DVRI generation process. Af-ter preprocessing, users will define the ROI by variousmeans provided by our system, e.g., choosing intensityranges, directly selecting structures from the segmenta-tion result set given structure thumbnail images, usinga graph interface, or brushing on the DVRI to indicatecertain interested areas. Our system is also capable ofdetecting the region of interest automatically by ana-lyzing the source DVRIs, the visualization history, andsome optional user input. A set of DVRIs will thenbe automatically generated to reveal missing informa-tion in the ROI effectively. We use genetic algorithmto search for a good opacity seed value for each atomicstructure in every generated DVRI. Given an opacityseed value, a 2D transfer function is then automati-cally generated for the corresponding atomic structure.With different 2D transfer functions being assigned toall the atomic structures in the volume, an intermediateDVRI is rendered and evaluated with the energy func-tion used in the genetic algorithm. Our energy functionconsists of three metrics, namely the visibility metric,the coherence metric, and the context metric. Thesethree metrics respectively guarantee visibility of unre-vealed voxels, maintain the coherence of the presentedinformation, and add in a certain amount of sharedcontext so that inter-structure relation can be easilyunderstood. When the evaluated quality of an interme-diate DVRI is good enough according to users’ needs,i.e., user-defined thresholds, it is output as one of thegenerated DVRIs. Every such successful generation issubsequently followed by an update to the structurevisibility and quality information. The DVRI genera-tion process is repeated until all structures of interestbecome visible and each atomic structure in the ROI isrevealed in high quality at least once. Finally, a subsetof the generated DVRIs can be chosen to generate an


animation that gives a comprehensive presentation ofthe structures of interest.

4 Data and DVRI Preprocessing

4.1 Decomposition

Given the segmentation, the voxels in the volumeare grouped into different structures. The resultingsegments, however, are sometimes too large to receiveenough visibility when they are rendered due to seriousself-occlusion at the voxel level. Therefore, we need tofurther break those segments down into smaller pieces.The decomposition process subdivides a segment untilit can receive enough visibilities when it is rendered in-dividually without the presence of other segments. Thesubdivision always iteratively produces two clusters ofequal sizes in order to prevent a segment becoming toomany small fragments. The clustering can be based onone or more attributes. Our current approach is to usevoxel depth values, i.e., distance to the view-plane, todo the partitioning. Voxels are sorted according to theirdepth values and then divided into two clusters of thesame sizes. The resulting segments are atomic struc-tures which are small clusters of voxels which belong tothe same structure.

4.2 Transfer Function Design for AtomicStructure

For each evolution step in the genetic algorithm, newcandidate solutions are generated. This involves gene-rating an intermediate DVRI given an atomic struc-ture opacity seed array returned by the genetic algo-rithm. There are three important considerations fordesigning a good transfer function for an atomic struc-ture. First, a certain percentage of the voxels in theatomic structure must have a certain level of visibilitywhen the structure is not occluded by other structureseven if there is self-occlusion among its voxels. Second,the information in an atomic structure should be con-veyed effectively. Third, the transfer function has tobe generated automatically and efficiently as it has tobe generated for each atomic structure before each eva-luation. To address all three concerns, we develop anefficient opacity transfer function for all the voxels inan atomic structure. While we adopt the widely usedgradient magnitude opacity-modulation, we introducea new technique called depth opacity-modulation. Theopacity value for voxel v in an atomic structure s is:

αv ={

Z(v), if Z(v) 6 1,

1, otherwise,

Z(v) = αs ·G(gv) · rdv−dmin(s)

where αs is the given seed opacity value for the atomicstructure, gv is the gradient magnitude of v, and G is amonotonically-increasing gradient magnitude opacity-modulation function which maps gradient magnitudesto opacity-modulation factors. dv is the depth valueof v, dmin(s) is the minimum depth of all voxels inthe atomic structure s, and r is the depth opacity-modulation rate which controls the influence of voxel’sdepth in modulation. As αs, G, r, and dmins are allthe same for voxels in the same atomic structure, thisformulation is essentially a 2D transfer function map-ping from voxel depth and gradient magnitude to opa-city. This function gives high opacity values to voxelsof high depth value (i.e., at the back side of the atomicstructure with respect to the viewpoint). It ensuresthat all of those voxels will still receive enough visibi-lity even if they are occluded by voxels of low depthvalue (i.e., at the front side of the atomic structure).Depth modulation can significantly reduce the numberof atomic structures as structure visibility guaranteecan now be achieved much more easily. Moreover, thegradient modulation transfer function G applies largeopacity-modulation factors to voxels with high gradi-ent values and small opacity-modulation factors to vox-els with low gradient values. It therefore enhances theclarity of the details in an atomic structure and im-proves effectiveness of information delivery as regionswith more intensity variation are often of more interestto users.

4.3 Structure-Based Analysis of DVRIs

After decomposition of the data, a set of atomicstructures are generated as the basis for analysis. Foreach DVRI, we first find out which structure is pre-sented in the DVRI. Then for each of the presentedstructures, we further analyze their quality with themetrics described in Section 5. The unclear or missingstructures in the region of interest will be the foci ofthe DVRIs to be generated.

4.4 Region of Interest Selection

Not all structures are of interest to the users as diffe-rent users have different visualization goals which resultin different regions of interest in the volume. The sim-plest ROI selection is achieved by selecting one or moreinterested data value (intensity) ranges. This will workfor simple volume datasets in which classes of structurescan be easily specified by just data values. In addition,as segmentation is assumed to be performed, users canalso directly select the structures of interest from thesegment gallery. Moreover, users can specify ROI onDVRI by brushing on it. After brushing, our systemwill list all the segmented structures that are hit by


rays casted from the brushed pixels for user selection ofinteresting structures.

5 Quality Evaluation Metrics

To obtain high-quality DVRIs that can effectivelyconvey the missing information, we need some metricsto measure their qualities. In this section, we intro-duce three metrics used in our system for DVRI qualityevaluation.

5.1 Visibility Metric

A voxel is said to be visible when its visibility islarger than a certain voxel-level visibility threshold. Asvoxels of high gradient magnitude are often of higherimportance than voxels of low gradient magnitude asthey represent feature outlines, the voxel-level visibilitythreshold should depend on the voxel’s gradient mag-nitude. While the threshold base value tvv is specifiedby the users, it is first modulated by the gradient mag-nitude opacity-modulation function G, which is men-tioned in Subsection 4.2, when it is applied to check ifvoxel v is visible. Therefore, the voxel-level visibilitythreshold used by our system for v is tvv ·G(gv). Basedon that, the structure visibility of an atomic structure sis defined as V (s) = Nvv (s)

Ntv (s)where Nvv (s) is the number

of visible voxels in s while Ntv (s) is the total number ofvoxels. It measures the percentage of visible voxels ins. If it is equal to or larger than the structure visibilityguarantee threshold tsv , then the corresponding atomicstructure is considered as visible enough.

5.2 Coherence Metric

Using only the visibility metric, we can already gene-rate a set of DVRIs to have the missing informationrevealed and to provide interesting structure visibilityguarantee. The missing information, however, is un-likely to be well-revealed if it is just randomly arrangedwith the surrounding context. Therefore we propose touse a coherence metric to help arrange the presenta-tion of information so that the missing information ispresented in a comprehensible and clear way.

In a 2D image, for users to perceive the shape of astructure, the structure shading variation on the imageis a very important clue. However, in a DVRI, as trans-parency is used, partial occlusion can often occur andsuch occlusion can distort the structure shading varia-tion. Hence, for a structure to be correctly and clearlyrevealed, its shading distribution on the image shouldnot be distorted too much by occlusion. Fig.1 illus-trates this coherence issue that should be considered.In Fig.1(a), the coherence of an engine part is de-stroyed as there is too much occlusion on the structure

Fig.1. Coherence problem. (a) A DVRI showing an engine part

(enclosed by a red rectangle) shown with bad coherence. While it

is visible, it is unclear in the image due to too much non-uniform

occlusion. (b) A DVRI showing the same part, which is displayed

clearly, with good coherence.

causing unclear appearance even though it is visible inthe image to some extent. It also looks like as if it is ofseveral fragments because of its inconsistent appearancein different regions due to different degrees of occlusion.After adjusting the opacity setting, the coherence ofthat engine part is restored and thus clearly shown inFig.1(b). Our coherence metric checks whether a struc-ture is presented correctly and clearly on a DVRI.

The coherence evaluation is conducted on a focusstructure, which is unique in a candidate solution, ina generated DVRI. Before actually computing the co-herence of the structure, we need to check if it is visi-ble enough in the image according to the previouslymentioned visibility metric. Being visible in the imageis a necessary condition, therefore, if the structure isnot visible enough in the image, the coherence is con-sidered as bad automatically. In that case, the algo-rithm returns (1 + (tsv − V (f))/tsv ) · Kh, where Kh

is a large constant, to represent bad coherence qua-lity. The worse the visibility, the worse the coherence.Assume the structure is visible enough in the image,i.e., V (f) > tsv , to compute the coherence metric, wefirst exclusively render the focus structure and save the


grayscale image of the rendered image to capture theluminance variation (i.e., shading distribution) of thefocus structure projection on the image. Then, we an-alyze the difference image imagediff of its grayscale im-age and that of the given DVRI. The statistical vari-ance σ2 of the difference values in its pixels is used asan estimation of structure coherence. Similar differencevalues in the pixels give small variance and thus implyhigh structure coherence as the shading distribution isgenerally preserved. On the other hand, large varianceimplies inconsistent change to the structure appear-ance, which means low structure coherence. Note thatuniform change of the shading distribution of a struc-ture projection will give zero variance of the differencevalues. In that case, the coherence is good because theshading distribution is not really disturbed. The algo-rithm for calculating the coherence metric H(c) given acandidate solution c is outlined in Algorithm 1. Usingthis metric, our system optimizes the coherence of thefocus structure. As each atomic structure in the ROIwill become a focus structure in one of the optimiza-tion iteration, the system will eventually display eachof them coherently and clearly at least once.

Algorithm 1. Coherence Metric

1: f ⇐ the focus structure

2: if V (f) > tsv then

3: render f alone to get its structure image

4: imagef ⇐ grayscale image of structure image of f

5: imagec ⇐ the grayscale image of the rendered ima-

ge produced by candidate solution c

6: imagediff ⇐ new image for difference storage

7: for each pixel (x, y) in imagediff do

8: imagediff (x, y) = |imagef (x, y)− imagec(x, y)|9: end for

10: P ⇐ the set of imagediff pixels which s occupies

11: H(c) ⇐ σ2(P )

12: else

13: H(c) ⇐ (1 + (tsv − V (f))/tsv ) ·Kh

14: end if

5.3 Context Metric

The relation among structures is very important in-formation. Hence, we want to introduce new informa-tion while keeping certain old information in each gen-erated DVRI so that the relation between the old andthe new can be easily understood. Our context met-ric consists of two parts, namely, data-domain contextmetric and image-domain context metric.

5.3.1 Data-Domain Context Metric

If there is no common structure presented in the

Fig.2. Context problem. (a) Some fish organs are shown in the

middle of a DVRI. (b) Fish bones are shown in a DVRI. (c) Both

structures, being the context for each other, are shown in the

same DVRI. With contextual information, users can easily es-

tablish the relation between the structures, i.e., the organs are

under the bones.

generated DVRIs, users may not be able to easily es-tablish relation among the structures revealed in them.Certain shared information should be maintained toserve as critical context. Fig.2 shows an illustration.Some fish organs are shown in the middle of a DVRI(see Fig.2(a)) while some fish bones are shown in an-other DVRI (see Fig.2(b)). Both features are shownclearly in their own images, however, users have no ideaabout the relation between the organs and the bones.

In the good example (see Fig.2(c)), both structures,being the context for each other, are shown in the sameDVRI, so users can easily see that the organs are un-der the bones. To address this issue, we develop thedata-domain context metric to measure how much oldinformation in the data-domain is kept in a generatedDVRI to facilitate users’ recognition of inter-structurerelations.

Given an old DVRI Iold and a new DVRI Inew , thedata-domain context metric Cdata(c) for Inew based onthe presented information in Iold is defined as

Cdata(Inew , Iold) =

|Snew ∩ Sold |

|Snew | , if |Snew | > 0,

0, otherwise,

where Snew and Sold are the sets of presented structures


in Inew and Iold respectively. It measures the percen-tage of atomic structures previously presented in Ioldwhich remain in the new DVRI Inew . The higher itsvalue, the more common structures are presented, andtherefore the easier the user can relate the new DVRIto the old DVRI using the shared context structures.

5.3.2 Image-Domain Context Metric

As users study the structures by looking at theDVRIs, we need an image-domain metric to estimatethe perceived contextual information in addition to thedata-domain one. Image similarity measurement is in-volved in our image-domain context metric. The mea-surement is based on edge images because of the obser-vation that edges on a 2D image are important percep-tion clues for structures. We adopt the same method forimage similarity measurement as in [9] although our useof them is different. To compute the image-domain con-text metric for Inew based on the presented informationin Iold , we first use a well-established optimal edge de-tector called Canny edge detector to compute the edgeimages for Inew and Iold . Then we apply Gaussian fil-tering to both edge images before the image compari-son. Finally we calculate the percentage of similar pix-els of them in the same location and use this percentageas our image-domain context metric Cimage(Inew , Iold).

While using this image-domain metric alone cansometimes give false results due to inaccurately de-tected edges on the rendered image, using it togetherwith the data-domain metric can alleviate this problemand give a good estimation.

5.3.3 Context-Aware Exploration Sequence

To have all generated DVRIs introduce new infor-mation coherently while keeping a certain amount ofold information shared with all the rest of the DVRIs isvery difficult. We therefore propose a concept, namelycontext-aware exploration sequence, which serves as afoundation of the final form of the context metric. Acontext-aware exploration sequence is a DVRI view-ing order in which a DVRI always shares a certainamount of context with its previous DVRI. That is, fora context-aware exploration sequence 〈x1, x2, . . . , xn〉,DVRIs xj and xj+1, with 1 6 j < n, share a certainamount of context so that the structures shown in themcan be easily related with each other. Exploiting thisconcept, our context metric just needs to ensure thatevery newly generated DVRI has enough shared contextwith at least one of the previously generated DVRIs orthe original DVRIs. It is because, logically, any newlygenerated DVRI can be related to an original DVRI insome context-aware exploration sequence if it can berelated to any of the previously generated DVRIs.

Given a candidate solution c generating a DVRIIc and a set of all previously shown DVRIs Θ (i.e.,both the previously generated DVRIs and the originalDVRIs), the final form of the context metric C(c) of Ic

is defined as follows.

C(c) =

0, if ∃Ip ∈ Θ such that

Cdata(Ic, Ip) > tdata and

Cimage(Ic, Ip) > timage ,

(1 + (δdata+ otherwise,

δimage) ·Kc,

δdata =

0, if Cdata(Ic, Im) > tdata ,

1− Cdata(Ic, Im)tdata

, otherwise,

δimage =

0, if Cimage(Ic, Im)

> timage ,

1− Cimage(Ic, Im)timage

, otherwise,

where Im ∈ Θ is the DVRI with the minimum valueof (δdata + δimage), i.e., the DVRI with the most con-textual information in the set. tdata and timage arethe threshold values for data-domain context metricand image-domain context metric respectively. If oneof those metrics is below its corresponding thresholdlevel, the difference between the calculated value andthe threshold will be multiplied by a large constant Kc

and reflected in the final context metric value. Thelarger the context metric value, the worse the context.If both data-domain metric and image-domain metricare satisfactory, 0 will be the resulted value to indicatethat the context is good enough.

6 Automatic DVRI Generation

Our system generates DVRIs using genetic algo-rithm with the quality metrics described in the previoussection. Genetic algorithm is a widely used techniquefor solving optimization problems. The solution encod-ing design and energy function design are the two majorchallenges for modeling a problem properly and findinga good solution. Details of genetic algorithm can befound in [7].

6.1 Solution Encoding

One possible encoding scheme is to use a series ofnumbers to represent the opacity mapping defined by atransfer function. For example, a 1D transfer functioncan be represented by a 1D array that defines differentopacity values for voxels with different intensities. Thisapproach is adopted by He et al.[1] and Wu and Qu[9].This, however, is not flexible enough to solve our prob-lem. In many cases, structures of similar intensities


occlude each other. Therefore, not all of them can re-ceive enough visibility when they are always assignedwith the same opacity value by the transfer function.With such a limitation, visibility cannot be guaranteedfor all the structures no matter how the transfer func-tion is set. The most flexible solution encoding is touse a large set of numbers to define the opacity valuefor each voxel. With such flexibility, we can preciselycontrol the opacity of each voxel in each DVRI, so visi-bility can definitely be guaranteed for all voxels. How-ever, the solution space becomes extremely large for thealgorithm to find a good solution.

We, therefore, adopt a divide-and-conquer approachto tackle this problem. We first divide the volume into aset of small components called atomic structures. Thenfor each atomic structure we only need an opacity seedvalue to automatically generate a transfer function forit. We may proceed to encode a solution with a series ofnumbers which represent opacity seed values for atomicstructures. With this approach, however, atomic struc-tures of the same structures may be assigned with dif-ferent opacity and thus appear strange and not coherentwith each other. Because of this, a structure is dividedinto atomic structures only when decomposition is nec-essary to reveal certain part of it. We formulate thesolution encoding for the genetic algorithm as an arrayof floating-point numbers: (x1, x2, . . . , xn) where xi isthe opacity seed value of structure or atomic structurei. Our solution space is small enough while we cangenerate high-quality solution that provides visibilityguarantee to interested structures.

6.2 Structure Categorization

Each of the structures and atomic structures shouldbe rendered differently as they have different roles andpositions in the DVRI, to achieve this, we categorizethem according to their roles in the DVRI and theirocclusion relations. We first analyze the occlusion re-lations among the structures and then build a datastructure called occlusion graph to store them for ef-ficient use of information later. The occlusion graph isbuilt by having atomic structure represented as nodesand adding directed edges from node u to node v if thestructure represented by u occludes that by v. Afterwe have the occlusion graph, we can categorize all theatomic structures as one of the following four categories:• Focus Structure. For each generation, our system

selects a focus structure, which is previously not vis-ible or clear enough according to our metrics, to berevealed in the new DVRI. It should become visibleenough in the generated DVRI to introduce new infor-mation. Therefore their opacity seed values should beset high.

• Occluding Context. We should show the focusstructure with their context included in the DVRI aswell so that they can be related to other structures inthe volume for comprehension. The context structuresoccluding the focus structure are labeled as occludingcontext. They should be assigned very low opacities, ifany, so that they would not occlude the focus structuretoo much in order to provide visibility guarantees tothe voxels of the focus structure. The effect of gradientmagnitude opacity-modulation should also be magni-fied to suppress their uniform regions while enhancingthe edges to give just enough clues of their existenceand reduce the occlusion to the focus structure. Onthe other hand, the effect of depth opacity-modulationcan be turned off as we do not need to give visibilityguarantees to voxels of occluding context.• Occluded Context. Context structures which are

occluded by focus structure are included in the occludedcontext category. Similar to the reasoning for occludingcontext, the effect of depth opacity-modulation can beturned off. The effect of gradient magnitude opacity-modulation can also be turned off as the structures inthis category are seriously occluded by the focus struc-ture. We do not need any suppression or enhancement.As they are occluded by the focus structure, to preservetheir existence as a context, we assign high opacity va-lues to them to allow them to be still visible.• Other Context. Any other structures not in-

cluded in the above three categories are categorizedas other context. They are neither occluding nor oc-cluded by the focus structure. We can directly givethem medium opacity values to serve as surroundingcontext. Opacity-modulation can simply be turned offfor them.

6.3 Energy Function and Solution Search

The energy function we use consists of the threequality metrics described in Section 5. Note that as thevisibility metric is combined into the coherence metric,we only have two terms. The energy function for a can-didate solution c is defined as E(c) = H(c)+C(c). Theoptimization process will not end successfully if E(c)is greater than or equal to one of the large constantsKh and Kc, that is, either one or both of the visibilityguarantee and context constraint is not satisfied. Onthe other hand, if both of the visibility guarantee andcontext constraint are satisfied, the optimization pro-cess will optimize the solution for the best informationcoherence in the given time. We run the genetic algo-rithm for a certain number of iterations then its bestcandidate is checked for its quality. If the candidate canintroduce new information while it can be related to thepreviously generated DVRIs, then it will be outputted.


Otherwise, we continue to run the genetic algorithmfor the same focus structure. The whole process is re-peated until all the interesting atomic structures havebeen revealed in high quality at least once.

7 Animation Generation

A subset of the generated DVRIs can be chosenby the users to generate an animation that presentsthe missing information comprehensively. The selectedDVRIs will serve as keyframes for the animation. Tran-sitional DVRIs will be generated between the keyframesto provide smooth transition. They are generated bylinearly interpolating the opacity value of each voxelfrom one keyframe DVRI to another.

7.1 Keyframe Ordering

While users can define the keyframe ordering them-selves, our system also provides an automatic keyframeordering approach. As the most important thing in akeyframe DVRI is the focus structure, logically the or-dering should be based on it. A natural ordering of thekeyframes is the depth ordering of the focus structurespresented in them. That is, the keyframes should beordered in a way such that the volume data is exploredfrom outside to the inside and from the front to theback. To sort the focus structures, if two focus struc-tures are of different classes according to the providedsegmentation, they will be ascendingly sorted accord-ing to the minimum depths (shortest distances to theviewplane) of the segments they belong to. Otherwise,they will be ascendingly sorted according to their ownminimum depths.

7.2 Temporal Coherence

After the keyframe ordering is obtained, we needto fix a temporal coherence issue that possibly ex-ists between consecutive keyframes. The problem is

illustrated in Fig.3. Suppose the keyframe ordering isFig.3(a), then Fig.3(b), and then Fig.3(c), it can beobserved that the engine body in the middle keyframeFig.3(b) has different opacity compared with Figs. 3(a)and 3(c). In effect, the animation will show a highlytransparent engine body, then a less transparent en-gine body, and finally a highly transparent engine bodyagain. As engine is a contextual structure in all thekeyframes, the change of opacity of unnecessary. More-over, this kind of opacity fluctuation during the ani-mation can introduce inconsistence and confusion is-sues. Therefore, it is desirable to have those kinds ofcontextual structures have stable opacity assignmentsthroughout the whole animation.

To address this problem, for each structure, our sys-tem first analyzes all the keyframes, and records thelowest non-zero opacity assigned when the structure isused as context. Then each keyframe will be checkedto make sure the context constraint will not be vio-lated, according to the context metric, after the lowestopacity in record is assigned to the structure. If thecontext condition is still good for all keyframes afterthe change, then the change will be actually commit-ted. This procedure can standardize the opacity of thecontextual structures when it is possible.

8 Experimental Results

Experiments are all performed on a standard PC ma-chine (Intel Core 2 Duo E6400, 3GB RAM) equippedwith a GeForce 8600GT graphics card.

8.1 CT Head

An experiment is conducted on a CT head dataset(128×128×231). The brain is selected as the ROI fromthe segment gallery and a DVRI, as shown in Fig.4(a),is rendered. Four images are then generated by our sys-tem in 157 seconds to guarantee the whole brain struc-ture to have at least 80% visibility. Fig.4(b) shows the

Fig.3. Temporal coherence problem. (a)∼(c) Keyframes in a user-defined order. The engine body has different opacity in (b) compared

with (a) and (c). This introduces inconsistence and confusion in the animation.


Fig.4. CT head experiment. (a) User-generated image. (b) System-generated DVRI that shows the outer view of the brain. (c)∼(e)

Other system-generated DVRIs that show the inner view of the brain.

outside view of the brain structure which is in yellow,and then the inner view of the brain structure is re-vealed in Figs. 4(c), 4(d), and 4(e). The skin and theskull shown in the original DVRI are preserved as con-text. The generated DVRIs help the user to explore thebrain structure almost completely. The amount of con-text is optimized in the optimization process so that thefocus structures (i.e., the atomic structures of the brain)can be shown clearly while the context is still visible toprovide important contextual information. An anima-tion is then produced to provide a comprehensive andsmooth presentation. The animation, which is gener-ated in 135 seconds, shows what is behind the skin andinside the brain gradually with well-preserved context.

Note that as the head dataset is a CT dataset, thequality of the brain structure is not very good. How-ever, this experiment demonstrates how our system canbe used to provide visibility guarantee to interestingstructures. This can help the user to explore thosestructures thoroughly.

8.2 CT Engine

We conducted another experiment on a CT enginedataset (256 × 256 × 128). The user-rendered DVRI(Fig.5(a)) shows some interesting structures in diffe-rent colors but most of them are not very clear. Theuser wants to see them clearly and check if there is any-thing missing in the nearby region. The ROI is specifiedby the user with brushing which is indicated by aquacolor in Fig.5(b). Since the engine structures are wellsegmented and do not have anything interesting insidethem, the structure visibility guarantee threshold is setto 50% to avoid unnecessary structure decompositionthat is used to reveal the inner parts of the structures.Our system automatically detects missing or unclearstructures and generates a set of DVRIs to reveal them.In this experiment, the system generates two DVRIs toreveal all the missing or unclear structures within theROI. The generation process takes 53 seconds. Fig.5(c)

shows clearly almost all the interesting structures whichare in different colors, however, part of the red struc-ture is occluded and thus missing. Therefore, anothergenerated DVRI, as shown in Fig.5(d), reveals the redstructure completely by lowering the opacity of the oc-cluding parts (i.e., the green structure and the yellowstructure). In both generated DVRIs, the engine bodyis preserved as the context. Finally, an animation basedon the original DVRIs and the two generated DVRIs isgenerated in 73 seconds.

Fig.5. CT engine experiment. (a) User-generated image. (b)

Same user-generated image with brushing ROI selection on it.

Strokes are in aqua color. (c) System-generated DVRI showing

all the interesting structures indicated by the user. (d) Another

system-generated DVRI clearly revealing the red structure that

is occluded before.

9 Discussion

Viewing the slices of a volumetric dataset one by


one can also guarantee the visibility of all voxels. How-ever, the coherence of structure is destroyed as a 3Dstructure becomes a stack of 2D slices. Moreover, thespatial inter-structure relation shown by 2D slices is notas clear as that in a 3D overview given by a DVRI. Ourapproach is based on DVR that shows important in-formation that can only be effectively conveyed in 3D.In addition, we guarantee visibility, preserve coherence,and provide critical context.

To reveal all the features in a volumetric dataset us-ing direct volume rendering is a challenging task. Onemajor problem is that users are not informed whetherall the information has been revealed. Even if theyknow what is missing or unclear, tuning a transferfunction to reveal those information still requires theirtremendous effort. On the other hand, the missing in-formation can be guaranteed to be presented in a cohe-rent way with the automatic approach, which requiresless effort from users, in our framework as demonstratedin this chapter.

Ghosted view is a classical approach in volume explo-ration that can improve the visibility of an importantstructure. It has many variances but the basic approachis the same. Given a structure of interest, the ghostedview removes the occluding region to reveal the struc-ture. Nonetheless, it does not guarantee visibility ofthe whole structure that there could be important re-gion within the structure of interest left unrevealed dueto self-occlusion. With our approach, self-occlusion isnot a problem as we guarantee visibility of voxels, thusthe occluded region will be shown in one of the gene-rated DVRIs.

Removing the visible voxels from front to back toshow occluded regions also has a major drawback thatit may not provide critical context. To show the struc-tures of interest, such a context may be removed aswell in the process. For example, the skin of a humanhead would have to be completely removed before thebrain can be seen. Our approach is flexible to providecontextual information of different depths and removeonly minimal amount of the occluding structures that isnecessary to show the occluded structures clearly whilethe occluding structures can still be served as impor-tant context.

The applications of our method are twofold: as asafety check and as a volume exploration method. Inone scenario, users can use other volume visualizationmethods to explore the data and save some representa-tive DVRIs. After they finish the visualization processand have a good understanding of the data, they canuse our system to check if any important informationis missing. Thus, our method serves as a safety checktool which can be incorporated into existing volumevisualization systems. In another scenario, users can

first generate some initial DVRIs using some automaticor semi-automatic methods. Then starting from theseinitial DVRIs, our system can be employed to semi-automatically generate new DVRIs one by one. Be-cause each new DVRI is guaranteed to contain somenew information and also keep some old information,users can use our method to explore different regionsof the data. In this manner, our method can serve as avolume exploration scheme.

Though the experiments are promising, our currentmetrics are still primitive and user studies are neededto validate them. For example, our energy functionincludes measurement of coherence which is a compli-cated issue involving human perception. All the gene-rated DVRIs have the same viewpoint and lighting pa-rameters. This is because we need to limit the degreesof freedom as the current problem with fixed viewpointis already highly challenging and complicated. Somespeedups exploiting this setting also become possible.

10 Conclusion and Future Work

In this paper, we have developed a framework tosemi-automatically generate DVRIs and an animationfor a given volumetric dataset and existing DVRIs. Thegenerated DVRIs can reveal all the missing informa-tion in users’ region of interest, and thus can improvethe safety of a volume visualization system and boostthe confidence of users in DVR. It promotes the wideradoption of DVR in real applications. Our frameworkcan guarantee that the missing information will be pre-sented in a coherent way while some old informationfrom the previous DVRI will be kept to provide userssome familiar context to understand relations amongstructures. A recursive procedure is used to gene-rate DVRIs using genetic algorithm which optimizes acarefully-designed energy function consisting of visibi-lity, coherence, and context terms. Our method can beused as a safety check for volume visualization systemsas well as a volume exploration method which allowsusers to start from some initial DVRIs and graduallyand recursively generate additional DVRIs until all in-teresting information is revealed.

Our framework can be extended in several ways.Currently, all the DVRIs are generated from the sameviewpoint under the same lighting setting. In the fu-ture, we will investigate how to generate a set of DVRIswith an additional freedom of changing the lighting andviewpoint. We believe there are many possibilities tomeasure the quality of a DVRI. We plan to incorporatemore metrics such as contrast and color harmonizationinto our extensible framework. Moreover, we will ex-ploit GPU for system acceleration. We will also applyour method as a built-in feature to some real systems.


Acknowledgments The authors would like tothank the anonymous reviewers for their valuable com-ments.

References

[1] He T, Hong L, Kaufman A, Pfister H. Generation of transferfunctions with stochastic search techniques. In Proc. IEEEVisual. Conf., San Francisco, USA, Oct. 27-Nov. 1, 1996,pp.227-234.

[2] Kindlmann G, Durkin J W. Semi-automatic generation oftransfer functions for direct volume rendering. In Proc.IEEE Symposium on Volume Visualization, Research Trian-gle Park, USA, Oct. 24, 1998, pp.79-86.

[3] Kniss J, McCormick P, McPherson A, Ahrens J, Painter J,Keahey A, Hansen C. Interactive texture-based volume ren-dering for large data sets. IEEE Computer Graphics andApplications, 2001, 21(4): 52-61.

[4] Konig A, Groller E. Mastering transfer function specificationby using volumepro technology. In Proc. Spring Confer-ence on Computer Graphics, Budmerice, Slovakia, Apr. 25-28, 2001, pp.279-286.

[5] Tzeng F Y, Lum E B, Ma K L. A novel interface for higher-dimensional classification of volume data. In Proc. IEEEVisual. Conf., Seattle, USA, Oct. 19-24, 2003, pp.505-512.

[6] Correa C, Ma K L. Visibility-driven transfer functions.In Proc. IEEE Pacific Visualization Symposium, Beijing,China, Apr. 20-23, 2009, pp.177-184.

[7] Mitchell M. An Introduction to Genetic Algorithms. Cam-bridge: MIT Press, MA, USA, 1996.

[8] House D, Bair A, Ware C. On the optimization of visualiza-tions of complex phenomena. In Proc. IEEE VisualizationConf., Minneapolis, USA, Oct. 23-28, 2005, pp.87-94.

[9] Wu Y, Qu H. Interactive transfer function design based onediting direct volume rendered images. IEEE Trans. Visual.Comput. Graph., 2007, 13(5): 1027-1040.

[10] Sheikh H R, Sabir M F, Bovik A C. A statistical evaluationof recent full reference image quality assessment algorithms.IEEE Trans. Image Processing, 2006, 15(11): 3441-3552.

[11] Sheikh H R, Bovik A C, Cormack L. No-reference qualityassessment using natural scene statistics: JPEG2000. IEEETransactions on Image Processing, 2005a, 14(11): 1918-1927.

[12] Gunawan I P, Ghanbari M. Reduced-reference video qualityassessment using discriminative local harmonic strength withmotion consideration. IEEE Transactions on Circuits andSystems for Video Technology, 2008, 18(1): 71-83.

[13] Wang Z, Bovik A C. Modern Image Quality Assessment. Mor-gan and Claypool Publichers, 2006.

[14] Wang C, Garcia A, Shen H W. Interactive level-of-detail selec-tion using image-based quality metric for large volume visual-ization. IEEE Transactions on Visualization and ComputerGraphics, 2007, 13(1): 122-134.

[15] Wang C, Ma K L. A statistical approach to volume dataquality assessment. IEEE Transactions on Visualization andComputer Graphics, 2008, 14(3): 590-602.

[16] Groller M E, Purgathofer W. Coherence in computergraphics. Technical Report, TR-186-2-95-04, Institute ofComputer Graphics and Algorithms, Vienna Universityof Technology, Austria, March 1995, Contact: [email protected].

[17] Lundstrom C, Ljung P, Ynnerman A. Local histograms fordesign of transfer functions in direct volume rendering. IEEETrans. Visual. Comput. Graph., 2006, 12(6): 1570-1579.

[18] Weiskopf D, Engel K, Ertl T. Interactive clipping techniquesfor texture-based volume visualization and volume shading.IEEE Trans. Visual. Comput. Graph., 2003, 9(3): 298-312.

[19] Viola I, Kanitsar A, Groller M E. Importance-driven featureenhancement in volume visualization. IEEE Trans. Visual.Comput. Graph., 2005, 11(4): 408-418.

[20] Kruger J, Schneider J, Westermann R. Clearview: An interac-tive context preserving hotspot visualization technique. IEEETrans. Visual. Comput. Graph., 2006, 12(5): 941-948.

[21] Bruckner S, Grimm S, Kanitsar A, Groller M E. Illustrativecontext-preserving exploration of volume data. IEEE Trans.Visual. Comput. Graph., 2006c, 12(6): 1559-1569.

[22] Rezk-Salama C, Kolb A. Opacity peeling for direct volumerendering. Computer Graphics Forum, 2006, 25(3): 597-606.

Wai-Ho Mak received theB.Eng. degree (first-class honors)in computer science and the M.Phil.degree in computer science and en-gineering both from the Hong KongUniversity of Science and Technology(HKUST) in 2007 and 2009 respec-tively. His research interests includescientific visualization and informa-tion visualization.

Yingcai Wu is a post-doctoralresearcher in the Department ofComputer Science at The Universityof California, Davis. He obtainedhis B.Eng. degree (2004) in com-puter science and technology fromSouth China University of Technol-ogy, China, and his Ph.D. (2009) de-gree in computer science and engi-neering from the HKUST. His re-

search interests are in medical volume visualization and vi-sual analytics.

Ming-Yuen Chan received theB.Eng. degree (2003) in computerengineering from the University ofHong Kong and the M.Phil. (2006)and the Ph.D. degrees (2009) in com-puter science from the HKUST. Hisresearch interests include scientificvisualization and medical imaging.

Huamin Qu is an associate pro-fessor in the Department of Com-puter Science and Engineering at theHKUST. His main research interestsare in visualization and computergraphics. He has conducted a widerange of research on scientific visu-alization, information visualization,real time graphics systems, virtualreality, and medical imaging. He re-

ceived the B.S. degree in mathematics from Xi’an JiaotongUniversity, China, the M.S. and the Ph.D. degrees in com-puter science from the Stony Brook University, New York.

Documents

Visibility-Aware Direct Volume Rendering - ycwu