Art and Science in VisualizationVictoria Interrante

1 Introduction

Visualization research and development involves the design, implementation andevaluation of techniques for creating images that facilitate the understanding of a set of data.The first step in this process, visualization design, involves defining an appropriaterepresentational approach, determining the vision of what one wants to achieve. Implementationinvolves deriving the methods necessary to realize the intended results – developing thealgorithms required to create the desired visual representation. Evaluation, or the objectiveassessment of the impact of specific characteristics of the visualization on application-relevanttask performance, is useful not only to quantify the usefulness of a particular technique but, morepowerfully, to provide insight into the means by which a technique achieves its success, thuscontributing to the foundation of knowledge upon which we can draw to create yet moreeffective visualizations in the future. In this chapter, I will discuss the art and science ofvisualization design and evaluation, illustrated with case study examples from my research. Foreach application, I will describe how inspiration from art and insight from visual perception canprovide guidance for the development of promising approaches to the targeted visualizationproblems. As appropriate I will include relevant details of the algorithms developed to achievethe referenced implementations, and where studies have been done, I will discuss their findingsand the implications for future directions of work.

1.1 Seeking Inspiration for Visualization from Art and Design

Visualization design, from the creation of specific effective visual representations forparticular sets of data to the conceptualization of new, more effective paradigms for informationrepresentation in general, is a process that has the characteristics of both an art and a science.General approaches to achieving visualizations that ‘work’ are not yet straightforward or well-defined, yet there are objective metrics that we can use to determine the successfulness of anyparticular visualization solution. In this section I will discuss ways in which the practices andproducts of artists and designers can help provide inspiration and guidance to our efforts todevelop new, more effective methods for communicating information through images.

Design, as traditionally practiced, is a highly integrative activity that involves distilling ideasand concepts from a great variety of disparate sources and assembling them into a concrete formthat fulfills a complex set of objectives. It is an inherently creative process that defies explicitprediction or definition yet whose results are readily amenable to comprehensive evaluation.Across disciplines, from graphic arts to architecture, the art of design is primarily learnt throughpractice and review and the careful critical study of work by others, and expertise is built upfrom the lifelong experience of “training one’s eyes”. Providing a “good environment fordesign” is critical to enabling and facilitating the process of design conceptualization. Creativeinsights are difficult to come by in a vacuum – designers typically surround themselves in theirwork area with sketches, images, models, references, and other material that has the potential toboth directly and indirectly provide inspiration and guidance to the task at hand. In addition,designers rely heavily on the ability to quickly try out ideas, abstracting, sketching out andcontemplating multiple alternative approaches before settling upon a particular design approach.

In visualization research, we can take a similar approach to the problem of designconceptualization – drawing inspiration from the work of others and from the physical worldaround us, and experimenting with new combinations and variants of existing techniques formapping data to images. We can also benefit from establishing fertile design environments thatprovide rich support for design conceptualization and varied opportunities for rapidexperimentation. Finally, it can sometimes be useful to work with traditional materials to createapproximate mockups of potential design approaches that allow one to preview ideas to avoidinvesting substantial effort in software development in what ultimately prove to be unproductivedirections.

Turning now from the process of design to the product, there is, in a wide variety of fields,from art to journalism, from graphic design to landscape architecture, a long history of researchin visual literacy and visual communication through drawings, paintings, photographs, sculptureand other traditional physical media that we have to potential to be able to learn from and use indeveloping new visualization approaches and methods to meet the needs of our own specificsituations.

In computer graphics and visualization, as in art, we have complete control not only overwhat to show but also over how to show it. Even when we are determined to aim for a perfectlyphysically photorealistic representation of a specified model, as in photography we have controlover multiple variables that combine to define the ‘setting of the scene’ that creates the mosteffective result. In many cases this includes not only selecting the viewpoint and defining thefield of view, setting up the lighting, and determining the composition of the environment, butalso extends to choosing the material properties of the surfaces of the objects that we wish toportray. For practical reasons of computational efficiency or because of the limitations ofavailable rendering systems we often choose to employ simplified models of lighting andshading, which can also be considered to be a design decision. In addition, we may choose touse non physically based ‘artificial’ or ‘artistic’ enhancement to emphasize particular features inour data, and we may selectively edit the data to remove or distort portions of the model toachieve specific effects.

Through illustration we have the potential to interpret physical reality, to distill theessential components of a scene, accentuate the important information, minimize the secondarydetails, and hierarchically guide the attentional focus. In different media, different methods areused to draw the eye to or away from specific elements in an image, and in each mediumdifferent styles of representation can be used to evoke different connotations of meaning.

When seeking to develop algorithms to generate simplified representations of data ormodels, it’s useful to consider where artists tend to take license with reality. They have similarmotivations to avoid the tedium and difficulty of accurately representing every detail in aphotorealistic manner, but at the same time they need to indicate enough detail, with enoughfidelity, to meet the expectations of the viewer and communicate the subject effectively.Numerous texts on methods of illustration present various artists’ insights on this subject [e.g.Loomis, Watson, Guptill, McCloud]. Vision scientists have also considered this question, fromthe point of view of seeking to understand how the brain processes various aspects of visualinput and it is interesting to note the connection between the findings in perception and commonpractices in artistic representation. For example, people are found to be highly insensitive to the

colors of shadows [Cavanagh], being willing to interpret as shadows patches whose luminance,and general geometry, is consistent with that interpretation, regardless of hue, and to be broadlytolerant of inconsistencies among shadows cast from different objects in a scene [Ostrovsky etal.], despite the significant role that cast shadows play in indicating objects’ positions in space[Kersten]. Although there is much about the processes of vision and perception that remainsunknown, research in visual perception has the potential to make explicit some of the intuitionthat artists rely upon to create images that ‘work’.

1.2 Drawing Insight for Visualization Design from Research in Visual Perception

In addition to seeking inspiration from art for the design of effective methods forconveying information through images, it is possible to use fundamental findings in humanvisual perception to gain insight into the ‘science behind the art’ of creating successful visualrepresentations. This can be useful because, although it is often, but not always, possible frominformal inspection to determine how well a single, particular visualization meets the needs of aspecific, individual application, or to comparatively assess the relative merits of alternativevisualization solutions for a particular problem, it is much less straightforward to achieve acomprehensive understanding of the reasons that one particular visualization approach is moresuccessful than another, and even more difficult to uncover the theoretical basis for why certainclasses of approaches are likely to yield better results than others. From a fundamentalunderstanding of the strengths and weaknesses and abilities and limitations and basic functionalmechanisms of the human visual system, we have the potential to become better equipped tomore accurately predict what sorts of approaches are likely to work and which aren’t, which canbe of immense benefit in helping us determine how to guide our research efforts in the mostpromising directions and to avoid dead ends.

Mining the vision research literature for insights into a particular visualization problem canbe a daunting task. The field of visual perception is broad and deep and has a very long and richhistory, with research from decades past remaining highly relevant today. The applicationdomains targeted in visualization are typically far more complex than the carefully controlleddomains used in perception studies, and extreme caution must be exercised in hazarding togeneralize or extrapolate from particular findings obtained under specific, restricted conditions.Also, the goal in vision research – to understand how the brain derives understanding from visualinput – is not quite the same as the goal in visualization – to determine how best to portray a setof data so that the information it contains can be accurately and efficiently understood. Thus it isseldom possible to obtain comprehensive answers to visualization questions from just a few keyarticles in the vision/perception literature but is more often necessary to distill insights andunderstanding from multiple previous findings, and then supplement this knowledge throughadditional studies.

Vision scientists and visualization researchers have much to gain from successfulcollaboration in areas of mutual interest. Through joint research efforts, interdisciplinary teamsof computer scientists and psychologists have already begun to find valuable answers toimportant questions in computer graphics and visualization, such as determining the extent towhich accurately modeling various illumination and shading phenomena, such as cast shadowsand diffuse interreflections, can facilitate a viewer’s interpretation of spatial layout [Hu,Madison].

In the remainder of this chapter I will describe in more concrete detail the application ofinspiration from art and insights from visual perception to visualization design and evaluation inthe context of selected case study examples from my research.

2 Case Study 1: Effectively Portraying Dense Collections of Overlapping Lines

Numerous applications in fields such as aerospace engineering involve the computation andanalysis of three dimensional vector fields. Developing techniques to effectively visualize suchdata presents some challenges. One promising approach has been to provide insight intoportions of the data using bundles of streamlines. However, special steps must be taken tomaintain the legibility of the display as the number of streamlines grows. Stalling et al. [97]proposed a very nice technique, based on principles introduced by Banks [94], for differentiallyshading one-dimensional streamlines along their length according to the local orientation of theline with respect to the view direction and a specified light source. Still, the problem ofpreventing the local visual coalescing of similarly oriented adjacent or proximate overlappinglines remained. Attempts to visually segregate the individual lines by rendering them in aspectrum of different colors rather than in the same shade of grey met with only partial success,because it was still difficult to accurately and intuitively appreciate the existence and themagnitude of the depth separation between overlapping elements in the projected view. Theproblem is that, in the absence of indications to the contrary, objects are generally perceived tolie on the background over which they are superimposed [Gibson]. In some situations, castshadows can be used to help disambiguate depth distance from height above the ground plane[Yonas], but this technique is most effective when applied to simple configurations in whichindividual objects can be readily associated with their shadows. A more robust solution to theproblem of effectively portraying clusters of overlapping and intertwining lines (figure 1) isinspired by examples from art, and explained by research in visual perception.

Figure 1: Two overlapping lines of roughly equivalent luminance but differing hue, shown in three subtlydifferent depictions.

For thousands of years, artists have used small gaps adjacent to occluding edges as a visualdevice to indicate the fact that one surface should be understood to be passing behind another.This convention can be observed in visual representations dating as far back as the Paleolithicpaintings within the caves of Lascaux (figure 2).

Figure 2: Since prehistoric times, artists have used gaps to indicate the passing of one surface behindanother, as shown in this photograph taken in a Smithsonian museum exhibit reproducing the creation ofthe second ‘Chinese Horse’ in the Painted Gallery of the cave of Lascaux. Photo by Tomás Filsinger.

Recent research in visual perception [Nakayama] helps explain why the technique ofintroducing gaps to indicate occlusion is so effective. In ordinary binocular vision, when weview one surface in front of another using both of our eyes, we see different portions of thefarther surface occluded by the nearer surface in the views from each eye (figure 3). Called ‘daVinci stereopsis’, in deference to the recognition of this phenomenon by Leonardo da Vinci, asreported by Wheatstone (1838), the perception of these inter-ocularly unpaired regions, whichhe likened to ‘shadows cast by lights centered at each eye’, has been shown to be interpreted bythe visual system as being indicative of the presence of a disparity in depth between twosurfaces.

Figure 3: When one surface is viewed in front of another using both eyes, a different portion of the moredistant surface is occluded in the view from each eye. Psychologists have found evidence that thepresence of these interocularly unpaired regions evokes a perception of depth disparity by our visualsystem [Nakayama and Shimojo].

The use of gaps to clarify depth ordering in computer-generated line drawings dates back tothe earliest days of computer graphics – one of the first ‘haloed’ line drawing algorithms waspresented by Appel et al. at SIGGRAPH in 1979. In 1997, we [Interrante and Grosch]

developed an algorithm for using ‘visibility-impeding halos’ to clarify the depiction of whatwere effectively clusters of streamlines in volume-rendered images of 3D line integralconvolution textures (figure 4). Our decision to take this approach, and the direction of theparticular path that we followed in developing it, was guided both by the inspiration fromexamples in art and by the insights from research in visual perception.

Figure 4: A side-by-side comparison, with and without visibility-impeding halos, illustrating theeffectiveness of this technique for indicating the presence of depth discontinuities and facilitatingappreciation of the extent of depth disparities between overlapping lines in the 3D flow. Data courtesy ofDr. Chester Grosch.

Our implementation was based on a simple modification to the basic LIC algorithm [Cabraland Leedom, Stalling and Hege] allowing the efficient computation of a matched pair ofstreamline and surrounding halo textures. We automatically defined a subtle and smoothlycontinuous 3D visibility-impeding ‘halo’ region that fully enclosed each streamline in theoriginal 3D texture by performing the LIC simultaneously over two input textures containingidentically located spots of concentric sizes. Because the streamline tracing only had to be doneonce for the pair of volumes, the overhead associated with the creation of the halos was kept to aminimum. Halos were implemented, during raycasting volume rendering, by decreasing thecontribution to the final image of any voxel encountered, after a halo was previously entered andsubsequently exited, by an amount proportional to the largest previously encountered haloopacity. (Because each line was necessarily everywhere surrounded by its own halo, it wasimportant to allow the voxels lying between the entrance and exit points of the first-encounteredhalo to be rendered in the normal fashion.) It should be noted that this particular implementationassumes a black background, and will fail to indicate the existence of depth discontinuitiesbetween lines that pass closely enough that their halos overlap in 3-space, even if the linesthemselves do not actually intersect.

3 Case Study 2: Using Feature Lines to Emphasize the Essential 3D Structure of a Form

The goal of an effective graphic representation is to facilitate understanding. Whencreating computer-generated visual representations of models or surfaces, we desire to portraythe data in a way that allows its most important features to be easily, accurately and intuitivelyunderstood. In applications where the information that we need to effectively communicate is the3D shape of a rendered surface, there is particular value in seeking both inspiration from thepractices of artists and insight from research in visual perception.

3.1 Nonphotorealistic Rendering in Scientific Visualization

A photographic depiction will capture the exact appearance of an object as it is actuallyseen, with subtle, complex details of coloration and texture fully represented at the greatestpossible level of detail and accuracy. Despite the option to use photography, however, a numberof scientific disciplines have historically retained artists and illustrators to prepare hand-drawnimages of objects of study [Ridgway, Loechel, Hodges]. One of the reasons for this is that in adrawing it is possible to create an idealized representation of a subject, in which structural orconceptual information is clarified in a way that may be difficult or impossible to achieve in eventhe best photograph. In some fields, such as cartography [Imhof], and archaeology [Addington,Dillon], specific stylizations are used to encode particular information about important featuresof the subject within the graphical representation.

When drawings are prepared for visualization purposes, in many cases they are createdusing photographs of the subject as reference. Through comparing these drawings with thephotographs from which they were derived, we have the ability to gain insight into the selectiveprocess used by the artist to identify and emphasize the important features in the data whileminimizing the visual salience of the extraneous detail. We can also observe at what points andin what respects the artist takes special care to remain faithful to details in the original depictionand where he chooses to exercise artistic license. Through these observations, and by consultingthe relevant literature in the instruction of illustration techniques, we can derive inspiration forthe development of visualization algorithms that aspire to achieve similar effects.

Searching for deeper insight into the process of determining how to design effectivepictorial representations, it can be useful to carefully consider all aspects of the questions of notonly how, but also why and when a hand-drawn interpretation of a subject is preferable to aphotograph. In some fields, such as zoology [Allen], it is not uncommon to find a wide range ofrepresentational modalities – from simple outline drawings to detailed, shaded, coloreddrawings, to photographs – used at different times, for different purposes. Apart from theinfluence of various practical concerns, the definition of an optimal representation will dependon the particular purpose for which it is intended to be used.

The critical issue, I believe, is that drawings are intended to be idealizations of actualphysical reality. Recognizing a subject from a drawing can require some translation. Thesuccess of the representation hinges on the extent to which it captures precisely the necessaryinformation for successful comprehension, which not only depends on the vision of the artist indefining what that information is and on his skill in portraying it faithfully, but also on the needsand experience level of the observer. With photographs, there is no interpretation required –every subtle detail of color and texture, shading and shadow, is exactly represented, for somespecific configuration of subject, viewpoint and lighting. Psychologists and others haveconducted numerous studies to examine questions of whether, when, and under whatcircumstances, it might be possible to increase perceptual efficiency by reducing the visualcomplexity of a pictorial representation [Ryan and Schwartz, Fussel and Haaland, Hirsh andMcConathy, Biederman and Ju]. Results are mixed, with the superiority of performance usingdrawings vs. photographs depending both on the task and on the quality and level of detail in thedrawing. However, a common theme in all cases is the potential danger inherent in using highly

simplified, schematic, or stylized representations, which can have the opposite effect tofacilitating perception, instead increasing cognitive load by imposing an additional burden ininterpretation. Studies have found that face recognition from outline drawings is exceptionallypoor [Davies]; but that adding “mass”, even through simple bi-level shading, improvesrecognition rates considerably [Bruce]. Face recognition rates from outline drawings can also beimproved by distorting outline drawings in the style of a caricature [Rhodes].

Carefully constructed nonphotorealistic representations have significant potentialadvantages for use in visualization. They allow the possibility to increase the dynamic range ofthe attentional demands on the observer by allowing greater differentiation in the salience of thevisual representation, and enable guiding the attentional focus of the observer throughhighlighting critical features and de-emphasizing the visual prominence of secondary details.However the success of such efforts depends critically on the ability to correctly define what toemphasize and how to emphasize it, in order to best support the task objectives.

Nonphotorealistic representations also have the potential to facilitate visualizationobjectives by allowing greater differentiation in the specificity of the visual representation. Byintroducing the possibility of ambiguity, through representational methods that are optimized forindication rather than specification, one has the potential to portray things in a way thatfacilitates flexible thinking. This is critical for applications involving design development, suchas architecture, where there is a need for visualization tools that facilitate the ability to work withideas at an early stage of conceptual definition, when a precise physical instantiation of themodel has not yet been determined, and one needs to foster the envisioning of multiplepossibilities [Anderson et al]. Nonphotorealistic rendering also allows the expression of multiplestyles, potentially establishing various ‘moods’ that can influence the subjective context withinwhich information is perceived and interpreted. Finally, in some applications, ambiguity has thepotential to be successfully used as a visual primitive to explicitly encode the level of accuracyor confidence in the data [Interrante 00].

3.2 Critical Features of Surface Shape that can be Captured by Lines

Gifted artists, such as Pablo Picasso, have demonstrated the ability to capture the essenceof a form in just a few critical strokes [e.g. Picasso]. In visualization, if we would like to achievea similar effect, we need to determine an algorithm for automatically defining the set of featurelines that we wish to represent. Both inspiration from art and insight from research in visualperception can be useful in helping to guide these efforts.

According to the Gestalt theory of visual perception, the process of visual understandingbegins with the separation of figure from ground. The lines that define this separation are thesilhouettes, and their use is ubiquitous in artists’ line drawings. The silhouette can be imaginedas the boundary of the shadow that would be cast by an object onto a planar background from aparallel light source oriented in the direction of the line of sight, with the locus of silhouettepoints indicating the outline of the object. Closely related to the silhouettes are the set of linescalled contours [Koenderink 90]. These are formed by the locus of points where the surfacenormal is orthogonal to the line of sight, and they typically correspond to boundaries acrosswhich there is a C0 discontinuity in depth. On a polygonally-defined object, the contour curves

can be identified as the locus of all edges shared by a front-facing polygon and a back-facingpolygon, plus all boundary edges. Complicating efforts to create effective visualizations usinghighlighted contour edges is the problem that their definition is viewpoint-dependent. Underconditions of stereo viewing, the set of surface points belonging to the contour curve defined bythe view from the left eye will not in general be the same as the set of surface points belonging tothe contour curve defined by the view from the right eye. If models are rendered in such a waythat implies a correspondence between highlighted contour curves shown in each view, the resultwill be to impede the stereo matching process and interfere with the observer’s ability toaccurately perceive the object’s 3D shape [Interrante 96]. In single images, however,highlighting silhouette and contour edges can be an effective way to emphasize the basicstructure of the form [Saito and Takahashi]. There is abundant evidence from psychophysicalresearch that our visual system is adept at inferring information about the 3D shape of an objectfrom the dynamic deformation of its silhouette as it rotates in depth [Wallach/O’Connell] andfrom the curvature characteristics of the outline of its 2D projection in static flat images[Richards].

Although silhouettes and contours are important shape descriptors, alone they are notordinarily sufficient to unambiguously describe the essential shape of a 3D object. In addition tousing silhouette and contour curves, artists often also include in their line drawings lines thatindicate other sorts of visual discontinuities. These include discontinuities in shape, shading,color, texture, and function. Recognizing this, methods have been developed for automaticallygenerating a line drawing style representation from a 2D image based on local functions of thepixel intensities [e.g. Pearson/Robinson]. Similarly, view-dependent methods have beendeveloped to identify shape-emphasizing feature lines in projected 3D data based on thedetection of C1 or C2 discontinuities in depth distance (from the viewpoint to the firstencountered surface) between neighboring image points [Saito and Takahashi].

In our research, we have sought to identify supplementary characteristic lines that can beused to highlight additional perceptually or intrinsically significant shape features of arbitrary 3Dobjects, independent of any particular viewpoint. For several reasons, the most promisingcandidates for this purpose are the valley and sharp ridge lines on a smoothly curving object andthe sharp creases, either convex or concave, on a polygonally defined surface. These lines deriveimportance first of all because they give rise to a variety of visual discontinuities. Prominentshading discontinuities are likely to be found at sharp creases on any object; on smoothly curvingforms, specular highlights are relatively more likely to be found along or near ridges, where thesurface normals locally span a relatively larger range of orientations, and sharp valleys have ahigher probability of remaining in shadow. Miller [94] demonstrated impressive results using“accessibility shading” to selectively darken narrow concave depressions in polygonally-definedmodels. Equally important as their association with shading discontinuities, crease lines, andvalley lines in particular, derive perceptual significance from their ability to specify the structuralskeleton of a 3D form in an intuitively meaningful way. Since 1954 [Attneave], psychologistshave found evidence that the points of curvature extrema play a privileged role in observers’encodings of the shape of an object’s contour. Recent research in object perception andrecognition has suggested that people may mentally represent objects as being composed of parts[Biederman85, Beusmans], with the objects perceived as subdividing into parts along their valleylines [Hoffman, Braunstein].

Drawing upon this inspiration from art, and insight from visual perception, we developedtwo distinct algorithms for highlighting feature lines on 3D surfaces for the purposes offacilitating appreciation of their 3D shapes. The first is an algorithm for highlighting valley lineson smoothly curving isointensity surfaces defined in volumetric data [Interrante 95] (figure 5).

Figure 5: By selectively highlighting valley lines, we aim to enhance the perception of important shapefeatures on a transparent surface in a visualization of 3D radiation therapy treatment planning data. Top:the skin rendered as a fully opaque surface. Middle: the skin rendered as a fully transparent surface, inorder to permit viewing of the internal structures. Bottom: the skin rendered as a transparent surface withselected opaque points, located along the valley lines. The bottom view is intended to facilitateperception/recognition of the presence of sensitive soft tissue structures that should be avoided byradiation beams in a good treatment plan. Data courtesy of Dr. Julian Rosenman. Middle and lowerimages © IEEE.

The basic algorithm works as follows. On a smoothly curving surface, valley lines aremathematically defined as the locus of points where the normal curvature is locally minimum inthe direction of least negative curvature [Koenderink90]. One can determine, for each point on asurface, whether it should be considered to lie on a ridge or valley line by computing theprincipal directions and principal curvatures of the surface at that point and then checking to seeif the magnitude of the strongest principal curvature assumes an extreme value compared to the

magnitudes of the corresponding principal curvatures at the surrounding points on the surface inthe corresponding principal direction. To compute the principal directions and principalcurvatures at any point in a volumetric dataset, one can begin by taking the grey-level gradient[Hoehne], in our case computed using a Gaussian-weighted filter over a 3x3x3 voxelneighborhood, to indicate the surface normal direction,

re3. Choosing any two arbitrary

orthogonal directions re1 and

re2 that span the tangent plane defined by

re3 gives an orthogonal

frame at the point. It is then straightforward to obtain a principal frame by computing the

Second Fundamental Form, A =

ƒ ƒ

ƒ ƒ

ωω ωω

ωω ωω

113

123

213

223

, where ƒωω ji3 is determined by the dot product of

rei

and the first derivative of the gradient in the re j direction, and then diagonalizing it to obtain

D =

κκ

κκ

1

2

0

0 and P

u u

v v

=

υυ υυ

υυ υυ

1 2

1 2

, where A = PDP-1 and κκ1 > κκ2 . The principal directions r′e1 and

r′e2 are then given by υυ υυ1 1 1 2u ve e

r r+ and υυ υυ2 1 2 2u ve e

r r+ respectively. In our implementation, we

highlighted what we determined to be perceptually significant valley lines by increasing theopacities of candidate valley points by an amount proportional to the magnitude of the principalcurvature at the point, if that curvature exceeded a fixed minimum value.

Our second algorithm [Ma] was developed for applications involving the visualization ofsurface mesh data (figure 6). In this case the goal was to determine which mesh edges were mostimportant to show in order to convey a full and accurate impression of the 3D shape of thesurface in the absence of shading cues. In addition to silhouette and contour edges, ouralgorithm marked for display selected internal crease edges where the angle betweenneighboring triangles was locally relatively sharp, in comparison with the angles subtendedacross other edges in the immediate vicinity.

Figure 6: By highlighting feature lines on a surface mesh, we may enhance appreciation of the essentialstructure of the form, a goal that assumes particular importance under conditions where the use of surfaceshading is problematic. Clockwise from top left: the full original mesh; the silhouette and contour edgesonly; the silhouette and contour edges highlighted in a surface rendering in which polygon color isdefined purely as a function of the value of a scalar parameter that is being visualized over the mesh;same as previous, except that the feature line set is augmented by crease edges determined by ouralgorithm to be locally perceptually significant. Data courtesy of Dimitri Mavriplis.

4 Case Study 3: Clarifying the 3D Shapes, and Relative Positions in Depth, of ArbitrarySmoothly Curving Surfaces via Texture

The final case study I describe in this chapter concerns the development of visualizationtechniques intended to facilitate the accurate and intuitive understanding of the 3D shapes andrelative positions in depth of arbitrary smoothly curving, transparent surfaces that are not easilycharacterized by a small number of feature lines. Examples of such surfaces arise in manyapplications in visualization, from medical imaging to molecular modeling to aerospaceengineering, where scientists seek insight into their data through the visualization of multiplelevel surfaces in one or more 3D scalar distributions. In striving to accomplish this goal I againfound great value in drawing upon both inspiration from the practices used by accomplishedartists and insight from fundamental research in visual perception.

4.1.1 Cues to 3D Shape and Depth

As a first step in determining how to most effectively convey the 3D shapes and depths ofsmoothly curving transparent surfaces in computer-generated images, it is useful to brieflyconsider the questions of 1) how we ordinarily infer shape and depth information from visual

input and 2) why the shapes and depths of transparent surfaces can be difficult to adequatelyperceive, even in everyday experience. Since a full discussion of shape and depth perception isbeyond the scope of this chapter, I focus here on the most important, relevant aspects of sceneconstruction that are typically under the control of the visualization developer: viewpoint andprojection, lighting, and the definition of surface material properties.

After occlusion, which unambiguously specifies the depth order relationships ofoverlapping surfaces, linear perspective is one of the strongest available pictorial cues to depth.Under linear perspective, lines that are parallel in 3D appear in a projected image to convergetowards one or more vanishing points as they recede into the distance, and objects accordinglyappear smaller/increasingly skewed, with increasing distance from the viewpoint/increasingeccentricity from the center of projection [Kennedy/Juricevic]. In an orthographic projection, ora perspective projection subtending a very narrow field of view, these convergence cues and sizegradients are forfeited. The selection of an appropriate vantage point with respect to an object isalso a consideration of some importance. In choosing preferred views for objects, observersappear to use task-dependent strategies [Perret/Harries]. Non-generic viewpoints, in whichaccidental alignments or occlusions of particular object features fortuitously occur, have thegreatest potential to be misleading [Freeman].

Lighting is a complex and important factor that can influence the perception of a scene inmultifaceted ways, as is well understood in fields related to the cinema and stage. Forvisualization purposes, in addition to effects of shadows on depth perception, which werementioned earlier, we need to consider how best to control lighting parameters to facilitate anaccurate perception of shape from shading. It has long been recognized that people areaccustomed to things being lit from “above” [Luckiesh], and that among the deleterious effectsof directing light toward a surface from below is an increased chance of fostering depth reversal,where convexities are perceived as concavities and vice versa [Ramachandran]. Recent researchsuggests more specifically a bias toward the assumption that lighting is coming from the above-left, possibly attributable to cerebral lateralization [Mamassian02]. Somewhat less dramatic butalso significant are the shape-enhancing effects of defining lighting to be at an oblique angle, asopposed to head-on [Johnson/Passmore], whereby shading gradients on surfaces receding indepth are enhanced.

It is perhaps in the definition of surface material properties that the greatest hithertountapped potential lies for facilitating shape and depth perception in visualizations of surfacedata. Before addressing this topic, however, I would like to discuss the remaining issue of whytransparent surfaces are so difficult to adequately perceive, which will help suggest how materialproperties might be selected to best advantage.

Plain transparent surfaces clearly provide impoverished cues to shape and depth. Shape-from-shading information is available only through the presence of specular highlights, which, inbinocular vision, are perceived not to lie on the curved surface of an object but to float behindthe surface if the object is locally convex, and in front of the surface if it is locally concave[Blake/Bulthoff]. Cues to the presence of contours, where the surface normal is orthogonal tothe line of sight, are provided by the distorting effects of refraction; however this comes at asignificant cost to the clarity of visibility of the underlying material. When artists portraytransparent surfaces, they similarly tend to rely heavily on the use of specular highlights (Lucas

Cranach) [Friedlander], specular reflection (Don Eddy) [Kuspit] and/or refractive distortion(Janet Fish) [Katz].

While it would be misleading downplay the potential merits of employing a fullyphysically correct model of transparency for visualization applications – refraction provides anatural way to emphasize silhouettes and contours, and might also provide good cues to thethickness of a transparent layer – it is at the same time clear that the visualization goal ofenabling the effective simultaneous understanding of multiple layers will not be met simply byachieving photorealism in the surface rendering. Something must be done to supplement theshape and depth cues available in the scene. The solution that we came up in our research withwas to ‘stick something onto’ the surface, in the form of carefully designed, subtle, uniformlydistributed texture markings, that can provide valuable cues to the surface shape, along withexplicit cues to the surface depth, in a way that is not possible through reliance on specularhighlights.

Before moving on to the discussion of surface textures, a final point in regard to therendering of transparent surfaces bears mentioning. There are several alternative models forsurface transparency, corresponding to different types of transparent material. The first model(figure 7a) corresponds to the case where you have an opaque material, such as a gauze curtain,that is very finely distributed, so that over a finite area it is both partly present and partly absent.This is the type of transparency represented by the ‘additive model’: I = If·αf + Ib·(1-αf) where αf

is the (wavelength independent) opacity of the transparent foreground material and If is itsintensity, while Ib is the intensity of the background material. If a surface rendered using thismodel is folded upon itself multiple times, the color in the overlap region will, in the limit,converge to the color of the material; shadows cast by this material will be black. A secondmodel (figure 7b) corresponds to the case where you have a semitransparent material thatimpedes the transmission of certain wavelengths of light. This type of transparency can beapproximated by a ‘multiplicative model’: I = αf Ib, where αf is the (wavelength dependent)transmissivity of the foreground material. Multiple layers of this material will, in the limit,converge to looking black; shadows cast by this material will be colored. Other approaches,incorporating both additive and multiplicative combinations of foreground and backgroundmaterial, are also possible.

Figure 7: Examples of two different models of transparency. Left: additive transparency, exhibited bymaterials such as gauze that are intrinsically opaque but only intermittently present; Right: multiplicativeor subtractive transparency, exhibited by materials such as colored glass that selectively filter transmittedlight.

4.1.2 Using Texture on Surfaces to Clarify Shape

Having determined to attempt to clarify the 3D shapes of external transparent surfacesthrough the addition of sparsely distributed texture markings, the question now becomes: whatsort of markings should we add? If we could define the ideal texture pattern to apply to anarbitrary smoothly curving surface in order to enable its 3D shape to be most accurately andeffectively perceived, what would the characteristics of that texture pattern be? To answer thisquestion we again turn for inspiration to the observation of the practices of artists and illustrators,and for insight to the results of research in psychology on the perception of shape from texture.

In stipple drawings, artists carefully control the density of pen markings in order to achievea desired distribution of tone. With line drawings, in addition to achieving variations in tone,there is the additional concern of carefully controlling the directions of the lines in order toemphasize the surface shape. Although there is no precise or universally recognized conventionfor defining the directions of pen strokes in line drawings, artists and illustrators have frequentlynoted the importance of using stroke direction appropriately, and have variously citedadvantages, for effectively conveying shape, in using lines which “follow the contours of theform” [Zweifel] or which run “at right angles to the length of the form” [Sullivan].

The significance of texture’s effect on shape (slant) perception was first emphasized in theresearch literature and formally studied by James Gibson [1950]. Using two different wallpaperpatterns on large flat boards, viewed through a circular aperture, he found that slant perceptionwas not only significantly more accurate under either texture condition than under the controlcondition of no texture, but also that accuracy was greatest in the case of the more “regular”texture pattern. In subsequent studies, comparing the effects of different aspects of ‘textureregularity’, researchers found evidence that regularity in element size, element shape, andelement placement all had a positive effect in improving slant perception accuracy [Flock].Ultimately, it was determined that linear convergence cues (from perspective projection) play thedominant role in slant perception from texture [Attneave, Braunstein]. Looking at the effects oftexture on shape perception in the case of curved surfaces, Cutting and Millard [1984] foundevidence primarily for the importance of ‘texture compression’ cues, manifested as the changesin the projected shapes of circular texture elements. In later studies, Cumming et. al [1993]found support for these findings. Evaluating the relative impacts of veridical size, density, andcompression gradients on curvature perception accuracy under stereo viewing conditions, theyfound that perceived curvature was least when element compression was held constant, while thepresence or absence of appropriate gradients in element size and/or spacing had little effect oncurvature perception accuracy. These results are important because they provide clear evidencethat the particular characteristics of a surface texture pattern can significantly affect shapeperception, even in the presence of robust, veridical cues to shape from stereo.

Still, it remains unclear what sort of texture we should choose to apply to a surface in orderto facilitate perception of its shape. Stevens [1981], informally observing images from theBrodatz [1966] texture album pasted onto a cylindrical form and viewed monocularly underconditions that minimized shading cues, reported obtaining a compelling impression of surfacecurvature only in the cases of the wire mesh and rattan textures, the most regular, syntheticpatterns. However, observing an extensive variety of line-based texture patterns projected onto acomplicated, doubly curved surface, obliquely oriented so as to exhibit contour occlusions, Toddand Reichel [1990] note that a qualitative perception of shape from texture seems to be afforded

under a wide range of texturing conditions. Computer vision algorithms for the estimation ofsurface orientation from texture generally work from assumptions of texture isotropy, or texturehomogeneity. Rosenholtz and Malik [1997] found evidence that human observers use cuesprovided by deviations from both isotropy and homogeneity in making surface orientationjudgments. Stone [1993] notes that particular problems are caused for perception by textureswhich are not ‘homotropic’ (in which the dominant direction of the texture anisotropy variesacross the texture pattern). Op artists such as Brigit Riley [ref] have exploited this assumption tocreate striking illusions of relief from patterns of waving lines. Mamassian and Landy [1998]found that observers’ interpretations of surface shape from simple line drawings are consistentwith the following biases under conditions of ambiguity: to perceive convex, as opposed toconcave, surfaces; to assume that the viewpoint is from above; and to interpret lines as if theywere oriented in the principal directions on a surface. Knill [2001] suggests that texture patternswith oriented components, which under the assumption of texture pattern homogeneity areconstrained to follow parallel geodesics on developable surfaces, may provide more perceptuallysalient cues to surface shape than isotropic patterns. Finally, Li and Zaidi [2000, 2001] haveshown that observers can reliably discriminate convex from concave regions in front-facingviews of a vertically oriented sinusoidally corrugated surface only when a perspective projectionis used and the texture pattern contains patterns of oriented energy that follow the first principaldirection. However, in more recent studies, considering a wider range of viewpoints, they havefound indications that the texture conditions necessary to ensure the veridical perception ofconvexity vs. concavity are more complicated than previously believed [Li 2002].

Because of historical limitations in the capabilities of classical texture mapping softwareand algorithms, with few exceptions nearly all studies investigating the effect of surface textureon shape perception that have been conducted to date have been restricted either to the use ofdevelopable surfaces – which can be rolled out to lie flat on a plane – or to the use ofprocedurally defined solid texture patterns, whose characteristics are in general independent ofthe geometry of the surfaces to which they are applied. For several years we have believed thatimportant new insights into texture’s effect on shape perception might be gained through studiesconducted under less restrictive surface and texture pattern conditions. In the final part of thissection I will describe the algorithms that we derived for the controlled synthesis of arbitrarytexture patterns over arbitrary surfaces and the results of the studies we have recently undertakenin pursuit of a deeper understanding of how we might best create and apply custom texturepatterns to surfaces in scientific datasets in order to most effectively facilitate accurateperception of their 3D shapes.

In our first studies [Interrante et. al 97], involving indirect judgments of shape and distanceperception under different texture conditions on layered transparent surfaces extracted fromradiation therapy treatment planning datasets (figure 8), we created a variety of solid texturepatterns by scan-converting individual texture elements (spheres, planes, or rectangular prisms)into a 3D volume at points corresponding to evenly distributed locations over a pre-identifiedisosurface. We found clear evidence that performance was better in the presence of texture, butwe did not find a significant main effect of texture type. As expected, “sticking something ontothe surface” helped, but the question of how best to define helpful texture markings remainedopen. Unfortunately, none of the textures we were able to achieve using this discrete elementapproach yet resembled anything one might find in an artist’s line drawing.

Figure 8: A variety of sparse textures applied to the same external transparent surface. Clockwise fromupper left: spots of various sizes (with the larger spots it is easier to infer the local surface normaldirection, but aesthetically they are more annoying to look at); short strokes locally oriented in the firstprincipal direction, with length proportional to the first principal curvature (upper right); short strokes ofrandom lengths, randomly oriented (lower right); lines parallel to various object space axes.

Shortly afterward, we developed an improved method for synthesizing a more continuoussurface texture pattern that everywhere followed the first principal direction over an arbitrarydoubly curved surface [Interrante 97]. To achieve this texture pattern we began by scattering anumber of discrete high intensity point elements evenly throughout an otherwise empty 3Dvolume, according to an approximate Poisson distribution. The locations of each point elementwere determined by dividing the volume into uniformly sized cells and randomly selecting alocation within each cell, under the restriction that no location could be selected that was withina pre-determined distance from any previously chosen point. In a separate process, weprecomputed the vector field of the first principal directions of the isolevel surfaces defined atevery point in the 3D volume dataset, using the principal direction computation approachdescribed in section 3.2 of this chapter. Finally, we used 3D line integral convolution to ‘combout’ the distributed point texture along the lines of curvature defined by the principal directionvector field (figure 9), to obtain a single solid texture that could be applied to any isolevelsurface in the volume data (figure 10). In order to avoid artifacts in the pattern at the pointswhere the first and second principal directions switched places, we forced the filter kernel lengthto reduce to zero in the vicinity of these umbilic points.

Figure 9: Left: a texture of evenly distributed points over three slices in a volume; Right: the results afteradvecting the texture according to the first principal direction vector field using 3D line integralconvolution.

Figure 10: A 3D line integral convolution texture applied to a transparent isointensity surface in a 3Dradiation therapy treatment planning dataset. Data courtesy of Dr. Julian Rosenman.

Although these results were encouraging, important tasks remained. The first was toobjectively evaluate the relative effectiveness of the new LIC-based principal direction texturingapproach, and in particular to rigorously examine the impact of texture orientation on shapeperception in the case of complicated doubly curving surfaces. The second was to pursuedevelopment of a flexible, robust principal direction texturing method that could be applied topolygonal datasets (e.g. figure 11). The two principal challenges in that regard were to define amethod for obtaining accurate estimates of the principal directions at the vertices of an arbitrarypolygonal mesh, and the second was to determine how to synthesize an arbitrary texture patternover an arbitrary doubly curved surface, in a way that avoids both seams and stretching, and suchthat the dominant orientation in the texture pattern everywhere locally follows the first principaldirection vector field.

Figure 11: A line drawing of a brain dataset, generated by Steve Haker [Girshick et. al] in which tinystraight strokes are oriented in the first principal direction at vertices in the surface mesh.

While several methods have been previously proposed for estimating principal directionsand principal curvatures at points on meshes [e.g. Taubin, Desbrun], we have found in practicethat all exhibit unexplained large errors in some cases. Recently, we set out to investigate thesources of these errors, and in the process developed a new method for principal directionestimation that appears to produce better results [Goldfeather]. Unfortunately, space does notpermit a full description of that approach here, however the essential insights are these: 1) largeerrors can, and do, occur at points that are far from being umbilic; 2) errors are most a problemwhen the underlying mesh parameterization is not regular; and 3) we can use the vertex locationand (approximate) surface normal information available at the neighboring points to a vertex toachieve a least squares cubic surface fit to the mesh at that point, which appears to offer betterpotential for a more accurate fit than when the surface is restricted to be quadratic. The openquestions that remain are: how to robustly resolve problems that arise due to the first and secondprincipal directions switching places on either side of critical points; how to gracefully determinean appropriate texture or stroke direction across patches of umbilic points, where the principaldirections are undefined; and, how to balance the concern of emphasizing shape with the concernof minimizing the salience of extraneous detail. Not all surface perturbations are worth drawingattention to, and, depending on the application, it may be desirable to enforce certain smoothnesscriteria before using a computed principal direction vector field for texture definition purposes.

In the first of our most recent experiments intended to gain insights into methods for usingtexture effectively for shape representation [Interrante et, al 00], we investigated the effect of thepresence and direction of luminance texture pattern anisotropy on the accuracy of observers’judgments of 3D surface shape. Specifically, we sought to determine 1) whether shapeperception is improved, over the default condition of an isotropic pattern, when the texturepattern is elongated in the first principal direction; and 2) whether shape perception is hindered,over the default condition of an isotropic pattern, when the texture is elongated in a constant orvarying direction other than the first principal direction. We had five participants, using asurface attitude probe [Koenderink 92], make judgments about local surface orientation at 49evenly spaced points on each of six different smoothly curving surfaces, under each of fourdifferent luminance texture conditions. All of the texture patterns for this study were created via3D line integral convolution, using either a random/isotropic vector field (rdir), a first principaldirection vector field (pdir), a vector field following a constant uniform direction (udir), or avector field following a sinusoidally varying path (sdir). Sample stimuli are shown in figure 12.The experiment was repeated under two different viewing conditions, flat and stereo. Chartssummarizing the results are shown in figure 13. In the flat viewing condition, we found thatperformance was significantly better in the cases of the pdir and rdir patterns than in the cases ofthe sdir and udir patterns. Accuracy was significantly improved in the stereo vs. flat viewingcondition, for all texture types. Performance remained marginally better in the cases of theisotropic and principal direction patterns than under the other two texture conditions, butsignificance was achieved only in the rdir case. These results are consistent with the hypothesisthat texture pattern anisotropy can impede surface shape perception when the elongated markingsare oriented in a way that is different from the principal direction, but they do not support thehypothesis that principal direction textures will facilitate shape perception to a greater extentthan isotropic patterns.

Figure 12: Representative examples of the sample stimuli used in our first recent experiment investigatingthe effect of texture orientation on the accuracy of observers’ surface shape judgments. From left to right:Isotropic (rdir), Uniform (udir), Swirly (sdir), and Principal Direction (pdir).

Figure 13: Pooled results (mean angle error) for all subjects, all surfaces, by texture type. Left: flatpresentation; the differences {pdir,rdir} < {sdir,udir} were significant at the 0.05 level. Right stereopresentation; only the differences {rdir < sdir, udir} were significant at the 0.05 level.

In a follow-up study, we repeated the experiment using displacement textures instead ofluminance textures, and found the same pattern of results. However, two important questionswere raised by this work. First: why does shape perception seem to be most accurate in theprincipal direction orientation condition, when there is little ecological justification for a texturepattern being oriented in the principal directions across a doubly curved surface? Is it because,from a generic viewpoint, the contours traced by a principal direction texture have the greatestpotential to reveal the surface curvature to a maximum extent, while the contour traced out bythe texture flow along any other direction at that point and for the same view will be intrinsicallymore flat, which may represent a loss of shape information that is not recoverable? Second: onarbitrary doubly curved surfaces there are two orthogonal directions in which the normalcurvature generically assumes a non-zero extrema. Although these directions can be reliablyclassified into two types, the first principal direction and the second principal direction, there isnot a clear algorithm for determining which of these two directions a singly-oriented directionaltexture should follow at any point, in order to minimize artifacts due to the apparent turning ofthe texture pattern in the surface. Is it possible that the effectiveness of the pdir textures used inthis first experiment was compromised by these ‘corner’ artifacts, and that we might be able tomore effectively facilitate shape perception using an orthogonally bi-directional principaldirection oriented pattern — one that has 90-degree rotational symmetry?

In order to investigate these questions we needed to conduct further studies, and to develop amore general texture synthesis method capable of achieving a wider variety of oriented patternsover surfaces.

Inspired by Efros and Leung’s [1999] algorithm for synthesizing unlimited quantities of atexture pattern that is nearly perceptually identical to a provided 2D sample, we developed a fastand efficient method for synthesizing a fitted texture pattern, without visible seams or projectivedistortion, over a polygonal model, such that the texture pattern orientation is constrained to bealigned with a specified vector field at a per-pixel level [Gorla et. al]. Our method works bypartitioning the surface into a set of equally-sized patches, then using a two-pass variant of theoriginal Efros and Leung method to synthesize a texture for each patch, being careful to maintainpattern continuity across patch boundaries, and performing the texture look-up, at each point, inan appropriately rotated copy of the original texture sample, in order to achieve the desired localpattern orientation.

Using this system to render a new set of textured surface stimuli, we undertook a secondexperiment [Interrante et. al 02] intended to evaluate the information carrying capacities of twodifferent base texture patterns (one singly oriented and one doubly oriented), under threedifferent orientation conditions (pdir, udir and sdir), and two different viewing conditions(upright and backward slanting). In a four alternative forced choice task, over the course of 672trials, three participants were asked to identify the quadrant in which two simultaneouslydisplayed B-spline surfaces, illuminated from different random directions, appeared to differ intheir shapes. We found that participants were consistently able to more reliably perceive smallershape differences when the surfaces were textured with a pattern whose orientation followed oneof the principal directions than when the surfaces were textured either with a pattern thatgradually swirled in the surface or that followed a constant uniform direction in the tangentplane. We did not find a significant effect of texture type (performance was only marginallybetter overall in the two directional case), or of surface orientation (performance was onlymarginally better overall in the tilted vs. front-facing case), nor evidence of an interactionbetween texture type and surface orientation. Sample stimuli and summary results are shown infigures 14-15. These findings support the hypothesis that anisotropic textures not aligned withthe first principal direction may support shape perception more poorly, for a generic view, thanprincipal direction oriented anisotropic patterns, which can provide cues to the maximal amountof surface normal curvature in a local region. However this study did not yield much insight intothe potential effects on shape perception of principal direction texture type.

In our third recent experiment [Kim et. al 03], we focused on the question of whethersome principal direction texture patterns might be more effect for conveying shape than others,and, if so, then what the particular characteristics of these principal direction patterns might be.Five participants each adjusted surface attitude probes to provide surface orientation estimates attwo different locations over five different surfaces under each of four different textureconditions. With five repeated measures, we had a total of 200 trials/pp. We comparedperformance under the control condition of no texture to performance under three differenttexture type conditions: a high contrast, one-directional line pattern that everywhere followed thefirst principal direction (1dir) ; a lower contrast, one-directional line integral convolution pattern

Figure 14: Representative stimuli used in our second experiment to investigate the relative extents towhich differently oriented patterns have the potential to mask the perceptibility of subtle shapedifferences. The texture conditions are, from left to right: principal direction, swirly direction, uniformdirection.

Figure 15: Summary results of our second experiment. Accuracy increased with increasing magnitude ofshape difference in all cases, but increased at a faster rate under the principal direction texture condition.Error bars represent 95% confidence intervals.

that similarly followed the first principal direction (lic); and a medium-high contrast, two-directional grid pattern that was everywhere in alignment with both principal directions (2dir).All patterns had equivalent mean luminance. Sample stimuli are shown in figure 16. We usedthe statistical software package ‘MacAnova’, developed by Prof. Gary Oehlert from theDepartment of Statistics at the University of Minnesota, to perform a three-way, within subjectsmixed analysis of variance (ANOVA) to evaluate the statistical significance of the results. Wefound significant main effects of probe location (p=0.0000264) and texture type (p=0.0002843),and a significant two-way interaction between texture type and probe location (p<0.00000001).We did not find a significant main effect of subject id (p = 0.18) nor a significant interactionbetween subject and texture type (p = 0.62). We used Tukey’s HSD (“Honestly SignificantDifference”) method to perform post-hoc pairwise comparisons of the means of the angle errorsunder the different texture conditions. We found that the following differences were statisticallysignificant at the 0.01 level: 2-dir < 1-dir, 2-dir < None, 1-dir < None, and LIC < None. Thedifference between performance in the 2-dir and LIC conditions was not statistically significantat the 0.01 level, nor was the performance difference between the LIC and the 1-dir conditions.Charts summarizing these results are shown in figure 17.

Figure 16: A test surface from our third experiment, in the control condition of no texture (left), and (fromleft to right) under the three studied principal direction texture conditions: 1dir, lic and 2dir.

Figure 17: The cumulative experiment 3 results.

Through all of the efforts summarized in this section, we have realized that determining thecharacteristics of a texture pattern that is best able to facilitate surface shape perception is not asstraightforward an undertaking as it at first might seem. Conducting controlled experiments is adelicate and time-consuming business, and success is never guaranteed. However, through ourefforts we have been able to answer many important questions about the suitability of principaldirection oriented patterns for shape representation, and the open questions that remain provide awelcome challenge for future work.

5 Conclusions

The process of creating an effective visual representation of a set of data is both an art anda science, requiring extensive efforts in visualization design, implementation and evaluation. Forvisualization design, there are significant potential benefits in seeking inspiration from previousgraphical work in art, illustration, visual communication and design, and in seeking insights fromresearch in vision and visual perception. The process of implementation – figuring out how todevelop the algorithms necessary to translate our vision of the results we want to achieve into areality – though dealt with only lightly in this chapter, are of extreme importance, and this hashistorically been where the field of visualization has seen its greatest successes. Evaluation,through informal observation or, more rigorously, through controlled observer experiments, canbe critical in clarifying our understanding of the strengths and weaknesses of alternativevisualization approaches, and for assessing the practical merits of a particular visualizationapproach for a specific task. Most importantly, evaluation helps us to better understand not onlywhat works and what doesn’t, and by how much, but also to gain insight into why. Based on thisinsight, we are better equipped to go back to the design stage and work on developing yet moreeffective approaches to meet our visualization objectives.

References

[1] Lucile R. Addington. Lithic Illustration: drawing flaked stone artifacts for publication, TheUniversity of Chicago Press, 1986

[2] Gerald R. Allen and D. Ross Robertson. Fishes of the Tropical Eastern Pacific, Universityof Hawaii Press, 1994

[3] Lee Anderson, James Esser and Victoria Interrante. A Virtual Environment for ConceptualDesign in Architecture, 9th Eurographics Workshop on Virtual Environments/7thInternational Workshop on Immersive Projection Technology, May 2003, to appear.

[4] Arthur Appel, F. James Rohlf and Arthur J. Stein. The Haloed Line Effect for Hidden LineElimination, Proceedings of SIGGRAPH '79, pp. 151-157.

[5] Fred Attneave and Richard K. Olson. Inferences About Visual Mechanisms fromMonocular Depth Effects, Psychonomic Science, 4, 1966, pp. 133-134.

[6] Fred Attneave. Some Informational Aspects of Visual Perception, Psychological Review,61(3), 1954, pp. 183-193.

[7] David C. Banks. Illumination in Diverse Codimensions, Computer Graphics Proceedings,Annual Conference Series, 1994, pp. 327-334.

[8] Irving Biederman and Ginny Ju. Surface versus Edge-Based Determinants of VisualRecognition, Cognitive Psychology, 20(1), January 1988, pp. 38-64.

[9] Irving Biederman. Human Image Understanding: Recent Research and a Theory, in Humanand Machine Vision, Azriel Rosenfeld, ed., Academic Press, 1985, pp. 13-57.

[10] Andrew Blake and Heinrich Bülthoff. Shape from Specularities: Computation andPsychophysics, Philosophical Transactions of the Royal Society of London, B, 331, 1991,pp. 237-252.

[11] Myron L. Braunstein and John W. Payne. Perspective and Form Ratio as Determinants ofRelative Slant Judgments, Journal of Experimental Psychology, 81(3), 1969, pp. 584-590.

[12] Myron L. Braunstein, Donald D. Hoffman and Asad Saidpour. Parts of Visual Objects: Anexperimental test of the minima rule, Perception, 18(6), 1989, pp. 817-826.

[13] Phil Brodatz. Textures; a photographic album for artists and designers, Dover, 1966

[14] Vicki Bruce, Elias Hanna, Neal Dench, Pat Healey and Mike Burton. The Importance of'Mass' in Line Drawings of Faces, Applied Cognitive Psychology, 6(7), December 1992, pp.619-628.

[15] Brian Cabral and Leith (Casey) Leedom. Imaging Vector Fields Using Line IntegralConvolution, Computer Graphics Proceedings, Annual Conference Series, 1993, pp. 263-270.

[16] Patrick Cavanagh and Yvan G. Leclerc. Shape from Shadows, Journal of ExperimentalPsychology: Human Perception and Performance, 15(1), February 1989, pp. 3-27.

[17] Bruce G. Cumming, Elizabeth B. Johnston and Andrew J. Parker, Effects of DifferentTexture Cues on Curved Surfaces Viewed Stereoscopically, Vision Research, 33(5/6),1993, pp. 827-838.

[18] James E. Cutting and Robert T. Millard. Three Gradients and the Perception of Flat andCurved Surfaces, Journal of Experimental Psychology: General, 113(2), 1984, pp. 198-216.

[19] Graham Davies, Hadyn Ellis and John Shepherd. Face Recognition Accuracy as a Functionof Mode of Representation, Journal of Applied Psychology, 63(2), April 1978, pp. 180-187.

[20] Mathieu Desbrun, Mark Meyer, Peter Schroder, and Alan H. Barr. Discrete DifferentialGeometry Operators in nD, preprint, July 22, 2000

[21] Brian D. Dillon, ed. The Student's Guide to Archaeological Illustrating, Institute ofArchaeology, University of California, Los Angeles, 1981

[22] Alexei A. Efros and Thomas K. Leung. Texture Synthesis by Non-Parametric Sampling,Proceedings of the International Conference on Computer Vision, 2, 1999, pp. 1033-1038.

[23] Howard R. Flock and Anthony Moscatelli. Variables of Surface Texture and Accuracy ofSpace Perceptions, Perceptual and Motor Skills, 19, 1964, pp. 327-334.

[24] William T. Freeman. The Generic Viewpoint Assumption in a Framework for VisualPerception, Nature, 368(6471), April 1994, pp. 542-545.

[25] Max J. Friedländer and Jakob Rosenberg, The Paintings of Lucas Cranach, CornellUniversity, 1978

[26] Diana Fussel and Ane Haaland. Communicating with Pictures in Nepal: results of practicalstudy used in visual education, Educational Broadcasting International, 11(1), March1978, pp. 25-31.

[27] James J. Gibson. The Perception of the Visual World, Houghton-Mifflin, 1950

[28] James J. Gibson. The Perception of Visual Surfaces, American Journal of Psychology, 63,1950, pp. 367-384.

[29] Ahna Girshick, Victoria Interrante, Steve Haker and Todd LeMoine. Line DirectionMatters: An Argument for the Use of Principal Directions in 3D Line Drawings, FirstInternational Symposium on Non Photorealistic Animation and Rendering, June 2000, pp.43-52.

[30] Jack Goldfeather and Victoria Interrante. A Novel Cubic-Order Algorithm forApproximating Principal Direction Vectors, ACM Transactions on Graphics, in re-review.

[31] Gabriele Gorla, Victoria Interrante and Guillermo Sapiro. Texture Synthesis for 3D ShapeRepresentation, IEEE Transactions on Visualization and Computer Graphics, 2003, toappear.

[32] Arthur L. Guptill. Rendering in Pen and Ink, Watson-Guptill Publications, 1976

[33] Jack M. H. Beusmans, Donald D. Hoffman and Bruce M. Bennett. A Description of SolidShape and its Inference from Occluding Contours, Journal of the Optical Society ofAmerica. A, 4(7), July 1987, pp. 1155-1167.

[34] Karl Heinz Höhne and Ralph Bernstein. Shading 3D-Images from CT Using Gray-LevelGradients, IEEE Transactions on Medical Imaging, 5(1), 1986, pp. 45-47 (with a correctionin 5(3):165).

[35] Kathy Hirsh and Deirdre A. McConathy. Picture Preferences of Thoracic Surgeons,Journal of BioCommunications, Winter 1986, pp. 26-30.

[36] Elaine R.S. Hodges. The Guild Handbook of Scientific Illustration, Van NostrandReinhold, 1989

[37] Donald D. Hoffman and Whitman A. Richards. Parts of Recognition, Cognition, 18(1-3),December 1984, pp. 65-96.

[38] Helen Hu, Amy A. Gooch, William B. Thompson, Brian E. Smits, John J. Rieser and PeterShirley. Visual Cues for Imminent Object Contact in Realistic Virtual Environments,Proceedings of IEEE Visualization 2000, pp. 179-185.

[39] Eduard Imhof. Cartographic Relief Presentation, De Gruyter, 1982

[40] Victoria Interrante and Chester Grosch. Visualizing 3D Flow, IEEE Computer Graphicsand Applications, 18(4), July 1998, pp. 49-53.

[41] Victoria Interrante and Sunghee Kim. Investigating the Effect of Texture Orientation on thePerception of 3D Shape, SPIE Conference on Human Vision and Electronic Imaging VI,SPIE 4299, January 2001, pp. 330-339.

[42] Victoria Interrante, Henry Fuchs and Stephen Pizer. Enhancing Transparent Skin Surfaceswith Ridge and Valley Lines, Proceedings of IEEE Visualization '95, pp. 52-59.

[43] Victoria Interrante, Henry Fuchs and Stephen Pizer. Conveying the 3D Shape of SmoothlyCurving Transparent Surfaces via Texture, IEEE Transactions on Visualization andComputer Graphics, 3(2), April-June 1997, pp. 98-117.

[44] Victoria Interrante, Sunghee Kim and Haleh Hagh-Shenas. Conveying 3D Shape withTexture: Recent Advances and Experimental Findings, Human Vision and ElectronicImaging VII, SPIE 4662, January 2002, pp. 197-206.

[45] Victoria Interrante. Harnessing Rich Natural Textures for Multivariate Visualization, IEEEComputer Graphics and Applications, 20(6), November 2000, pp. 6-11.

[46] Victoria Interrante. Illustrating Transparency: communicating the 3D shape of layeredtransparent surfaces via texture, PhD Dissertation, UNC-Chapel Hill, 1996

[47] Victoria Interrante. Illustrating Surface Shape in Volume Data via Principal Direction-Driven 3D Line Integral Convolution, Computer Graphics Proceedings, AnnualConference Series, 1997, pp. 109-116.

[48] Koenderink, Jan J., Andrea van Doorn and Astrid M. L. Kappers. Surface Perception inPictures, Perception, 52, 1992, pp. 487-496.

[49] Alan Johnson and Peter J. Passmore. Shape from Shading I: Surface Curvature andOrientation, Perception, 23(2), February 1994, pp. 169-189.

[50] Vincent Katz. Janet Fish: Paintings, Harry N. Abrams, New York, 2002

[51] John M. Kennedy and Igor Juricevic. Foreshortening Gives Way to Forelengthening,Perception, 31(7), July 2002, pp. 893-894.

[52] Dan Kersten, Pascal Mamassian and David C. Knill. Moving Cast Shadows InduceApparent Motion in Depth, Perception, 26(2), February 1997, pp. 171-192.

[53] Sunghee Kim, Haleh Hagh-Shenas and Victoria Interrante. Showing Shape with Texture:Two Directions Seem Better than One, Human Vision and Electronic Imaging VIII, SPIE5007, January 2003, pp. 332-339.

[54] David C. Knill. Contour into Texture: Information Content of Surface Contours andTexture Flow, Journal of the Optical Society of America, A, (1), January 2001, pp. 12-35.

[55] Jan J. Koenderink. Solid Shape, MIT Press, 1990

[56] Donald Kuspit. Don Eddy: the art of paradox, Hudson Hills Press, 2002

[57] Andrea Li and Qasim Zaidi. Perception of Three-Dimensional Shape from Texture is Basedon Patterns of Oriented Energy, Vision Research, 40(2), January 2000, pp. 217-242.

[58] Andrea Li and Qasim Zaidi. Information Limitations in Perception of Shape from Texture,Vision Research, 41(12), June 2001, pp. 1519-1533.

[59] Andrea Li and Qasim Zaidi. Limitations on Shape Information Provided by Texture Cues,Vision Research, 42(7), March 2002, pp. 815-835.

[60] William E. Loechel. Medical Illustration; a guide for the doctor-author and exhibitor,Charles C. Thomas, 1964

[61] Andrew Loomis. Creative Illustration, The Viking Press, 1947

[62] Matthew Luckiesh. Light and Shade and their applications, Van Nostrand, 1916

[63] Kwan-Liu Ma and Victoria Interrante. Extracting Feature Lines from 3D UnstructuredGrids, Proceedings of IEEE Visualization '97, pp. 285-292.

[64] Cindee Madison, William Thompson, Daniel Kersten, Peter Shirley and Brian Smits. Useof Interreflection and Shadow for Surface Contact, Perception & Psychophysics, 63(2),February 2001, pp. 187-194.

[65] Pascal Mamassian and Ross Goutcher. Prior Knowledge on the Illumination Position,Cognition, 81(1), August 2001, pp. B1-B9.

[66] Pascal Mamassian and Michael S. Landy. Observer Biases in the 3D Interpretation of LineDrawings, Vision Research, 38(18), September 1998, pp. 2817-2832.

[67] Scott McCloud. Understanding Comics: the invisible art, Harper Perennial, 1994

[68] Gavin Miller. Efficient Algorithms for Local and Global Accessibility Shading, ComputerGraphics Proceedings, Annual Conference Series, 1994, pp. 319-326.

[69] Olivier Monga and Serge Benayoun. Using Partial Derivatives of 3D Images to ExtractTypical Surface Features, Computer Vision & Image Understanding, 61(2), March 1995,pp. 171-189.

[70] Ken Nakayama and Shinsuke Shimojo. Da Vinci Stereopsis: Depth and SubjectiveContours from Unpaired Image Points, Vision Research, 30(11), 1990, pp. 1811-1825.

[71] Yuri Ostrovsky, Patrick Cavanagh and Pawan Sinha. Perceiving IlluminationInconsistencies in Scenes, AI Memo #2001-029, MIT, November 2001

[72] Don E. Pearson and John A. Robinson. Visual Communication at Very Low Data Rates,Proceedings of the IEEE, 73(4), April 1985, pp. 795-812.

[73] David I. Perrett and Mark H. Harries. Characteristic Views and the Visual Inspection ofSimple Faceted Objects and Smooth Objects: 'Tetrahedra and Potatoes', Perception, 17(6),June 1988, pp. 703-720.

[74] Pablo Picasso. Study of a Bull's Head, 5 November 1952

[75] Vilayanur S. Ramachandran. Perceiving Shape from Shading, Scientific American, 259(2),August 1988, pp. 76-83.

[76] Gillian Rhodes, Susan Brennan and Susan Carey. Identification and Ratings of Caricatures:implications for mental representations of faces, Cognitive Psychology, 19(4), October1987, pp. 473-497.

[77] Whitman Richards, Jan Koenderink and Donald Hoffman. Inferring Three-DimensionalShapes from Two-Dimensional Silhouettes, Journal of the Optical Society of America, A,Optics and Imaging Science, 4, July 1987, pp. 1168-1175.

[78] John L. Ridgway. Scientific Illustration, Stanford University Press, 1938

[79] Ruth Rosenholtz and Jitendra Malik. Surface Orientation from Texture: Isotropy orHomogeneity (or Both)?, Vision Research, 37(16), August 1997, pp. 2283-2293.

[80] T. A. Ryan and Carol B. Schwartz. Speed of Perception as a Function of Mode ofRepresentation, American Journal of Psychology, 69, 1956, pp. 60-69.

[81] Takafumi Saito and Tokiichiro Takahashi. Comprehensible Rendering of 3-D Shapes,Computer Graphics, 24(4), 1990, pp. 197-206.

[82] Detlev Stalling and Hans-Christian Hege. Fast and Resolution Independent Line IntegralConvolution, Computer Graphics Proceedings, Annual Conference Series, 1995, pp. 249-256.

[83] Detlev Stalling, Malte Zöckler and Hans-Christian Hege. Fast Display of Illuminated FieldLines, IEEE Transactions on Visualization and Computer Graphics, 3(2), April/June 1997,pp. 118-128.

[84] Kent A. Stevens. The Information Content of Texture Gradients, Biological Cybernetics,42, 1981, pp. 95-105.

[85] James V. Stone. Shape From Local and Global Analysis of Texture, PhilosophicalTransactions of the Royal Society of London, B, 339, 1993, pp. 53-65.

[86] Edmund J. Sullivan. Line; an art study, Chapman & Hall, 1922

[87] Gabriel Taubin. Estimating the Tensor of Curvature of a Surface from a PolyhedralApproximation, Proceedings of the 5th International Conference on Computer Vision(ICCV'95), June 1995, pp. 902-907.

[88] James T. Todd and Francene D. Reichel. Visual Perception of Smoothly Curved Surfacesfrom Double-Projected Contour Patterns, Journal of Experimental Psychology: HumanPerception and Performance, 16(3), 1990, pp. 665-674.

[89] Hans Wallach and D. N. O'Connell. The Kinetic Depth Effect, Journal of ExperimentalPsychology, 45(4), April 1953, pp. 205-217.

[90] Ernest W. Watson. The Art of Pencil Drawing, Watson-Guptill Publications, 1968

[91] Charles Wheatstone. On some remarkable, and hitherto unobserved, phenomena ofbinocular vision, Philosophical Transactions of the Royal Society of London, 128, 1838,pp. 371-394.

[92] Albert Yonas, Lynn T. Goldsmith and Janet L. Hallstrom. Development of Sensitivity toInformation Provided by Cast Shadows in Pictures, Perception, 7(3), 1978, pp. 333-341.

[93] Frances W. Zweifel. A Handbook of Biological Illustration, University of Chicago Press,1961.