THE ART OF SEEING: VISUAL PERCEPTION IN asantell/ art of seeing: visual perception in design and evaluation of non-photorealistic rendering by anthony santella a dissertation submitted

  • View
    215

  • Download
    3

Embed Size (px)

Text of THE ART OF SEEING: VISUAL PERCEPTION IN asantell/ art of seeing: visual perception in design and...

  • THE ART OF SEEING: VISUAL PERCEPTION INDESIGN AND EVALUATION OF

    NON-PHOTOREALISTIC RENDERING

    BY ANTHONY SANTELLA

    A Dissertation submitted to the

    Graduate SchoolNew Brunswick

    Rutgers, The State University of New Jersey

    in partial fulfillment of the requirements

    for the degree of

    Doctor of Philosophy

    Graduate Program in Computer Science

    Written under the direction of

    Doug DeCarlo

    and approved by

    New Brunswick, New Jersey

    May, 2005

    guestPlaced Image

  • ABSTRACT OF THE DISSERTATION

    The Art of Seeing: Visual Perception in Design and

    Evaluation of Non-Photorealistic Rendering

    by Anthony Santella

    Dissertation Director: Doug DeCarlo

    Visual displays such as art and illustration benefit from concise presentation of in-

    formation. We present several approaches for simplifying photographs to create such

    concise, artistically abstracted images. The difficulty of abstraction lies in selecting

    what is important. These approaches apply models of human vision, models of image

    structure, and new methods of interaction to select important content. Important loca-

    tions are identified from eye movement recordings. Using a perceptual model, features

    are then preserved where the viewer looked, and removed elsewhere. Several visual

    styles using this method are presented. The perceptual motivation for these techniques

    makes predictions about how they should effect viewers. In this context, we validate

    our approach using experiments that measure eye movements over these images. Re-

    sults also provide some interesting insights into artistic abstraction and human visual

    perception.

    ii

  • Acknowledgements

    Thanks go to the many people whose help and support was essential in making this

    work possible. None of this would have happened without my advisor Doug DeCarlo.

    Thanks go also to my other committe members: Adam Finkelstein, Eileen Kowler,

    Casimir Kulikowski and Peter Meer for their advice and encouragement at various (in

    some cases many) stages of this process.

    Thanks go also to the many friends and family members who have supported and

    kept me sane through this long process. I wouldnt have survived it without my parents

    and brothers Nick and Dennis. Special thanks go to Bethany Weber. Thanks also to

    Jim Housell, all the old NYU crowd, the grad group at St. Peters and all the supportive

    souls in the CS Department, RuCCS and the VILLAGE.

    Finally, thanks go to Phillip Greenspun for photos used in several renderings that

    appear in chapters 7 and 9, as well as models Marybeth Thomas, Adeline Yeo and

    Franco Figliozzi. Special thanks to Georgio Dellachiesa for looking equally thoughtful

    in countless illustrative examples.

    iii

  • Table of Contents

    Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ii

    Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii

    List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii

    1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

    1.1. Inspirations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

    1.1.1. Artistic Practice . . . . . . . . . . . . . . . . . . . . . . . . 3

    1.1.2. Psychology . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

    1.1.3. Computer Graphics . . . . . . . . . . . . . . . . . . . . . . . 7

    1.2. Our Goal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

    2. Abstraction in Computer Graphics . . . . . . . . . . . . . . . . . . . . 11

    2.1. Manual Annotation . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

    2.2. Automatic Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

    2.3. Level Of Detail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

    3. Human Vision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

    3.1. Eye Movements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

    3.1.1. Eye Movement Control . . . . . . . . . . . . . . . . . . . . . 19

    3.1.2. Salience Models . . . . . . . . . . . . . . . . . . . . . . . . 20

    3.2. Eye Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

    3.3. Limits of Vision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

    3.3.1. Models of Sensitivity . . . . . . . . . . . . . . . . . . . . . . 24

    3.3.2. Sensitivity Away from the Visual Center . . . . . . . . . . . . 26

    3.3.3. Applicability to Natural Imagery . . . . . . . . . . . . . . . . 26

    iv

  • 4. Vision and Image Processing . . . . . . . . . . . . . . . . . . . . . . . 30

    4.1. Image Structure Features and Representation . . . . . . . . . . . . . 30

    4.2. Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

    4.3. Edge Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

    5. Our Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

    5.1. Eye tracking as Interaction . . . . . . . . . . . . . . . . . . . . . . . 38

    5.2. Using Visibility for Abstraction . . . . . . . . . . . . . . . . . . . . . 40

    6. Painterly Rendering . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

    6.1. Image Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

    6.2. Applying the Limits of Vision . . . . . . . . . . . . . . . . . . . . . 43

    6.3. Rendering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

    6.4. Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

    7. Colored Drawings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

    7.1. Feature Representation . . . . . . . . . . . . . . . . . . . . . . . . . 50

    7.1.1. Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . 50

    7.1.2. Edges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

    7.2. Perceptual Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

    7.3. Rendering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

    7.4. Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

    8. Photorealistic Abstraction . . . . . . . . . . . . . . . . . . . . . . . . . 64

    8.1. Image Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

    8.2. Measuring Importance . . . . . . . . . . . . . . . . . . . . . . . . . 65

    8.3. Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . 67

    9. Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

    v

  • 9.1. Evaluation of NPR . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

    9.1.1. Analysis of Eye Movement Data . . . . . . . . . . . . . . . . 75

    9.2. Experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

    9.2.1. Stimuli . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

    9.2.2. Subjects . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

    9.2.3. Physical Setup . . . . . . . . . . . . . . . . . . . . . . . . . 78

    9.2.4. Calibration and Presentation . . . . . . . . . . . . . . . . . . 79

    9.3. Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80

    9.3.1. Data Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . 80

    9.3.2. Statistical Analysis . . . . . . . . . . . . . . . . . . . . . . . 82

    9.4. Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

    9.4.1. Quantitative Results . . . . . . . . . . . . . . . . . . . . . . 86

    9.4.2. Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

    9.5. Evaluation Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . 92

    10. Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

    10.1. Image Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

    10.1.1. Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . 95

    10.1.2. Edges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99

    10.2. Perceptual Models . . . . . . . . . . . . . . . . . . . . . . . . . . . 100

    10.3. Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102

    11. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105

    References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108

    Curriculum Vita . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116

    vi

  • List of Figures

    1.1. (a) Henri de Toulouse-Lautrecs Moulin RougeLa Goulue (Litho-

    graphic print in four colors, 1891). (b) Odd Nerdrums Self-portrait

    as Baby (Oil, 2000). Artists control detail as well as other features

    such as color and texture to focus a viewer on important features and

    create a mood. La Goulues swirling under-dress is a highly detailed

    focal point of the image, and contributes to the pictures air of reck-

    less excitement. Artists have a fair amount of latitude in how they

    allocate detail to create an effect. Nerdrum renders his eyes (usually

    one of the most prominent features in a portrait) in a sfumato style

    that makes them almost nonexistent. Detail is instead allocated to the

    childs prophetic gesture. These choices change a common baby pic-

    ture into something mysterious and unsettling. . . . . . . . . . . . . 4

    1.2. Judith Schaechters, Corona Borealis (Stained glass, 2001). Skill-

    ful artists use t