Visual Perception in Familiar, Complex TasksVisual Perception in Familiar, Complex Tasks Jeff B....

Visual Perception in Familiar, Complex TasksVisual Perception in Familiar, Complex Tasks

Jeff B. Pelz, Roxanne Canosa, Jason Babcock, & Eric KnappenbergerVisual Perception Laboratory

Carlson Center for Imaging ScienceRochester Institute of Technology

Students & other collaboratorsStudents & other collaborators

Students:Roxanne Canosa (Ph.D. Imaging Science)Jason Babcock (MS Color Science) Eric Knappenberger (MS Imaging Science)

Collaborators:Mary Hayhoe UR Cognitive ScienceDana Ballard UR Computer Science

Visual Perception in Familiar, Complex TasksVisual Perception in Familiar, Complex Tasks

Goals:

� To better understand:Visual Perception and Cognition

� To inform design of: AI and Computer Vision Systems

Strategic VisionStrategic Vision

A better understanding of the strategies and attentional mechanisms underlying visual perception and cognition in human observers can inform us on valuable approaches to computer-based perception.

Motivation: Cognitive ScienceMotivation: Cognitive Science

The mechanisms underlying visual perception in humans are common to those needed to implement successful artificial vision systems.

� Sensorial Experience

� High-level Visual Perception

� Attentional Mechanisms

� Eye Movements

Visual Perception and Cognition

Motivation: Computer ScienceMotivation: Computer Science

The mechanisms underlying visual perception in humans are common to those needed to implement successful artificial vision systems.

� Computer Vision

� Active Vision

� Attentional Mechanisms

� Eye Movements

Artificial Intelligence

MotivationMotivation

As computing power and system sophistication grow, basic computer vision in constrained environments has become more tractable. But the value of computer vision systems will be shown in their ability to perform higher level actions in complex settings.

Limited Computational ResourcesLimited Computational Resources

Even in the face of Moore’s Law, computers will not have sufficient power in the foreseeable future to solve the “vision problem” of image understanding by brute force.

ChallengesChallenges

Computer-based perception faces the same fundamental challenge that human perception did during evolution:

limited computational resources

Inspiration Inspiration -- “Active Vision”“Active Vision”

Active vision was the first step. Unlike traditional approaches to computer vision, active vision systems focused on extracting information from the scene rather than brute-force processing of static, 2D images.

Inspiration Inspiration -- “Active Vision”“Active Vision”

Visual routines were an important component of the approach. These pre-defined routines are scheduled and run to extract information when and where it is needed.

Goal Goal -- “Strategic Vision”“Strategic Vision”

“Strategic Vision” will focus on high-level, top-down strategies for extracting information from complex environments.

Goal Goal -- “Strategic Vision”“Strategic Vision”

A goal of our research is to study human behavior in natural, complex tasks because we are unlikely to identify these strategies in typical laboratory tasks that are usually used to study vision.

Limited Computational ResourcesLimited Computational Resources

In humans, “computational resources” means limited neural resources: even if the entire cortex were devoted to vision, there are not enough neurons to process and represent the full visual field at high acuity.

Limited Neural ResourcesLimited Neural Resources

The solutions favored by evolution confronted the problem in three ways;

1. Anisotropic sampling of the scene

2. Serial execution

3. Limited internal representations

1. Anisotropic sampling of the scene1. Anisotropic sampling of the scene

Retinal design:The “foveal compromise” design employs very high spatial acuity in a small central region (the fovea) coupled with a large field-of-view surround with limited acuity.

periphery center periphery

Demonstration of the “Demonstration of the “Foveal Compromise”Foveal Compromise”

Stare at the “+” below.

Without moving your eyes, read the text presented in the next slide:

Anisotropic Sampling: Anisotropic Sampling: Foveal CompromiseFoveal Compromise

+ If you can read this you must be cheating

Anisotropic Sampling: Anisotropic Sampling: Foveal CompromiseFoveal Compromise

Despite the conscious percept of a large, high-acuity field-of-view, only a small fraction of the field is represented with sufficient fidelity for tasks requiring even moderate acuity.

2. Serial execution

Serial ExecutionSerial Execution

The limited acuity periphery must be sampled by the high-acuity fovea. This sampling imposes a serial flow of information, with successive views.

Serial Execution: Eye MovementsSerial Execution: Eye Movements

Extraocular Muscles:Three pairs of extra-ocular muscles provide a rich suite of eye movements that can rapidly move the point of gaze to sample the scene or environment with the high-acuity central fovea.

Background: Eye Movement TypesBackground: Eye Movement Types

Smooth pursuit/Optokinetic response

Vestibular-ocular response

Vergence

Saccades

Imagestabilization during object

and/or observermotion

Image destabilization - used to shift gaze to new locations.

Background: Eye Movement TypesBackground: Eye Movement Types

Saccadic Eye Movements:

Rapid, ballistic eye movements that move the eye from point to point in the scene

Separated by fixations during which the retinal image is stabilized

Serial Execution; Image PreferenceSerial Execution; Image Preference

3 sec viewing

Serial Execution: Complex TasksSerial Execution: Complex TasksSaccadic eye movements are also seen in complex tasks

2. Serial execution

Integration of Successive FixationsIntegration of Successive Fixations

How are successive views integrated?What fidelity is the integrated representation?

[fixate the cross]

Perhaps fixations are integrated in an internal representation

Limited Representation: “Change blindness”Limited Representation: “Change blindness”

If successive fixations are used to build up a high-fidelity internal representation, then it should be easy to detect even small differences between two images.

Try to identify the difference between Try to identify the difference between Image Image AA & B& B

Image A

Try to identify the difference between Try to identify the difference between Image A & BImage A & B

Try to identify the difference between Try to identify the difference between Image A & Image A & BB

Image B

Image A

Image B

Image A

Image B

The question: The question:

Can we identify oculomotor strategies that observers use to ease the computational and memory load on observers as they perceive the real world?

The question: The question:

To answer that question, we have to monitor eye movements in the real world, as people perform real extended tasks.

One problem is the hardware:

Measuring eye movementsMeasuring eye movements

Many eyetracking systems require that head move-ments (and other natural behaviors) be restricted.

Dual Purkinje eyetrackerScleral eye-coils

RIT Wearable EyetrackerRIT Wearable Eyetracker

The RIT wearable eyetracker is a self-contained unit that allows monitoring subjects’ eye movements during natural tasks.

RIT Wearable EyetrackerRIT Wearable Eyetracker

The headgear holds CMOS cameras and IR source. Controller and video recording devices are held in a backpack.

Perceptual strategies Perceptual strategies

Beyond the mechanics of how the eyes move during real tasks, we are interested in strategies that may support the conscious perception that is continuous temporally as well as spatially.

When subjects’ eye movements are monitored as they perform familiar complex tasks, novel sequences of eye movements were seen that demonstrate strategies that simplify the perceptual load of these tasks.

In laboratory tasks, subjects usually fixate objects of immediate relevance. When we monitor subjects performing complex tasks, we also observe fixations on objects that were relevant only to future interactions.

The next slide shows a series of fixations as a subject approached a sink and washed his hands:

““LookLook--ahead” ahead” FixationsFixations

Early fixations are on the water faucet; the first object for interaction.

t = -700 msec t = 0 msec

Even before the faucet is reached, the subject looks ahead to the soap dispenser.

t = 1500 msec

Fixations return to items of immediate interaction.

t = 2000 msec

Gaze returns again to the dispenser just before the reach to it begins.

t = 2600 msec

Gaze remains on the dispenser until the hand reaches it.

t = 2800 msec

Perceptual StrategiesPerceptual Strategies: Where, when, why?: Where, when, why?

The “look-ahead” fixations are made in advance of “guiding” fixations used before interacting with objects in the environment.

guiding fixationlook-ahead fixation interaction

2000 msec 800 msec

Conclusion Conclusion

Humans employ strategies to ease the computational and memory loads inherent in complex tasks. Look-aheadfixations reveal one such strategy:

opportunistic execution of information-gathering routines to pre-fetch information needed for future subtasks.

Future Work Future Work

Implementing this form of opportunistic execution in artificial vision systems can test the hypothesis that strategic visual routines observed in humans can benefit computer-based perceptual systems.

Visual Perception in Familiar, Complex TasksVisual Perception in Familiar, Complex Tasks Jeff B....

Documents

CAREER WORKSHOP APRIL 9, 2014 Selecting a Program Anant Paravastu Chemical Engineering FLORIDA STATE UNIVERSITY Ken Knappenberger Chemistry and Biochemistry

Intellix BMT 300 - EKOS GROUP, Mobil Merkez Ürünleri ... · PDF fileKelman TRANSFIX™ family of dissolved gas analysis ... • Familiar Perception software included for download

Distinct neural processes for the perception of familiar ...€¦ · evident early (sub-200ms), involving the core face network (e.g., Eimer, 2000). The later N250 and P300 components

256-339-7406 Tracie Rakestraw, Children’s Director … · Staff members may be reached by calling the church office: 256-739-0505. Andrew Siegenthaler, Pastor HRISTJonathan Knappenberger,

State-Resolved Metal Nanoparticle Dynamics Viewed through ...fonlo.org/2019/readingmaterials/Knappenberger 2.pdf · electromagnetically coupled network.4 The optical properties of

Consumer Perception Perception.pdf · perception. • To become familiar with the Gestalt view of perception. • To understand the process of perceptual categorization and inference

Investor Perception Survey of CARIFORUM - Caribbean · PDF file3.3 FDI determinants ... Only questionnaires completed by a decision-maker familiar with the company’s investment strategy

Sensation and Perception - CourseNotes · Sensation and Perception 183 PSYCHOLOGY: Exploring Behavior Chapter: Sensation and Perception Sensation and Perception Sensation versus Perception

Assessment of the Domestic Perception of Armenian ... · Armenian pharmaceutical products familiar to respondents 20 ... without reminding. After some hints very few respondents recall

Processing deficits for familiar and novel faces in ... · Binder et al., 2006). Visual processing deficits in PA could therefore undermine the rapid and accurate perception of the

Attractiveness Enhances The Perceived Familiarity Of ... · The face-space model of face perception (Valentine, 1991) proposes that faces ... • no effect of attractiveness for familiar

€€ Nominator (Nominador): Lucy Knappenberger €€ Nominee ......€€ Award Category €€ (Categoría de premio): Omni Service Champion €€ Reason for Nomination €€

Perception Chapter 6. Perception Selective Attention Perceptual Illusions Perceptual Organization Form Perception Motion Perception Perceptual Constancy

Urban Codes // Parallel Worlds · experiences to alter perception of existing or familiar urban spaces. This paper is a reflection on a mixed reality artwork created ... presents

Exploring Students Students Students Students’ Perception Perception Perception Perception of and Reaction Reaction Reaction Reaction To

PERCEPTION 120 PERCEPTION 220

1 Perception Chapter 6. 2 Perception Selective Attention Perceptual Illusions Perceptual Organization Form Perception Motion Perception Perceptual

Is Perceptual Experience Normally Multimodal? · Web viewThere is a host of familiar phenomena—in particular, speech perception and the perception of spatial phenomena—that philosophers

EVALUCIÓN DEL IMPACTO FAMILIAR · Evaluación del impacto familiar del asma bronquial infantil 166 Mishel, M. (1983). Parent’s perception of uncertainty concerning their hospitalized

Cognition Definition of Perception Sub Process of Perception Perception Selectivity