Depth cues

How your brain and eyes make 3

dimensions out of 2?

Objective

After participating in this lecture you’ll be able to:-

• Break into detail the cues to depth that normally operates simultaneously.

Depth Perception

• Depth perception is the visual ability to perceive the world in 3D.

• It is a trait common to many higher animals.

• Depth perception allows the beholder to accurately estimate the distance to an object.

How to make a 2D image appear as 3D!

• Output is typically 2D Images

• Yet we want to show a 3D world!

• How can we do this?– We can include ‘cues’ in the image that give

our brain 3D information about the scene– These cues are visual depth cues

Visual Depth Cues• Monoscopic Depth Cues (single 2D image)

• Binocular / Stereoscopic Depth Cues (two 2D images)– two images of a scene at the same time from

different positions

• Motion Depth Cues (series of 2D images)

As the human retina is basically just a flat screen on the inside of the eye it can only receive information in 2 dimensions these being height (y) and width (x).

Vision System

Anatomy of an eye…Light travels into your eyes and is focused by the lens onto

your retinas.

The retina is the layer of tissue at the back of each eye that converts the light energy that enters our eyes into neural impulses.

The brain processes these neural impulses and the pattern of neural activity that results is what we mean when we say we see something.

What do we see?

As you can see in the figure at below, the image that is cast onto your retinas is upside-down and backwards.

WHY?

This is because the lens and cornea of the eye gather light about the rays that travel in straight lines—the straight light rays from the top of the tree are cast onto the bottom of your retina, those from the bottom of the tree are cast onto the top of your retina, those from the left go to the right of the retina, and those from the right go to the left of the retina.

....then how do we see 3D?

• Human experience however tells us that the world is 3 dimensional.

• We experience both distance and depth in our personal environment.

• So where does this 3rd dimension comes from? • Answer: Our perceptual processes create it by

interpreting information from two types of depth cues. These are binocular and monocular depth cues.

• To perceive binocular depth cues = must have two eyes

• To perceive monocular cues = only one eye is needed

What is 3D?

Right Handed Coordinate System

+X

+Y

+Z

+X

+Y

+Z

Viewing a 3D world

We have a model in this world and would like to view it from a new position.

We’ll call this new position the camera or eyepoint. Our job is to figure out what the model looks like on the display plane.

+X

+Y

+Z

Parallel Projection

+X

+Y

+Z

Perspective Projection

+X

+Y

+Z

3D Vision and Displays

• Traditional graphics and Desktop VR system display 3D environments as only a 2D projection.

• However – most of us see in 3D in the real world.

• If virtual environments are to allow us to interact using natural skills and if the feeling of immersion or presence is to be maximized, then maybe we should be able to use our full range of natural visual perceptual abilities.

Cont.

• There are ways of implementing stereoscopic (3D) displays which include HMD’s and desktop systems.

• You can also improve the depth cues in 2D displays which can make some tasks (object selection, for example) easier, and make the virtual environment richer.

Binocular cues Monocular cues

• innate (natural)

• biological

• learned

• environmental

Although both types of cue are presented separately in this resource, in reality they all work together

There are two major binocular depth cues these are convergence and retinal disparity

ConvergenceThis acts like a rangefinder to tell us how far away an object is

If an object is distant our eyes look straight ahead however as the object gets nearer our eyes swivel inwards (converge) to keep the object in focus

The brain measures the amount of convergence to provide us with a rough estimate of distance

Binocular Depth Cues

Retinal disparity

Human eyes are spaced apart, separated by the nose. The average distance of this spacing is about 2 ½ (two-and-a-half) inches or 6 cm apart .

This means that the brain receives two slightly different pictures of the same scene

Look at the picture below, it is the equivalent of visual input to the brain from both eyes. If you look carefully you will notice differences. This is due to the distance between the 2 eyes viewing the scene from a slightly different angle.

The brain takes both images puts them together and uses the slight differences between them to create a perception of depth.

This is exactly how Victorian stereoscopes worked, two similar images of the same scene are presented one to each eye. The brain merges both images and the differences create an impression of depth

There are many monocular depth cues, these include:-

Monocular Depth Cues

• Superimposition/ Interposition– An object that occludes another is closer

• Relative size– Usually, the larger object is closer

• Height in the horizontal plane– The higher the object is (vertically), the further it is

• Linear perspective– parallel lines converge at a single point

• Texture gradient– more detail for closer objects

• Atmospheric effects– further away objects are blurrier

• Brightness– further away objects are dimmer

• Motion parallax– Distant object pass by slowly, closer object pass by more quickly

• Superimposition / Interposition or overlap is said to obscures when one object obscure a more distant one.

• The object that is blocking the view of the other object is closer.

Superimposition / interposition or overlap

Superimposition / interposition or overlap

We assume the tree is in front of the house because it obscures our view of it. It is superimposed on the front of the house, it overlaps it.

Likewise we know the tutor is nearer to us than the blackboard because she obscures (overlaps) a portion of it

Relative size

• Relative size refers to the comparison of retinal image sizes of identical or nearly identical objects.

Relative size

When we view 2 objects we know to be the same size and one appears smaller than the other we assume that the smallest is further away

We don’t interpret this picture as one large tank followed by smaller vehicles. We know all the tanks are the same size thus the smaller ones must be further away

Height in the horizontal planeIn this scene the furthest objects, the trees, are also the highest. We know the boats are nearer to us than the trees because they are lower down

The same effect works in real life as well as paintings

Linear perspective

Linear perspective is seemingly obvious monocular depth cue in that lines that are parallel in a 3D world converge at an infinite distance, in prespective, on a vanishing point.

This is a familiar views of rail road tracks, lonely dessert highways etc.

In fact, the basics of linear perspective are attributed to the painter Brunelleschi in about 1420 and were put to famous first use by the Italian Renaissance artists such as Leonardo da Vinci.

Linear perspective

Parallel lines converge in the distance so the further away the lines are from the observer the closer together they will appear

This fact was discovered and used by renaissance artists to great effect

artistic examples include this

and this

Linear perspective also occurs in the real world

Railroad tracks, lonely dessert highway, sky scrapers etc.

Texture gradient

Here the grain or texture appears to become finer as distance increases

Texture gradient can be used to determine the way that a surface with a uniform surface is receding away from the viewer.

The boulders in this Martian landscape demonstrate this, notice how they get smaller in

the distance

Atmospheric effects and brightness

• Look at the scene…

• Atmospheric Effects– further away objects are blurrier– Very far objects can also

appear bluer in color (think of mountains) because of the way that light is scattered (aerial depth cues)

• Atmospheric Brightness– further away objects are

dimmer

Relative BrightnessHighlights and shadows can provide information about an object's dimensions and depth. Because our visual system assumes the light comes from above, a totally different perception is obtained if the image is viewed upside down.

Kinetic Monocular Depth Cues

• In the real world, information about object depth is also available from the way their images move on the retina. This is called depth from motion.

• Motion cues are ones that can be taken advantage of by comparing images from a sequence of “frames” presented to a single eye; depth is inferred from the types of motion evident in the image.

Motion Parallax

• Imagine you are sitting in a moving train and looking out the window. The trees that are close to the road appear to zip by, but the mountains that are off in the distance move much more slowly.

• The inverse relation between motion and distance is called MOTION PARALLAX.

• It is clear that the amount of motion parallax is an excellent cue to depth: lots of motion means close, little motion means far away!

Motion Parallax

• Motion parallax might also be the process we are adopting when we move our heads from side to side to judge the relative proximity of 2 objects – however, we might also be using stereopsis when doing this act.

Motion parallax

When an observer passes through a scene it moves through the observers field of vision at different speeds

Distant objects appear to pass by slowly

Near objects appear to pass by more quickly

Remember that these cues can be used on an individual basis, they are however most likely to be used in conjunction with each

other. Look at the following pictures and see how many monocular cues you can identify

Relative size

Superimposition

Linear perspective

Texture gradient

Linear perspective

Height in the horizontal plane

Relative size (of the vehicles)

Superimposition

Seeing is believing

Binocular Depth Cues

• Visual Binocular cues are available through comparison of the (different) images seen by the 2 eyes.

• 3 types of Binocular Depth Cues1. Stereopsis 2. Accommodation 3. Convergence

Disparity• Disparity is a binocular cue to depth.• Disparity gives us stereopsis (from Greek

meaning “solid sight”) • Disparity is a geometric fact – resulting

from the fact that both eyes get slightly different views of the world. It is a very strong cue to depth.

• It arises from the fact that our two eyes have a slight differences in the left and right retina images to infer depth perception.

Binocular Depth Cues

1. Stereopsis • Stereopsis is the mental ability to create 3D

image from 2D or flat stereo images. • It comes from two Greek roots, stereo meaning

solidity, and opsis meaning vision or sight. • Stereopsis provides 3D object that has width (x

axis), height (y axis) and depth (z axis). • The ability to provide depth makes stereopsis as

an added value to the ability of the mind to see.

Disparity (Stereopsis)

Stereopsis

• Stereopsis can help resolve the spatial ambiguities such as the Necker Cube:-

• Focus on the red dot!

Stereoscopic Vision

• Probably evolved as a means of survival.• With stereo vision, we can see WHERE objects

are in relation to our own bodies with much greater precision--especially when those objects are moving toward or away from us in the depth dimension.

• We can see a little bit around solid objects without moving our heads and we can even perceive and measure "empty" space with our eyes and brains.

Stereoscopic Vision

• In fact there is a great individual difference in stereovision abilities.

• Some people have no trouble fusing stereo images, others can fuse the images with some practice, and still others can never see depth from the images no matter how long they practice.

• Such observers (5-10% of humans) are STEREOBLIND!

STEREO BLIND

• Amblyopia (lazy eye)?Amblyopia, commonly known as lazy eye, is the eye condition noted by reduced vision not correctable by glasses or contact lenses and is not due to any eye disease.

• The brain, for some reason, does not fully acknowledge the images seen by the amblyopic eye.

• This almost always affects only one eye but may manifest with reduction of vision in both eyes.

• It is estimated that three percent of children under six have some form of amblyopia.

Treatment of Amblyopia

• If not detected and treated early in life, amblyopia can cause a permanent loss of vision with associated loss of stereopsis (two eyed depth perception).

• Detection and correction before the age of 2 offers the best chance for restoration of normal vision. However, treatment can improve this condition even in adulthood.

• Amblyopia can be treated fairly successfully between the ages of 2 and 6, but the success decreases with age. (best results from treatment occurs between ages 6 months - 2 years).

• Treatment is usually simple, employing glasses, drops, exercises and/or patching.

• Though true amblyopia can not be cured (after the age of 6) treatment for the older child is usually successful in improving vision and should be attempted.

• Treatment of amblyopia after the age of 6 is not dependent upon age but requires more effort including vision therapy.

If You've Got Stereo Vision, Count Your Blessings!

• According to the web site of the American Academy of Opthalmology, September, 1996: "many occupations are not open to people who have good vision in one eye only [that means people without stereo vision]"

Stereoscopic Vision

• Here are a few examples of occupations that depend heavily on stereo vision:

• Baseball player • Waitress • Driver • Architect • Surgeon • Dentist

Stereoscopic Vision

Here are just a few examples of general actions that depend heavily on stereo vision: • Throwing, catching or hitting a ball • Driving and parking a car • Planning and building a three-dimensional object • Threading a needle and sewing • Reaching out to shake someone's hand • Pouring into a container • Stepping off a curb or step

Binocular Depth Cues

2. Accommodation - When we try to focus on far away objects, the ciliary muscles stretches the eye lens, making it thinner. The kinesthetic sensations of the contracting and relaxing ciliary muscles (intraocular muscles) is sent to the visual cortex where it is used for interpreting distance/depth.

Binocular Depth Cues

3. Convergence - By virtue of stereopsis the two eye balls focus on the same object. In doing so they converge. The convergence will stretch the extraocular muscles. Kinesthetic sensations from these extraocular muscles also help in depth/distance perception. The angle of convergence is larger when the eye is fixating on far away objects.

Let’s do an Experiment (step 1 of 3)

- Center your nose over the brown eye above. - Focus your eyes on the single brown eye. - Put your free thumb in front of your nose. - Continue to focus on the eye.. (you can use any other objects eg. Pen)

TRY IT!!

If both eyes are on, you will see two thumbs framing one eye.

Now, switch your focus to your thumb.

You should see two eyes framing one thumb

What about VE’?• What is the point of all these?

– The idea is that your VEs make use of as many depth cues as possible in order to maximise the feelings of presence.

– Most depth cues are provided by runtime 3D graphic system – hence you don’t normally have to worry about shading, shadowing, relative size, perspective, texture and so on.

– Most VE systems allow you to alter atmospheric effects as well (for eg. Use blue fog in VRML) and also monitor things like collision detection.

– At the end of all these, you may still have a VE that appears flat – if you wish to create a greater sense of depth then maybe you ought to consider a stereo display.

Stereo display (visual aid)

• Stereoscopy, stereoscopic imaging or 3-D (three-dimensional) imaging is any technique capable of recording three-dimensional visual information or creating the illusion of depth in an image.

• The illusion of depth in a photograph, movie, or other two-dimensional image is created by presenting a slightly different image to each eye.

• Many 3D displays use this method to convey / show images.

Stereo Display devices

• Head-mounted displays

• 3D glasses

• LCD shutter glass

Interesting phenomena!

• Wiggle stereoscopy

Interesting phenomena!• This method, possibly the most simple sterogram

viewing technique, is to simply alternate between the left and right images of a stereogram.

• Most people can get a crude sense of dimensionality from such images, due to persistence of vision and parallax. Closing one eye and moving the head from side-to-side helps to understand why this works. Objects that are closer appear to move more than those further away.

• This effect may also be observed by a passenger in a vehicle or low-flying aircraft, where distant hills or tall buildings appear in three-dimensional relief, a view not seen by a static observer as the distance is beyond the range of effective binocular vision.

Interesting phenomena!

• Advantages of the wiggle viewing method include:

• No glasses or special hardware required • Most people can "get" the effect much quicker

than cross-eyed and parallel viewing techniques • It is the only method of stereoscopic

visualisation for people with limited or no vision in one eye

Interesting phenomena!• Disadvantages of the "wiggle" method:

• Does not provide true binocular stereoscopic depth perception • Not suitable for print, limited to displays that can "wiggle" between

the two images • Difficult to appreciate details in images that are constantly "wiggling" • Most wiggle images use only two images, leading to an annoyingly

jerky image. A smoother image, more akin to a motion picture image where the camera is moved back and forth, can be composed by using several intermediate images (perhaps with synthetic motion blur) and longer image residency at the end images to allow inspection of details.

Interesting phenomena!

• Although the "wiggle" method is an excellent way of previewing stereoscopic images, it cannot actually be considered a true three-dimensional stereoscopic format. An individual looking at a wiggling image is not at all experiencing stereoscopic viewing, they are still only seeing a flat two-dimensional image that is "wiggling".

• To experience binocular depth perception as made possible with true stereoscopic formats, each eyeball must be presented with a different image at the same time - this is not the case with "wiggling" stereo. The "wiggle" effect is similar to walking around one's environment while blinking one eyes.