Estimating Scale using Depth From Focus for Mobile Augmented
Reality
The Use of Surrounding Visual Context in Handheld AR: Device vs.
User Perspective RenderingKlen opi PuciharDr. Paul CoultonDr. Jason
Alexander
Hello everybody, my name is Klen and I come form Lancaster
University. Today I will talk to you about one of the key issues
that affect handheld AR.
In the next 15 minutes we will dwell on the journey of
surrounding context use and try to uncover how is surrounding
context affected by the dual-view problem.
Lets start by looking at what is the dual-view problem.1
Handheld AR is typically implemented using the magic lens
paradigm [6], where the device acts as a transparent glass pane
showing a digitally enhanced view of the scene laying behind the
pane.
Unfortunately. As you may have noticed by now, devices such as
mobile phones are not transparent, hence virtual transparency is
required.
This virtual transparency is typically implemented by utilizing
a back facing camera and a device screen. Because the device camera
is not in the center of the screen and the device is held in the
hand and not worn on user head such setup results in perspective
difference between the observer and the device camera. This results
in a mismatch between the phone and its surroundings known as the
dual-view problem.
Lets look at the video which demonstrates the dual-view problem.
2
Device-perspective
________________User-perspective
The video I plan to show now demonstrates two magic lens types.
The first magic-lens type is device-perspective magic-lens. This
implementation renders camera captured images onto device screen
without any modifications. This results in a mismatch between phone
and the surroundings also known as dual-view problem. It is
important to note that this lens type is used in majority of AR
implementations.
Alternative to device-perspective rendering is user-perspective
rendering. User-perspective lenses modify the camera imagery in
order to try and match the perspective seen by the observer. This
results in exact match of phone and surroundings. The
implementation you see now is a simplified variation of
user-perspective magic-lens we call fixed-POV user-perspective
magic-lens. We used this implementation in two user studies I will
talk about later on in this presentation.
From the video it is obvious that the misalignment and incorrect
scale, also known as dual-view problem, might have an effect on the
use of the surrounding visual context. Therefore we hypothesis that
within the class of tabletop size AR workspaces, the dual-view
problem impedes users ability to:
3
Cross-context
Cross-context interaction is defined as a task that requires
crossing the border between the magic-lens and the
surroundings.
Complete tasks that require crossing border between the lens and
the surroundings.
We call this cross-context interaction which is important when
the users hand interacts within the AR workspace and crosses the
boundaries between real and augmented views. Such is the case
in:1.) AR supported user sketching where user transcribing digital
instructions/routes to paper or maps, 2.) Another such example is
when user brings tangible objects into and out of the augmented
view, e.g. AR chess with physical pieces4
View-merge
View-merge interaction is defined as tasks that requires merging
magic-lens view with wider surroundings.
Beside affecting the cross-context interaction we also
hypothesis that the dual-view problem impedes users ability to
efficiently merge the magic lens view with the wider
surroundings;
We call this view-merge interaction which is particularly
important in scenarios with high information density such as AR on
printed media. One example of these are 1.) digitally enhanced text
documents, such as music score enhanced with audio. 2.) navigation
and route planning on augmented paper maps, 3.) augmented magazine
games such as crossword puzzle hints.
These two interaction patterns highlight the importance of the
use of the surrounding context. We believe that improving the
surrounding context will facilitate: 1.) more fluid interaction as
less time will be wasted interpreting different views, 2.) allow
easier use of potentially valuable surrounding information, 3.)
provide greater understanding of information context, and aid the
construction and retention of a mental model of the information
space.
To evaluate the effect of dual-view problem on each interaction
pattern we designed and run two user studies.
5
Study A: Cross-context
15 Participants;2x2 within-subjects design with independent
variables size and type.
The first study evaluated the Cross context interaction on drag
and drop action, where a mouse controlled virtual hand was used to
to drag a ball into the basket.
SLIDE
1.) 15 Participants;2.) 2x2 within-subjects design with
independent variable size and type.
6
Klen Copic Pucihar () - Build animatio of how the process
works,
Study B: View-merge
Experiment Design A = Experiment Design BOnly one independent
variable lens type is tested.
The second user study evaluated the View-merge interaction on a
map navigation task. Here participants were asked to perform a path
following task in which they counted the number of corners from
start to finish.
While navigating users had to:1.) avoid closed roads, 2.) follow
three instruction arrows. The visibility of instruction arrows
alternated from phone to screen and back to phone.
Let me note here that the most difficult part of this task is
expected to be at the beginning, from start to first instruction
arrow, in the above case till corner 3. This is because at this
point users saw the map for the first time. After they count a few
corners they become more familiar with the map so the task is
expected to become easier.
SLIDE
The experiment design is the same as in Study A. The only
difference is that here we test only one independent variables lens
type.
Lets have a quick look at quantitative results.
7
Klen Copic Pucihar () - Build animatio of how the process
works,
Quantitative Results:Study A:A significant effect of lens type
on task time and path deviation was only detected for tablet-sized
lenses;A strong interaction between lens type and size;Limited
effect of lens size;Study B:A significant effect of lens type only
for startup-time.What is causing this interaction?
Results of cross-context interaction showed:1.) A significant
effect of lens type on task time and path deviation was detected.
However, only in case of tablet-sized lenses; In case of
device-perspective lens participants required more time and
deviated significantly more from optimal path,2.) A strong
interaction between lens type and size;3.) Limited effect of lens
size;
In case of view merge interactions. Results show a significant
effect of lens type only for startup-time when view-merging is
expected to be most difficult.
The most interesting part of our results is the fact that when
we set up the experiment we considered lens type and size are
independent variables. However, the strong interaction between lens
type and size suggests these two variables are inter-related. But
what is causing this interaction?
8
Klen Copic Pucihar () - Build animatio of how the process
works,
Magnification might be causing the strong interaction between
lens type and size!?Magnifying lensMinimizing lens
By analyzing task completion strategies, observational and
question results we discovered that users generally dislike
magnification because :1.) Magnifying magic lenses occlude a larger
portion of the observers FOV than that rendered on the screen,
creating information blind spots; Demonstrated on left image as
missing words from the text. This is not the case on minimizing
lenses on your right.2.) Lenses with larger magnification show a
smaller fragment of the real-world scene thus reducing the number
of landmarks.
This raised our suspicion that magnification might be causing
the strong interaction between lens type and size. To confirm our
hypothesis, we derived a magnification model for user and
device-perspective lenses.9
Magnification model
Applying a fisheye lens to the tablet would eliminate the
magnification, reducing the effect of magic lens type.
MINIMIZATIONMAGNIFICATION
The magnification model (Figure on left) is derived from a
theoretical analysis of: (1) Ideal transparency and (2)
Camera-generated virtual transparency.
To analyze how magnification changes with magic lens type and
size, we plan to plot the minimization curves for 4
magic-lenses.
To achieve user-perspective rendering, the camera- generated
transparency is adjusted to match ideal transparency. This results
in realistic rendering with no magnification (we=wc, MU=we/wc=1).
So phone and table sized user perspective lens are represented as a
horizontal line at 1. On the graph magnification is area bellow 1,
and minimization is area above 1.
For device-perspective rendering, the camera-generated
transparency is not adjusted. This results in varying magnification
defined by magnification model shown on the left.
When we plot phone sized lenses it is clear that for most
observer and magic lens distances such lens minimizes. Therefore
device-perspective phone sized lens has an advantage over
user-perspective lens. The opposite is true in case of tablet sized
lenses. The device-perspective lens (blue plot) always magnifies
the scene. As a result user-perspective lens has the magnification
advantage.
This confirms that magnification is responsible for the
interaction between lens size and type and allows us to conclude
that:
The strong significant effect of magic lens type in large
lenses, is not a result of increased dual-view effect (a
consequence of magic lens size), but rather the result of a
cumulative effect of the dual- view problem and magnification (see
table on Figure 5d). Consequently, applying a fisheye lens to the
tablet-sized device-perspective magic lens would eliminate the
magnification, reducing the effect of magic lens type. 10
Klen Copic Pucihar () - Add magnify/ minimiza coloringKlen Copic
Pucihar () - Change advantage for comparrison parameter.
remove the spatial advantage rows.Klen Copic Pucihar () - Create
an animation where firs hrizontal line then red and then blue plots
come visible on the graph.Klen Copic Pucihar () - Change pic to
only show the final equarion.Klen Copic Pucihar () - Make this
smaller so text can go higher if time.
The most appropriate rendering type is the one where both
advantages coexist; The most appropriate rendering type is the one
where both advantages coexist; When these advantages do not
coexist, the designer needs to decide which property is more
important. The better spatial awareness or magnification
advantage.
In order to help future developer to select the optimal
magic-lens type we formulate these findings into a design
guidelines.
To Magnification parameter, we add Dual-view parameter
1.) Previous studies showed that dual-view problem affects users
spatial awareness, therefore not being affected by dual view
problem, as is the case in user-perspective lenses, is bound to be
an advantage. This fact is summarized by the second and forth row
of the table.
Now that the table is generated, it may be interpreted by using
the following two rules:1.) The most appropriate rendering type is
the one where both advantages coexist; Reflecting on our study,
this was true in case of user-perspective tablet size lens.
2.) When these advantages do not coexist, the designer needs to
decide which property is more important. The better spatial
awareness, or magnification advantage. Reflecting to our study,
during the startup time of path following task, user perspective
rendering should be chosen, however, in later stages switch to
device-perspective rendering should be considered.
I think my time is running out so let me wrap this presentation
up by showing you this slide.
11
The message I am trying to convey with this slide is known as
the most fundamental characteristic of AR which in essence strives
to bring virtual and real world closer together.
I believe that the work presented to you today makes a step into
the right direction.
12
Demo on Wednesday 15:20 - 16:00
The strong effect of magic lens type in large lenses, is not a
result of increased dual-view effect (a consequence of magic lens
size), but rather the result of a cumulative effect of the
dual-view problem and magnification.
But because jumping from the cliff edge can not be compared with
a video of this action, I invite you all to join my demo both on
Wednesday afternoon.
You will have a chance to experience the effect of dual-view
problem on the surrounding context in real world AR applications I
prepared for you.1.) One will let you interact with music by
playing section of the score and the other will help you draw your
favorite cartoon character on the wall. Hope to see you all on
Wednesday.
Thank you for your time.
Questions13
