Projection-Based Augmented Reality
... in Engineering Applications
André Stork / Oliver Bimber / R. de AmicisFraunhofer-IGD, DarmstadtFraunhofer CRCG, ProvidenceGRIS, TU Darmstadt
The Magic MirrorM.C. Escher, 1946
PBAR – Projection-based Augmented Reality
2
André Stork, Oliver Bimber,R. de Amicis
Overview
• Motivation• Introduction• The Extended Virtual Table• Engineering applications• Conclusions and future work
Still Life with Mirror M.C. Escher, 1934
PBAR – Projection-based Augmented Reality
3
André Stork, Oliver Bimber,R. de Amicis
Motivation
projection-based (SSP: CAVEs, domes, etc. / workbenches, walls, etc.)
projection-based (SSP: CAVEs, domes, etc. / workbenches, walls, etc.)
HMDs (opaque)
screen-based(fish-tank)
VR ...
screen-based(video-mixing)HMDs (see-through or video-mixing)
... AR
Mixed Reality (MR)
Augmented Reality (AR)
Real Environment
Augmented Virtuality (AV)
Virtual Environment
Virtuality Continuum (VC)PaulMilgram
PBAR – Projection-based Augmented Reality
4
André Stork, Oliver Bimber,R. de Amicis
MotivationWhy are projection-based VR systems attractive?
• scalable· resolution· FOV/deg. of
immersion • integratable into
real environment
• light weight/no glasses
• focal length• reduced motion-
sickness (reduced swimming effects)
technological reasons ergonomic reasons economic reasons
• well established and applied
• wide variety of systems on market
• application-specific functionality (not all-purpose)
PBAR – Projection-based Augmented Reality
5
André Stork, Oliver Bimber,R. de Amicis
MotivationWhat could be the benefits of a projection-based AR approach?
Adoption of the technological, ergonomic and economic advantages of well-established projection-based VR systems
· technology· concepts· applications· experience· etc.
Possibility of a VR/AR combinationHigher acceptance Hand with Reflecting Sphere
M.C. Escher, 1935
PBAR – Projection-based Augmented Reality
6
André Stork, Oliver Bimber,R. de Amicis
MotivationWhat are the main limitations of projection-based VR systems ?• Front/rear-projection systems cause occlusion problems• Semi-immersive systems are constrained to a limited
viewing space (window violation)
What would be the main limitations of projection-based AR systems ?• Mobility (devices are usually stationary)• Cost (usually higher)• Multi-user (support a limited number of users
simultaneously)• Not all-purpose (have application-specific functionality)• Interactivity
PBAR – Projection-based Augmented Reality
7
André Stork, Oliver Bimber,R. de Amicis
Introduction
What is our approach ?
• Integration of mirror-beam splitters (half-silvered mirrors) into today’s projection-based VR systems
• Mirror-beam splitters with:· different geometry (e.g. planar, convex, concave)· different configurations (e.g. single or multiple inter-playing
mirrors)• Projection system
· non-immersive, semi-immersive, immersive• View-dependent off-axis rendering (single/multiple
viewer)
PBAR – Projection-based Augmented Reality
8
André Stork, Oliver Bimber,R. de Amicis
Introduction
What are the challenges ?
Still Life with Reflecting SphereM.C. Escher, 1934
• Real-time stereoscopic rendering · mirror/view dependent scene
transformation (affine/non-affine)· correct illumination· correct depth-handling
• Distortion correction· optical (e.g. projector-geometry,
refraction, mirror flexion, curved optics)· non-optical (e.g. tracking)
• Appropriate interaction
PBAR – Projection-based Augmented Reality
9
André Stork, Oliver Bimber,R. de Amicis
The Extended Virtual Table
40”x40”x10mm half mirror (float-glass)Angle between projection plane and mirror can be changed by rotating the mirror
Physical Arrangement
PBAR – Projection-based Augmented Reality
10
André Stork, Oliver Bimber,R. de Amicis
Working principleThe Extended Virtual Table
PBAR – Projection-based Augmented Reality
11
André Stork, Oliver Bimber,R. de Amicis
The Extended Virtual Table
VR Mode
Gaze-directed (non-simultaneous) visual extension of virtual environments
Projection PlaneMirror
PBAR – Projection-based Augmented Reality
12
André Stork, Oliver Bimber,R. de Amicis
The Extended Virtual Table
AR Mode
Virtual-real object occlusion through phantoms (Breen et al., 1996)Ratio of intensity of the transmitted light and the reflected light depends on angle between mirror and projection planeANDon the lighting situation
60°
80°no clipping plane
PBAR – Projection-based Augmented Reality
13
André Stork, Oliver Bimber,R. de Amicis
Use of VR and AR within the PLC
VR as key factor for industry and key technology to remain competitive.
VR replaces expensive physical prototypes through its digital counterpart within the product development process
AR small brother of VR ( yet !) First prototypes have been
developed in the construction, training and maintenance area
Augmented Prototyping as new field of application
ProductDefinition Design Analysis
Produc-tionPlanning
Produc-tion Use
Mainte-nance Recycling
Product life cycle
Engineering applications
PBAR – Projection-based Augmented Reality
14
André Stork, Oliver Bimber,R. de Amicis
Engineering applications
New kinds of applications:· Hybrid Assembling · Augmented Design Review· Hybrid Modeling and sketching · Visual Inspections of modeled parts· etc., e.g. hybrid ergonomic design PMU
VMU
PBAR – Projection-based Augmented Reality
15
André Stork, Oliver Bimber,R. de Amicis
Hybrid assembling
Manipulating the distantobject ‚behind‘ the mirror Occlusion of the virtual part
through the real part
Direct manipulation above the VT
Move the virtual object‚through‘ the mirror
Engineering applications
PBAR – Projection-based Augmented Reality
16
André Stork, Oliver Bimber,R. de Amicis
Visual inspection of molded partsEngineering applications
PBAR – Projection-based Augmented Reality
17
André Stork, Oliver Bimber,R. de Amicis
Augmented PrototypingVR :• Visualisation and interaction• BUT limited haptic feedback
Rapid Prototyping (RP) :• Fast physical prototype• good haptic feedback • BUT only one finish at a time
Augmented Prototyping
Engineering applications
PBAR – Projection-based Augmented Reality
18
André Stork, Oliver Bimber,R. de Amicis
Sketching free-form surfaces in Mixed Realities
Engineering applications
PBAR – Projection-based Augmented Reality
19
André Stork, Oliver Bimber,R. de Amicis
Engineering applications
PMU
VMU
Hybrid Mock-Up(HMU)
Hybrid assembling
Direct interaction with physical mock-up throughreach-in set-up
PBAR – Projection-based Augmented Reality
20
André Stork, Oliver Bimber,R. de Amicis
Ceiling projectionTwo beamers Project pre-filtered stereo images simultaneously Passive shuttering using polarized light
Alternative set-upsThe Extended Virtual Table
PBAR – Projection-based Augmented Reality
21
André Stork, Oliver Bimber,R. de Amicis
Augmented Design ReviewThe Virtual Showcase:Conic half-silver mirror‚walk-arounds‘
Engineering applications
PBAR – Projection-based Augmented Reality
22
André Stork, Oliver Bimber,R. de Amicis
Multi-user AR-Setup
Half-silver mirror• On a projection plane several views are
presented perspectively correct• A real object inside the mirror prism can be
augmented
Future steps
PBAR – Projection-based Augmented Reality
23
André Stork, Oliver Bimber,R. de Amicis
Conclusions
VR
AR
ProductDefinition Design Analysis
Produc-tionPlanning
Produc-tion Use
Mainte-nance Recycling
Product life cycle
PBAR – Projection-based Augmented Reality
24
André Stork, Oliver Bimber,R. de Amicis
IGD-Darmstadt
Univ. of Technology-Darmstadt ZGDV-Darmstadt CAPCom business & solutions -
Darmstadt IGD-Frankfurt Univ. of Frankfurt
IGD-Rostock
Univ. of Rostock
ZGDV-Rostock
Ctr. for Research in Computer Graphics-Providence
Brown Univ.- Providence
Rhode Island School of Design- Providence
Univ. do Minho- Azurem Centro de Computacao Grafica Univ. of Technology-Nayang Ctr. for Advanced Media Technology- Nayang
Univ. of Technology-Vienna
Thank you !