Projection based augmented reality

Projection-Based Augmented Reality

... in Engineering Applications

André Stork / Oliver Bimber / R. de AmicisFraunhofer-IGD, DarmstadtFraunhofer CRCG, ProvidenceGRIS, TU Darmstadt

The Magic MirrorM.C. Escher, 1946

PBAR – Projection-based Augmented Reality

2

André Stork, Oliver Bimber,R. de Amicis

Overview

• Motivation• Introduction• The Extended Virtual Table• Engineering applications• Conclusions and future work

Still Life with Mirror M.C. Escher, 1934


3


Motivation

projection-based (SSP: CAVEs, domes, etc. / workbenches, walls, etc.)

projection-based (SSP: CAVEs, domes, etc. / workbenches, walls, etc.)

HMDs (opaque)

screen-based(fish-tank)

VR ...

screen-based(video-mixing)HMDs (see-through or video-mixing)

... AR

Mixed Reality (MR)

Augmented Reality (AR)

Real Environment

Augmented Virtuality (AV)

Virtual Environment

Virtuality Continuum (VC)PaulMilgram


4


MotivationWhy are projection-based VR systems attractive?

• scalable· resolution· FOV/deg. of

immersion • integratable into

real environment

• light weight/no glasses

• focal length• reduced motion-

sickness (reduced swimming effects)

technological reasons ergonomic reasons economic reasons

• well established and applied

• wide variety of systems on market

• application-specific functionality (not all-purpose)


5


MotivationWhat could be the benefits of a projection-based AR approach?

Adoption of the technological, ergonomic and economic advantages of well-established projection-based VR systems

· technology· concepts· applications· experience· etc.

Possibility of a VR/AR combinationHigher acceptance Hand with Reflecting Sphere

M.C. Escher, 1935


6


MotivationWhat are the main limitations of projection-based VR systems ?• Front/rear-projection systems cause occlusion problems• Semi-immersive systems are constrained to a limited

viewing space (window violation)

What would be the main limitations of projection-based AR systems ?• Mobility (devices are usually stationary)• Cost (usually higher)• Multi-user (support a limited number of users

simultaneously)• Not all-purpose (have application-specific functionality)• Interactivity


7


Introduction

What is our approach ?

• Integration of mirror-beam splitters (half-silvered mirrors) into today’s projection-based VR systems

• Mirror-beam splitters with:· different geometry (e.g. planar, convex, concave)· different configurations (e.g. single or multiple inter-playing

mirrors)• Projection system

· non-immersive, semi-immersive, immersive• View-dependent off-axis rendering (single/multiple

viewer)


8


Introduction

What are the challenges ?

Still Life with Reflecting SphereM.C. Escher, 1934

• Real-time stereoscopic rendering · mirror/view dependent scene

transformation (affine/non-affine)· correct illumination· correct depth-handling

• Distortion correction· optical (e.g. projector-geometry,

refraction, mirror flexion, curved optics)· non-optical (e.g. tracking)

• Appropriate interaction


9


The Extended Virtual Table

40”x40”x10mm half mirror (float-glass)Angle between projection plane and mirror can be changed by rotating the mirror

Physical Arrangement


10


Working principleThe Extended Virtual Table


11



VR Mode

Gaze-directed (non-simultaneous) visual extension of virtual environments

Projection PlaneMirror


12



AR Mode

Virtual-real object occlusion through phantoms (Breen et al., 1996)Ratio of intensity of the transmitted light and the reflected light depends on angle between mirror and projection planeANDon the lighting situation

60°

80°no clipping plane


13


Use of VR and AR within the PLC

VR as key factor for industry and key technology to remain competitive.

VR replaces expensive physical prototypes through its digital counterpart within the product development process

AR small brother of VR ( yet !) First prototypes have been

developed in the construction, training and maintenance area

Augmented Prototyping as new field of application

ProductDefinition Design Analysis

Produc-tionPlanning

Produc-tion Use

Mainte-nance Recycling

Product life cycle

Engineering applications


14



New kinds of applications:· Hybrid Assembling · Augmented Design Review· Hybrid Modeling and sketching · Visual Inspections of modeled parts· etc., e.g. hybrid ergonomic design PMU

VMU


15


Hybrid assembling

Manipulating the distantobject ‚behind‘ the mirror Occlusion of the virtual part

through the real part

Direct manipulation above the VT

Move the virtual object‚through‘ the mirror



16


Visual inspection of molded partsEngineering applications


17


Augmented PrototypingVR :• Visualisation and interaction• BUT limited haptic feedback

Rapid Prototyping (RP) :• Fast physical prototype• good haptic feedback • BUT only one finish at a time

Augmented Prototyping



18


Sketching free-form surfaces in Mixed Realities



19



PMU

VMU

Hybrid Mock-Up(HMU)

Hybrid assembling

Direct interaction with physical mock-up throughreach-in set-up


20


Ceiling projectionTwo beamers Project pre-filtered stereo images simultaneously Passive shuttering using polarized light

Alternative set-upsThe Extended Virtual Table


21


Augmented Design ReviewThe Virtual Showcase:Conic half-silver mirror‚walk-arounds‘



22


Multi-user AR-Setup

Half-silver mirror• On a projection plane several views are

presented perspectively correct• A real object inside the mirror prism can be

augmented

Future steps


23


Conclusions

VR

AR

ProductDefinition Design Analysis

Produc-tionPlanning

Produc-tion Use

Mainte-nance Recycling

Product life cycle


24


IGD-Darmstadt

Univ. of Technology-Darmstadt ZGDV-Darmstadt CAPCom business & solutions -

Darmstadt IGD-Frankfurt Univ. of Frankfurt

IGD-Rostock

Univ. of Rostock

ZGDV-Rostock

Ctr. for Research in Computer Graphics-Providence

Brown Univ.- Providence

Rhode Island School of Design- Providence

Univ. do Minho- Azurem Centro de Computacao Grafica Univ. of Technology-Nayang Ctr. for Advanced Media Technology- Nayang

Univ. of Technology-Vienna

Thank you !

Software

Projection based augmented reality