Augmented and Mixed Reality - oamk.fi · Project “AR4BC” Augmented Reality for Building and Construction ... Augmented Reality + Augmented Virtuality ... CATIA -> STEP -> Reduced

1

Augmented and Mixed Reality

Applications at VTT

Charles Woodward

April 16, 2010

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Mobile Augmented Reality

one of the technologies to

“most likely to alter industries, fields of research,and the way we live”.

- MIT’s annual review 10 Emerging Technologies 2007

- Gartner’s Top Ten Disruptive Technologies 2008-2012

- First consumer applications on camera phones 2009

- Commercial breakthrough predicted 2010

2


INTRODUCTION


VTTINDUSTRY

UNIVERSITIESBasicresearch

Appliedresearch

Developmentwork

VTT Technical Research Centre of Finland

Largest contract researchinstute in Northern Europe

Producing research,development, testing andinformation servicesServing public sector andcompanies as well asinternational organizations

Figures2700 employees245 M€ turnoverFunding1/3 government,1/3 industry, 1/3 jointly funded

3


Virtual and Augmented Reality at VTT

General VR/AR expertise around VTT14 research teams in VR/AR roadmap workApplications from games to digital media,construction to production, ships, cars, military, …Technology incl. core VR/AR solutions, sensors,GSM/GPS positioning, multimodality, 3D audio,motion tracking, ubi & ambient intelligence,Special strengths: Multi-disciplinary research,strong industry relations and experience

Augmented Reality teamResearch personnel 9+ peopleAR core technology & applications

AR at VTT – situation todayAR driven projects > 2 M€ annually+ VTT coordinated 10 M€ EU project ManuVAROverall, AR is now bigger than VR


Augmented Reality (AR)Mixes virtual objects with view of real worldC.f. Virtual Reality (completely virtual worlds)Properties: real-time, interactive, 3DTracking + rendering + display

Introduction / AR

Reality Augmented Reality Augmented Virtuality Virtual Reality

Mixed Reality

Related technologiesAugmented VirtualityMediated RealityDiminished Reality

-> Mixed Reality (MR)

4


ARCatalogueViewing with data glasses / web cameraStereo 3D products appear from catalogueBased on marker detection from camera view

Applications, e.g.B2B marketing (by Lappset Group )3D viewing of engineering drawings (by Pöyry Inc.)

Also, camera phonesConsumer solutions

Example Application


3D animated chirdren’s TV seriesBy Futurecode (Finland) and Blue Arc (China)Launched April 2nd, 2010, MTV3 / SubTV Junior

Augmented magazine advertisementTV-magazines Katso and TVSeiska, March 20103D animations appear augmented from magazineInteraction with augmented Dibidog

Dibidogs

5


Mobile phone implementationIntroduced at Sanghai World Expo, May 1st, 2010Nokia Maemo (N900) and Symbian S60 platforms

Augmented childrens’ storybookKustannus Paasilinna, August 2010Various new interactions and narrativesMarkerless tracking, photorealistic rendering etc.

Dibidogs, Next Steps


ALVAR Subroutine Library

”A Library for Virtual and Augmented Reality””AR engine” developed by VTTFree trial download at www.vtt.fi/multimedia/alvar.htmlUsers e.g. Columbia Univ. (Goblin XNA), MIT (6th Sense)

Detecting and tracking 2D markersAccurate marker pose estimation, with erro concealing“Marker fields” can be defined manually, or autocalibratedHiding of markers as sepcial feature

UtilitiesCamera calibration; Correcting lense undistortions; Opticalflow tracking; Several basic filters and Kalman library;Finding exterior orientation of point-sets

Coming upMarkerless tracking; Rendering module;Mobile phone implementation

http://www.vtt.fi/multimedia/alvar.html

6


ENTERTAINMENT AND GAMES


Entertainment Application - CamBall

Augmented networked table tennisPlayed with real rackets, color markerVideo image of opponent shown at other endof tableVirtual ball, table and net

FeaturesBall speed, racket size can be adjustedSolo game against a wall, or against "oneself"Game viewing by Internet audienceRecording of game

BenefitsSocially and physically activatingNatural human-computer user interactionFits in small interiors, at home or officeSuits also for kids and elderly peopleNo special hardware required

7


Mobile Entertainment - SymBall

Virtual table tennis game for camera phonesCamera driven game control by feature/motion detectionData connection between players via Bluetooth or GPRSFirst networked camera driven game in the world

ImplementationSymbian OS/Series 60 (& J2ME)Features: sounds, ball speed, etc.3D stereo game using r/b goggles!

Future plan: Augmented versionUsing turnable camera, face detectionShow real opponent on phone display


Internet

Experimental Work ”TelePet”Augmented Networked Games for Dogs

To entertain dogs over Internet during master’s absenceGame control based on gesture detection

Example gameMaster throws virtual ball, dog has to “catch”Rewards, e.g. biscuit box operated over internetInitial animal tests - positive results

8


Vappunaamari.comVirtual masks for May Day party

Face detection/augmenting on camera phonesProvided as public service, > 1500 downloadsFeatures: send photo to friends, user created masks,compressing augmented video, …


INTERIOR DESIGN

9


Augmented Interior Design – Alternative Solutions

Basic solutionVirtual furniture object tied to eachmarkerInterior designed by moving markerson the floorDisplay on PC screen or data glasses

BenefitsFurniture is ”light” to move aroundEasy to try out different alternativesNo modelling of real environmentrequired

Things to improve …Web camera … isn’t very mobileData glasses / PC .. not tooconvenient


Augmenting in Digital Camera Images

Application ”ARInteriors”Single marker placed on floor, definingscale and perspectiveDigital camera images of the room areuploaded to computerVirtual furniture is selected and placedwith menu/mouseFeatures: lighting, shadows, hiding ofmarker etc.

BenefitsHardware (camera, PC) found inevery homeBetter image quality, resolution,accuracy etc.Photorealistic 3D rendering

10


Augmenting on TabletPC

Application “ARMobile”Tablet PC (Sony Vaio), cameraintegratedOptional data glassesUser interface to fit small screen,stylus inputNo configuration files requiredArbitrary marker orientationsMarker fields calculated automaticallyMoving, scaling and rotating objectsAdjusting of lights, with soft shadowsErasing of markersStoring of video clip & projectKalman filter & other stabilisationFeature based tracking when markersdisappear


Find appealingfurniture in ad

Download virtualfurniture viamatrix code

Instantly viewfurniture at homeon mobile phone

Verify by high-endPC solution (ifrequired)

Get to shop,purchase product

Augmenting on Mobile Phones

Application ”ARPhone”Placing virtual furniture on phone’s camera imageModels are loaded using matrix code in product catalogue or advertUser places the furniture in camera view by 3D rotation and zoomFurniture stays in place when moving the camera phoneBased on feature tracking - no markers requiredAlso, augmenting of real furniture

11


ARStudio

Unified solution for general AR applicationsFormats: real time / stored video <-> still imagesSingle/multimarker tracking -> markerless methodsMatching augmented content with video qualityPhotorealism, shadows, materials, lighting, …


VividAR

AR interior design solution by VividWorksBased on VTT’s technologyLaunched February 2010Consumer web solution later this spring

12


ARCHITECTURE AND CONSTRUCTION


Case Study: VTT’s Digitalo officesDigitalo = “DigiBuilding”Construction started Dec 2003Building ready Oct 2005

Various AR/VR methods usedRadiosity visualization3D LandscapesCAVE Virtual RealityAugmented Scale ModelAugmented Web CameraMobile Augmented Reality

Projects involvedCommissions by Senate PropertiesProject ARARAT 2004-2006

Augmented Reality in Architecture

13


Digitalo augmented on conference tableViewing through video glasses (stereo)Move around model, communicate with people

FeaturesVirtual menu for user interactionVirtual reality tour utilising feature based trackingPredefined walk-through routes, with hyperlinksRealism: textures, reflections, shadows etc.Add, move and remove parts (AR-catalogue)

Augmented Scale Model


Digitalo augmented in live web camera viewPresentation for future users of buildingFollow-up and comparison with original plansView with or without virtual buildingReal-time interaction (zoom, pan, etc.)Viewing inside to the building, etc.

Ongoing development (project AR4BC)Integrating 4D IFC models to live web cameras

Augmented Web Camera

14


Outdoors visualization of DigitaloPioneering work by VTT 2003

Handheld display devicePDA + WLAN + cameraLightweight, “user friendly”, inexpensiveAllows frozen view, multiple users, etc.

Client/server solutionServer laptop PC for 3D computationStreaming video between blient and serverSpeed 2 frames / sec

Outdoors trackingUsing large 80 x 80 cm markersMobility by moving markers around

Mobile Augmented Reality


Markerless implementation “AROnSite”Feature based trackingInteractive positioning of building to real sceneVisualization in any unprepared environment

Applications, e.g.Helsinki Music Hall visualization 2006

Unprepared Environments

15


Google Earth + AROnSiteBuilding placed in Google EarthSemi-automatic positioning usingGPS

Current applications, e.g.Paper mill visualizations by PöyryExcavator visualization by MetsoSkanska new Helsinkk offices

Future workAutomatic positioning (direction)Photorealism: e.g. lighting,shadows, maskingEtc., being implemented in projectAR4BC (see next slide)

“Google Earth on Earth”


Project “AR4BC”

Augmented Reality for Building and Constructionvertical representation of Finnish & international B&C sectorduration 2008 – 2010budget 420 k€

Reality Augmented Reality Augmented Virtuality Virtual Reality

Mixed Reality

Building Information Models (BIM)

Physical model authors• Structural• Mechanical• Architectural• Electrical

Process model authors• Schedule• Classification• Logistics• Safety

Information users• View• Review• Comments• Approvals• Status• Feedback

16


Goals

Independent of Google Earth data formats (Collada)

Support for general/proprietary maps (cf. inaccuracy of GE maps)

Support for Building Information Models (e.g. IFC)

Support for time component, 4D (e.g. MS Project)

Support for several building models at a time

Photorealistic visualization (CG tuned for AR)

User interface (wxWidgets & new features)

Visual feedback to design system

Augmented Reality + Augmented Virtuality = Mixed Reality


INDUSTRIAL APPLICATIONS

17


Power Plants

Remote Maintenance TasksITER fusion reactor mock-up at TampereMaintenance robot augmented / animatedinside test platformApplication e.g. for discrepancy checking(after accident or other changes)


Project ”NOSE”

Partners: Pöyry, Metso, Wärtsilä, Fortum,Intergraph, Nokia, …

“Nomadic Use of a Plant Model”2007-2009, budget > 700 k€Motto: “Learning by Walking”

18


Simantics MobileMobile AR user interfaceStudio for content creationMobile database & Internet

Locationing technologyPseudosatellites (GPS insidefactory hall), WLAN, gait etc.Model & template based trackingSensor fusion

Applications, e.g.Mobile trainingMobile operationMobile maintenance

NOSE Implementation

Simantics MobileDatabase

Metso DNAOPC Server

VTT Knowpap

Simantics Mobile

Pöyry Virtual Mill

Simantics Mobile Studio


Project ”AugAsse”

“Increasing Efficiency in Assembly Work byAugmented Reality”

Duration 2006-2008, budget 800+ k€Partners Valtra, Metso, Wärtstilä. Kalmar. Sandvik, …

Implementation phases3D Puzzle, multimodal interactionCamera phone implementation, client/serverIndustrial case: hydraulic block by Valtra PlcIncl. data pipeline CAD -> AR, user studies etc.

19


Data Pipeline

Data pipeline from 3D CAD to augmented assemblyCATIA -> STEP -> Reduced B-rep -> Design for Assembly -> Animations -> Augmenting

Production parameters

and assemblability fig

ures

Assembly structure

Production parameters

and assemblability fig

ures

Assembly structure

Work Phase:23

Return tube

Tool:27 mm wrench

Work Phase:23

Return tube

Tool:27 mm wrench


Client/servereasy to implement, general solutiontracking and other heavy computation can be done on serverno need to store large model data on mobile deviceusing subimages, no need for any 3D support on mobile device

Still imagesin most cases, the assembly station is staticit is sufficient that the augmented content is “live”it is more ergonomic to view the image without having to point at targetrobust, provides good image size (accuracy) for trackingenables general implementation, e.g. on Java ME

Mobile Phone Implementation

20


Group Statistics

30 0:36:34 0:07:42 0:01:24

29 0:31:45 0:04:31 0:00:50

26 1,96 1,455 ,285

25 ,32 ,627 ,125

26 3,54 1,964 ,385

25 1,48 1,873 ,375

Instruction typepaper instructionsaugmented-realityinstructionspaper instructionsaugmented-realityinstructionspaper instructionsaugmented-realityinstructions

Assembling duration(h:mm:ss)

Observed frequencyof using a wrong tool

Observed frequencyof false positioning

N Mean Std. DeviationStd. Error

Mean

AugAsse User Study

Assembling is on average quicker with animatedinstructions and results in less mistakes than with paperinstructions.The usage of wrong tool and false positioning are onaverage less frequent with augmented-reality instructionsthan with paper ones.A target for improvement is the HMD hardware. Operationwith the helmet and wires was often experienceduncomfortable.


Project ”ManuVAR”

”Manual Work Support Throughout SystemLifecycle by VR/AR”

EU FP7 NMP “Beyond lean manufacturing”18 partners, 70+ participants, VTT as coordinatorDuration 2009-2012, budget 9.7M€

The general objective is to develop• (1) technology platform and• (2) methodological framework

for supporting all aspects of manual work throughout thesystem lifecycle with the ultimate goal to

1. increase productivity and quality and reduce costof manual work in the whole lifecycle

2. facilitate adaptation to product customization andchanges

3.support efficient knowledge and skillmanagement through the lifecycle

21


Satellite assembly Remote training Manufacturing design

Maintenance ofheavy machinery

Industrial plantmaintenance

Thales Alenia Space NEM Solutions TNO

Tecnatom

Metso Minerals

ManuVAR Applications


RECENT PROJECTS: MIXED REALITY

22


Mobile Mixed RealityCo-creation project “MMR” with Nokia


Mixed Reality Video ConferenceTeleconference btw real people and SL avatars

Avatars appear augmented in real room

People’s gestures are replicated by avatars

Collaboration with virtual objects

Partners VTT, TKK, Nokia, IBM

23


Multitouch InteractionMultitouch table developed by VTT

LCD display with cameras in the corners

Any flat surface can be used for multitouch interaction

Application example: connecting floor plan with CAVE,navigation in 3D architectural model

At Sanghai World Expo 2010: multitouch in the air,to control giant projector walls, tens of people

Collaborative work,cf. “Minority Report”


Thank You!

Dr. Charles WoodwardVTT Technical Research Centre of Finland

Vuorimiehentie 3, Espoo, FinlandTel: +358 40 500 1514

[email protected]

http://www.vtt.fi/multimedia

mailto:[email protected]

http://www.vtt.fi/multimedia

Documents

Augmented and Mixed Reality - oamk.fi · Project “AR4BC” Augmented Reality for Building and Construction ... Augmented Reality + Augmented Virtuality ... CATIA -> STEP -> Reduced