© Fraunhofer IGD Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
Special considerations for Standards in Mobile ARInternational Workshop on AR Standards – 11. + 12. October 2010, [email protected] - Institute for Graphics Research IGD
© Fraunhofer IGD Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
Fraunhofer IGDInstitute for Computer Graphics Research
Mission: Technologies and Applications of Visualization, Interaction and Communication in the field of ICT
Director: Prof. Dr. techn. Dieter W. Fellner 4 Locations: Darmstadt, Rostock (Germany), Singapore and Graz
(Austria) 180 Researchers (FTE) 14,7 Mio. € Turnover -> Department Virtual and Augmented Reality
© Fraunhofer IGD Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
Dep. Virtual & Augmented RealityResearch Topics
Computer Vision
Computer Graphics
Interaction Technologies
© Fraunhofer IGD Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
Overview
The aim of the presentation is triggering consciousness about the idea of AR and presenting an open approach practically ..
Where are the problems in standardization of AR? What are the common structures and where are variations? What is the role of those structures in the context of AR? Technological hardware issues Why and what kind of standards are there and do we need? Our approach to overcome the mentioned issues and help
developing standards
Cannot cover the whole horizon of problems .. But can try …
© Fraunhofer IGD Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
ProblemsGeneral goal of having standard(s) which apply to general needs of AR.
What is the problem? There are tracking technologies for AR, but they are in an early
stage. Every application type needs its custom strategy, technology and specialized content.
What do we want to cover with a standard? Due to the complexity of many parties involved in the field of AR and
the high potential of possibilities. The terminology AR is not clearly differentiated generally and can
lead to confusion. Which sense is augmented by what and through what? 2D,3D, training, leading, informing, disinforming, helping, covering
etc.. Diverging hardware and software technologies Bringing the paradigm to content masses
© Fraunhofer IGD Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
CommonTechnologies
CommonTechnologies
Arising potentials of technological developmentin contrast to commonly used technologies in apps Mass -
Solution
Customized Mass-Solution
CommonTechno-logies Cutting
Edge-Solution
Scientific SpecificLaboratory-Solutions
Time
Development
Provide tools for everyoneLayar, Wikitude etc..
Branded…i.e. iLiving
Complexity
© Fraunhofer IGD Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
Augmented RealityAbstract Building Blocks
ContextEngine
Content
AnnotationEngine
ContextInformation
Interaction Engine
ApplicationModel
Reference Model Annotation Model
Sensors
Output
InteractionDevice
© Fraunhofer IGD Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
Computer Vision Based Augmented Reality
Camera
VisionEngine
Content
3D AnnotationEngine
Extrinsics (+ Intrinsics,Occlusions,Lights,Models etc..)
Interaction Engine
Image+Intrinsics
Display ImageScreen
ApplicationModel
Reference Model Annotation Model
© Fraunhofer IGD Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
Role of Reference Model
Retrieval of the camera Pose (6Dof) Find the absolute coordinates of user and objects
GPS, Compass, Acceleration Data Use prototype model for registration (i.e. Poster, Marker, Lines…)
Representation Absolute in latitude, longitude, altitude, roll, pitch, yaw Relative in Cartesian coordinates and angles (or axis angle, quaternion..) In relation to some object or object group – real or virtual?
Interaction Find differences of formulated and real world (POI,AOI,VOI)
This creates potential content for interaction Track movable and deformable objects Detect user interaction/behavior using sensors (Hands, gestures etc..)
Allow feedback of generated data
© Fraunhofer IGD Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
Role of Visual Annotations in ARBuilding elements in GUIsVisual Aids Direct Visual Aids (DVA): Superimposed 3D-models/animations
(direct association with object) Indirect Visual Aids (IVA): “Annotations” related to tracked object
(reminder)
Mental Model Building User’s internal representation of the task Can be improved by providing context information
Passive Parts Information about the task in general Higher-level instructions global picture
Collaborative Annotations (expert trainer)
DVA
IVA
© Fraunhofer IGD Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
Role of the interaction model
Customizing the Annotation Model Manipulation Scene Graph Interactive Content
Customizing the Reference Model Dynamically change tracking behavior and point of focus (POI) Allow interchange in between Annotation and Registration Model Use of declarative formulation as far as possible. Allow scripting for specialization Tracking and Events: 2D-Projected Positions with distance hint Detection Events (i,e. Object enters/leaves display) Interaction of user/objects with the scene
Content must be delivered free or encrypted.. Who will do this?
© Fraunhofer IGD Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
Hardware technology issues on mobile devices
Low computational resources in relation to modern PCs Limited memory OpenGL ES Different camera modules (Intrinsics) Sensor support and calibration Partly supported hardware acceleration Software Interfaces and Languages Symbian (C++, Microkernel, Qt) Android (Java, C++ through NDK) iPhone (Obj-C, C++ support)
Algorithms need to be optimized for HW Costly and hard to maintain large set of handsets in one common
Code-Base
© Fraunhofer IGD Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
Open StandardsSustainability for AR-ContentApplications and generation tool chains should be based on.. X3D (ISO/IEC 19775:2004) HTML5 ECMAScript (ISO/IEC 16262:2002) JavaScript CSS STEP-Format KMLBenefits Sustainability Interoperability Porting Rapid development
© Fraunhofer IGD Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
X3DOM (x3dom.org)HTML5/X3D Integration Model
Simple car configurator to interactively change color and rims
N900 with miniB
© Fraunhofer IGD Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea15
X3DOM (x3dom.org)HTML5/X3D Integration ModelX3DOM: experimental open-source framework and runtime One way for integration of HTML5 and declarative 3D Ongoing discussion in the Web3D and W3C communities
Declarative content design Allows embedding 3D data directly into the HTML-DOM First approaches of CSS integration Application concepts map well to generic scene-graph
Runtime independent (supports native implementation, plugin, WebGL)Suitable for desktop and mobile systems alike
© Fraunhofer IGD Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
instantReality – mobileAR - BrowserExtendable generic AR-Web-Browser for mobile devices based on HTML5 and Javascript
© Fraunhofer IGD Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
mobileAR – Open Architecture Proposal for open realization of declarative approaches
Generic Application (arBrowser)OS
WEB-View (HTML5 +JS)
DOM-Declarative
JavaScript
IMA
X3Dom
webGL
SVG A2ML
Native Interface
VisionRender Sound
Open Annotation Open Context
Content
AR-KML
??
Sensors
Abst
ract
ion
Individualization of App
Reac
habi
lity
for m
asse
s of
deve
lope
rs
© Fraunhofer IGD Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
mobileAR – AR – Browser for mobile devicesAdditionally to Web-Browser a renderer and vision engine is integrated. Extends the internal Web-Browser with a simple but powerful API Interaction and visualization can be done declarative in HTML and/or
via JS Can render X3D via native renderer or even X3Dom Generic Vision Interface Integrates Vision Processor (VisionLib) for
Fiducials (Markers, Image-Markers) Poster-Tracking Object recognition Panorama and 3D-Model Based scene tracking Server based Tracking (bidirectional streaming)
Additional Features (about to come…) Remote Video & Audio Support Scenarios Encryption and packaging
Camera
Live/Pause
dImage
3D-View
X3D
optional
DebugView
WEB-View
HTML5+JS
© Fraunhofer IGD Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
Next steps…Identification and unification of common technology data Annotation Data Reference Data
Create open framework for declarative approaches Open Annotation Engine Open Context Engine Unify Vision and Sensor based approaches KML in the context of industrial maintenance applications Evaluate using show-cases
Extend native interfaces Incooperate Sound A2ML Better content management (Offline Usage, Encryption)
Make it free for masses for developing
© Fraunhofer IGD Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
Happy Workshopping
THANK YOU !
© Fraunhofer IGD Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
Building Block Hierarchy – What data to processContext EngineBased on available sensors in device
ContextEngine
LocalizationEngine
VisionEngine
AuditiveEngine
Other sensors..
GeolocationEngine
© Fraunhofer IGD Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
Building Block Hierarchy - DerivativesContext EngineBased on capabilities of device
AnnotationEngine
2D-AnnotationEngine
VideoEngine
AuditiveEngine
3D-AnnotationEngine
GeolocationEngine
Sound-Annotation
Engine
© Fraunhofer IGD Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
Parties involved in the game of AR
Physical Player The user of the app Network Provider Producer of Cellphones Application Developer Content
developer/provider Technology developer AR Standardization
…some of them want to
Learn something Network Save costs Save resources Save time Earn money Have fun Sustainability
© Fraunhofer IGD Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
Identification of needs...
App Devel
ContentProvide
r
ContentCreator
User(s)
CommunicationProvide
r
DeviceCreator
Techno-logy
Devel
The user of the app Network Provider Producer of Cellphones Application Developer Content developer/provider Technology developer AR Standardization
© Fraunhofer IGD Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
Applications (Fallback Model in Action)
IE8 Plugin (OpenGL and Raytracing) and native iOS app with x3dom layer Runtime independent – supports (native implementation,) plugin, and WebGL
© Fraunhofer IGD Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
Geo Located - Augmented RealityAbstract Building Blocks
LocalizationEngine
Content
AnnotationEngine
Position
InteractionEngine
ApplicationModel
Reference Model Annotation Model
Localization Sensors
Output
InteractionDevice
© Fraunhofer IGD Timo Engelke, A4, October 11th,2010, International Workshops on AR Standards, Seoul, Korea
Geo located and Vision Based Augmented RealityTechnically Abstract Building BlocksCamera
LocalizationEngine
Content
2D AnnotationEngine
Extrinsics
Interaction Engine
Display ImageScreen
ApplicationModel
Reference Model Annotation Model
Localization Sensors Intrinsics
Source: Layar