Tendencies in Industrial AR

Timo Engelke

AR Community Meeting

Munich, 09/13/14

Tendencies in Industrial AR

§  Tendencies §  Web-Trends §  ARVIDA Project §  Streaming Services

§  Web Standards Tendencies and their use

§  WebRTC §  WebComponents

Tutorial Outline

Related Technology Trends

3D Data & Documents

3D Printer

3D Scanner

GEO Daten

3D PLM

Web-based Engineering

Cloud based Rendering

Mixed Reality

Convergence of Application-

Platform

HTML5

Titanium

Peper NaCI

Windows8

W3C WebApps

Chrome OS PhoneGap

Divergence of System- Platform

Table

Android

SmartPhones

Desktop

Cloud-Bases

WindowsPhone

Mac OS

Amazon Kindel

Windows

iOS

Web-Service Architecture

Current State of 3D on the Web plugin-based extension to user agent

Client Flash 11 Silverlight 5 Java, Java3D, JOGL and JavaFX (SUN) O3D (Google): Javascript based scene-graph API X3D (ISO, web3d consortium): plugins with SAI interface MPEG-4 & MPEG-4 Part 11 (ISO, Moving Picture Experts Group)

Server ParaViewWeb, OnLive, Gaikai, …

General Issues:

=> Installation/Security => Bound to single Technology => Data/Event model is separated from HTML-page => plugin-specific scripting interface (e.g. SAI for X3D)

§  JavaScript API to OpenGL ES 2.0 §  Khronos (not W3C) specification §  First Version in 2010 §  Solid support in 2014 §  Chrome, Firefox, Opera, IE11 §  Available on Desktop and Mobile

§  General Issues: §  => Low level Graphics API §  => Most application build on unstandardized middleware §  => Most middleware solutions (e.g. three.js) build for entertainment

apps §  => Security §  => Application limited to W3C technology stack (e.g. data/image

loader)

WebGL the rising star

Scenario: Web-Application with interactive 3D Canvas

Declarative Scenegraph Part of HTML-document DOM Integration CSS/ Events

<canvas>

Imperative Procedural API Drawing context Flexible

2D (Final HTML5 spec)

3D (No W3C spec yet)

Declarative (X)3D in HTML Completes todays graphics technologies

Declarative (X)3D in HTML Embed a live scenegraph in the DOM

<!DOCTYPE html > <html >

<body> <h1>Hello X3DOM World</h1> <x3d xmlns=‘…’ profile=‘HTML’ > <scene> <shape> <box></box> </shape> </scene> </x3d> </body>

</html>

DOM / HTML Document Binary asset resources

Fraunhofer IGD / Autor / Abteilung

Explicit Binary Container -  Directly loaded to TypedArrays -  Data assignment in JS -  Multiple arrays per file -  Multiple files per scene

Images and Videos -  Encodes int/float arrays (e.g.

coordinate, normal, texCoords, generic-attributes) in RGBA-images

-  Multiple images per array -  Multiple images per scene

Separate structure and data + priority controlled rendering

X3DOM: Makes “Declarative 3D” available OS Polyfill build on HTML5, JS and Plugin/WebGL/Flash

Fraunhofer IGD / Autor / Abteilung

Application examples www.x3dom.org

§  Platform/operation system independent development §  Scalability §  Server-based Rendering §  Server-based Tracking §  Hybrid Rendering (Culling on the server-site, Streaming of the

Culling Geometry) §  Thin Clients & Interfaces to PDM-Environments §  Exchangeability of Web-Services §  Use of different rendering/tracking/simulation services §  Dynamic Processing of Augmented Reality Resources

Possibilities of Web-Technologies in Augmented Reality

http://www.arvida.de/

13

ARVIDA Consortium

§  To establish an Augmented/Virtual Reality Reference architecture §  that makes uses Web-technologies §  supports exchangeability of technological components §  to evaluate the performance of web-based Augmented Reality

§  Applications / Use Cases §  the project is driven by industrial use cases §  the requirements and KPIs for the evaluation are defiend by

industrial ene users §  Applications in the field of (Automtive, Ship Building, Aviation, §  Variance Analysis §  Assistence Systems §  Motion Analysis and ergonomics §  Product Validation

Goals of ARVIDA

Efficient processing of large data sets

Real applications tend to be huge HTML-files Unpleasant non-interactive user experience Browser are not build to hold GByte of DOM attribute data (e.g. multiple data copies)

Reference external sub-trees X3D “Inline” node black-/white-box interface?

Binary XML decompression x3z: (ISO) Decoding on JS-Level x3db: (ISO) Fast Infoset: No UA or JS-lib EXI: (W3C) Even worse

Transcoder Service Async data preparation and caching infrastructure §  RESTful service to translate CAD-data §  Optimization for specific classes

(e.g. mobile GPU) §  Asynchronous JOB-model returns target resource immediately §  Http update: 200, 202 (with time estimation), 404

Transcoder Service Async data preparation and caching infrastructure

§  2250 JT-Parts

§  32 Threads

§  ~ 60 Sec

§  WebSocket notification §  Efficient Cache infrastructure minimizes data processing §  Most operations are massively parallel and only IO bound §  Can use http/https or DB/PDM resource handler (e.g. TeamCenter)

Transcoder Service Example Siemens JT-Engine

Progressive Data Loading

§  Client-Based Rendering §  Web-GL based Rendering

on the Client §  e.g. X3DOM

§  Server-based Rendering §  Bi-direction Streaming: §  Streaming of the captured

camera image to the server §  Re-streaming of the

rendered images to the client

§  Hybrid Approaches §  Culling of the Geometry

on the server §  Transfer of the culled

Geometry to the client

Client-Server Scalability

Server

Mobile Device 1

request

Touch-Events, Camera

Port

Display

Port Assignment

Start Application instance

Start Mobile Application

Mobile Application

running Application instance running

Visualization Service Smart combination of server/client based rendering

Pro/Con Latency

HW-Req. Net trafic Secuity Simplicty Scalabiltiy

client ++ -- + -- ++ --

hybrid ++ + ++ + + +

server -- ++ -- ++ -- ++

Performance Client / Data in DOM

Client / Data in BG/SIG

Hybrid Server

Million Tri. 0.2 15 150 500

Hybrid Rendering 80. Mil. Tri. http://www.youtube.com/watch?v=h0SUWqJfQsE

instant3DHub - WebVIS Secure, efficient and open CAD-Data visualization

CAxWebApp

PLMXMLViewer

IssueViewer

ColorCoder

DBPDMAPI

HTTP/HTTPSREST

TeamCenter

ProvidesResources

OpenJTCATIA (V4/V5)

ACISIGES

InventorPro/E

SolidWorksX3D/VRML

Collada

WebVISLibrary (JS)

PartView

CRUD

API

ToolBoxAPI

Instant Reality Plugin

OpenGL VGRRRT

X3DOM

WebGL IPPFlash

Remote Buffer Viewer

WebRTC WSMJPG

TranscoderTranslation and

Optimisation Cache Service

PIXIVisualisation Service

CAx Data

GB

Images

CAx Data

CAx DataOpen

GL VGRRRT

WebVIS Secure, efficient and open CAD-Data visualization

WebVis Resource Service (Win7 Box)

WebDesktop (HTML5 Browser + WebGL)

PLM andCAD-Daten

Transcoder

WebPartCache

(Apache)

3DWebVis3DWebVis

3D WebVis(IFrame)

InteraktiveProdukt-struktur

User Auth.

CUD CADPart

Part Picking

DocumentDocument

WebPart

CUD Part

R Part

Request CADPart+ User Auth.

Request WebPart+ User Auth

WebPart Data

CRUD Part Meta-Data

WebVis Architektur V03, JB, 2013-10-07

CADPart Data

WebVIS Efficient CAx visualization in WebApps

Single JavaScript (JS) library for WebApp developer and integrator Minimalistic JS-interface hides service infrastructure and communication

createVis ( URI, ID, { optional Data } ); URI: CAx-Resource (e.g https://foo/bar.jt ) or “local://group” Can transport authentication tokens updateVis ( ID, color/highLight/visibility, value ); Update the element color, hightLight or visibility property addCallback ( ID, onClick/touch/over, callBack-Function ); allows the application to react on element specific user input

removeVis ( ID ); Remove a specific element

Allows WebApps to visualize CAx-Date (almost) directly Hidden and automated client and server based caching infrastructure

WebVIS – PLMXML Frontend Efficient CAx visualization in WebApps

Example: Review WebApp “PQM Review/Web"

WebRTC = Web Real-Time Communication

AR and WebRTC – What is WebRTC?

WebRTC is an API definition drafted by the World Wide Web Consortium (W3C) that supports browser-to-browser applications for voice calling, video chat, and P2P file sharing without plugins.

§  2010 Google acquired Global IP Solutions (GIPS) and therewith the basis for the technologies.

§  2011: §  W3C – Standardization Group §  Internet Engineering Task Force - IETF §  Release of free reference famework as source code

§  2012: §  First integrated in Chrome

§  2013: §  In Firefox and Opera

§  2014: §  Microsoft? – Maybe not?

WebRTC - History

§  Main Page: www.webrtc.org §  io13webrtc.appspot.com – Nice intro video and slides §  www.html5rocks.com/en/tutorials/webrtc/basics/ §  http://dev.w3.org/2011/webrtc/editor/webrtc.html

Low level development: www.webrtc.org

WebRTC - Resources

Basic Platforms: §  Chrome §  Chrome for Android §  Firefox §  Opera §  Native Bindings avaliable for:

§  Android (Java) §  OSX/iOS (OBJ-C) §  Linux §  Windows

WebRTC – on different Platforms and Services

Services:

WebRTC - Technologies

3 Core - Components

Acquiring audio and video

MediaStream (aka getUserMedia)

Communicating audio and video RTCPeerConnection

Communicating arbitrary data RTCDataChannel

WebRTC – Typical Scenarios for Network based driven AR

§  Local Tracking and remote Assistance

Remote Expert Repository, Signaling

Client, Tracks

WebRTC


§  Remote Tracking and remote Assistance

Remote Expert Repository, Signaling

Thin Client

WebRTC

Dedicated tracking


§  Typical Scenarios §  Remote Expert

§  Local Tracking and remote assistance §  Remote Tracking and remote assistance

§  Remote Tracking §  Send Image and get Pose back §  Send Image and get Augmented Image Back

§  Recording of Sessions §  Local Tracking and remote Rendering §  Remote Rendering


§  Hands on Demo §  Early Approach using local tracking

and remote assistance

WebComponents - Future

§  Resource: webcomponents.org §  As soon as available, might change the web

radically – including AR in the web

)

Example: Mobile VE-Applications UI elements to explore data-elements as part of the temp.