Wikitude SDK Advanced

Agenda for today

● Who is Wikitude

● What has the Wikitude SDK to offer

－Architecture and APIs

－Plugins API

● Latest Additions to the Wikitude SDK – 3D Tracking

3

v v v

Wikitude AR at a glance.

First AR app in the world in

2008

10,000 apps powered by

wikitude SDK

3D tracking technology

vREFERENCES

Wikitude’s path to success

20142013 2015

Business Model: B2CFocus: Wikitude AR App

2016201220112009

Founder stage in Salzburg garage:

First AR Browser launched.

2010

VCs brought in., Management hired:Martin Herdina, Andy Gstoll

Established Wikitude Inc. in USA

Business Model: B2BFocus: AR technology and tools

Achieved 25 million downloads and app installs gloabally

Launch of Wikitude SDK for 2D image recognition

Launch of Wikitude Studio, the AR CMS

Launch of Wikitude Cloud Recognition service

Launch of 3D tracking SLAM

Wikitude offices

wikitude GmbH (HQ) in Salzburg Austriawikitude Inc. in San Francisco, USA

wikitude Inc.

wikitude GmbH.

AR creator and CMSCloud Recognition

Wikitude AR Products & Tools

Connect your own CMS

Native API JavaScript API

Easy access to AR mobile development

Xamarin

PhoneGapCordova

titaniumunity

Android

iOS

GEO / sensor based AR

Pioneering products and technologies

Maturing products and technologies

2D Image Recognition & Tracking

Tailored SDKs for Smart Glasses

Partnership since 2014, optimized SDK for BT-200, full documentation →BT-300 in pipeline

Partnership since 2016 optimized SDK for R-7

Partnership since 2014, SDK for M100, full documentation →M300 in pipeline

Wikitude SDK Architecture

2D On-DeviceReco & Tracking

2D Cloud Reco3D Reco & Tracking

Unity3D

Unity Plugin

Plugins API

Custom CV plugin

Custom camera plugins

Your App

Your App Cordova

PluginTitanium Module

XamarinComponent

Your App

Cordova Titanium Xamarin

Your App

Your App

JavaScript APINative API

Your App

3D Rendering

Augmen-tations LBS

Plugin ManagerOpenGL ES2 Rendering

ARMv7 NEON optimizations

ARMv8 optimizationsCamera GPU

optimizations

OpenCL

RenderScript

Metal API

IMU

Hardware Abstraction Hardware Optimizations

C++ Layer

Plugins API - Modularization

13

Description

The Plugins APIs allows to extend the Wikitude SDK by 3rd party functionality. Plugins for the Wikitude SDK have access to the camera information, recognized images. This is perfect for additional functionality that also requires camera access. Plugins are written in C++ and can communicate both with the JavaScript API and the Native API.

Plugins API in detail

14

Custom Frame

Provider

Image Frame

Standard Frame

Provider

WikitudeCV EngineRendering

Custom CV Plugin

Plugins API

class Plugin { public:…void cameraFrameAvailable(const Frame&;

cameraFrame_); void update(const vector<RecognizedTarget>

recognizedTargets_); void startRender(); void endRender…

};

1234567

class RecognizedTarget {public:

const string&amp; getIdentifier() const;const Mat4&amp; getModelViewMatrix() const;const Mat4&amp; getProjectionMatrix() const;const float getDistanceToCamera() const;

};

Use-Cases for Plugins API

● Run own computer-vision related code in parallel to Wikitude SDK

－OCR Engine

－Barcode Reader

－Face Recognition

●Own the camera stream and the camera rendering

－Custom effects and shaders

－Remote camera stream with local processing

JavaScript Augmented Reality Experience

16

Architect Worlds consist of:

• .html file(s)

• .js file(s)

• .css file(s)

• Defines your augmented reality experience

• Use the Wikitude JS API provided within the Architect view context<script src=“architect://architect.js"></script>

• Loaded by the Wikitude SDKs Architect view

Architect world

17

Javascript execution triggers actions in the underlying C++ layer

Architect view composition

18

OpenGL ES 2 view* Renders camera stream* Renders augmentations

Web view* Loads main .html file* Transparent background

GeoObjects & HTML content

Menu barjQuery support

Geo-located POIs

Multiple Drawables- 3 images- 3 text labels

Web View

Augmented Reality View

Support for jQuery and web APIs

Image Recognition

Augment target images using JavaScript API capabilities (Drawables, Sounds, Animations,Videos, 3D Models)

var kitchenTracker = new AR.Tracker("Kitchen.wtc");var overlay = new AR.VideoDrawable("howdens.mp4", 0.65);var trackable2DObject = new AR.Trackable2DObject(

kitchenTracker, "Kitchen",{ drawables:

{ cam: overlay}});

Types of augmentations

22

Augmentations are represented by AR.Drawable subclasses

• Images - AR.ImageResource / AR.ImageDrawable

• Videos (also with alpha channel) - AR.VideoDrawable

• 3D models - AR.Model

• Web views - AR.HTMLDrawable

• Labels - AR.Label

….architecture, real estate….. ….gaming, filming industry….. ….indoor navigation, retail…..

New algorithms and concepts

3D Tracking / SLAMObject Reco & Tracking

Recognition & tracking (core engine)

3D

One core engine with many “flavors”

Small Scale Scenes / Spaces / Objects

- SLAM- Point cloud- Smaller scenes- desktop, small rooms and

environments

Object Recognition Edge Detection / CAD models- pre-registered CAD

models and displayed silhouettes

- Focus: shapes & lines

Positional Tracking Indoor Navigation”

- Computer vision plus IMU / sensor fusion

- Malls, airports, buildings- Untethered indoor gaming in

apartment, house or anywhere

Large Scale Scenes Spaces

- SLAM based- Point cloud- buildings / spaces- indoors, outdoors

3D Tracking Flavors1. Object Recognition Edge Detection / CAD models

Object Recognition Edge Detection / CAD models- pre-registered CAD

models and displayed silhouettes

- Focus: shapes & lines

Wikitude Lab – RnD Sneak Preview

3D Tracking Flavors2. Large Scale Scenes / Spaces

Large Scale Scenes Spaces

- SLAM based- Point cloud- buildings / spaces- indoors, outdoors

Watch here: https://www.youtube.com/watch?v=Zqb6Fo4uRkM

Positional Tracking – Demo Video

Philipp Nagele, CTOWikitude

See more.