44
Deep-Dive into the Wikitude SDK

Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK

Embed Size (px)

Citation preview

Page 1: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK

Deep-Dive into the Wikitude SDK

Page 2: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK

2

Who is talking

Philipp Nagele

● CTO of Wikitude (Product & Technology)

● MSc in Computer System Engineering from Halmstad University in Sweden

● Previous companies:

- 3united/VeriSign Inc.

- T-Mobile Austria

Page 3: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK

Agenda for today

●Who is Wikitude

●Architecture of the Wikitude SDK

●A deeper look at the JavaScript API and the Plugins API

Page 4: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK

v v v

Wikitude AR at a glance.

First AR app in the world in

2008

10,000 apps powered by

Wikitude SDK

3D tracking

technology

v

REFERENCES

Page 5: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK

5

HQ office in Salzburg, Austria

Page 6: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK

6

Page 7: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK

AR creator and CMSCloud Recognition

Wikitude AR Products & Tools

Connect your own CMS

Page 8: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK

Mock-ups only

… understand what you are looking at

Mock-ups only

Page 9: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK

… make the World your playground

Page 10: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK

….augment books

Mock-ups only

Page 11: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK

… augment print, ads, catalogues, packaging

Page 12: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK

… virtually mount products

Page 13: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK

Maintenance

Page 14: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK
Page 15: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK

Platform agnostic technology

15

AR/VR wearables TabletsSmartphones

multi platform technology core based on C++

Page 16: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK

Tailored SDKs for Smart Glasses

Partnership since 2014, optimized SDK for BT-200, full documentation →BT-300 in pipeline

Partnership since 2016 optimized SDK for R-7

Partnership since 2014, SDK for M100, full documentation →M300 in pipeline

Page 17: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK

Native API JavaScript API

Easy access to AR mobile development

Xamarin

PhoneGapCordova

titaniumunity

Android

iOS

Page 18: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK

Wikitude SDK Architecture

2D On-DeviceReco &

Tracking

2D Cloud Reco

3D Reco & Tracking

Unity3D

Unity Plugin

Plugins API

Custom CV plugin

Custom camera plugins

Your App

Your App Cordova

PluginTitanium Module

Xamarin Componen

t

Your App

Cordova

Titanium

Xamarin

Your App

Your App

JavaScript APINative API

Your App

3D Rendering

Augmen-tations LBS

Plugin Manager

OpenGL ES2 Rendering

ARMv7 NEON

optimizations

ARMv8 optimization

sCamera

GPU optimization

s

OpenCL

Metal APIIMU

Hardware Abstraction Hardware Optimizations

C++ Layer

Java/Obj C

Page 19: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK

Wikitude JS SDK Java API

Wikitude JS SDK has a very slim Java API

• ArchitectView• Lifecycle events (onCreate, onPause, onResume, …

methods)• Load Architect Worlds

Everything else is optional• ArchitectUrlListener• ArchitectWorldLoadedListener• CaptureScreenCallback

Page 20: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK

Architect view composition

20

OpenGL ES 2 view* Renders camera stream* Renders augmentations

Web view* Loads main .html file* Transparent background

Page 21: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK

JavaScript Augmented Reality Experience

21

Architect Worlds consist of:• .html file(s)• .js file(s)• .css file(s)

• Defines your augmented reality experience• Use the Wikitude JS API provided within the Architect

view context<script src=“https://wikitude.com/libs/architect.js"></script>

• Loaded by the Wikitude SDKs Architect view

Page 22: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK

Architect world

22

Javascript execution triggers actions in the underlying C++ layer

Page 23: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK

Types of augmentations

23

Augmentations are represented by AR.Drawable subclasses

• Images - AR.ImageResource / AR.ImageDrawable• Videos (also with alpha channel) - AR.VideoDrawable• 3D models - AR.Model• Web views - AR.HTMLDrawable• Labels - AR.Label

Page 24: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK

Wikitude JavaScript API

24

Model• AR.GeoLocation• AR.RelativeLocation• AR.Tracker• AR.ImageResource

View• AR.ImageDrawable• AR.VideoDrawable• AR.Model• AR.Label

Controller• AR.GeoObject• AR.Trackable2D• AR.PropertyAnimation• AR.AnimationGroup

Page 25: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK

Geo AR

25

How to define e.g. a point of interest (POI)

• Define where the augmentation is placed by creating AR.GeoLocations (latitude/longitude coordinates)

• Define the augmentation by creating AR.Drawables

• Combine location and drawables using AR.GeoObjects

= The same geo object is displayed at different locations

POI CPOI A

POI A

Poi B

Page 26: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK

Pokémon Go like Geo AR

26

How to build a Pokémon Go like AR experience:

Page 27: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK

Pokémon Go like Geo AR

27

How to build a Pokémon Go like AR experience:

• Define the position of a Pokémon• Relative to the current user position using AR.RelativeLocation

//a relative location being 12 meters south, 20 meters west and 13 meters lower than current position of the uservar relativeLocation = new AR.RelativeLocation(null, -12, -20, -13);

Page 28: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK

Pokémon Go like Geo AR

28

How to build a Pokémon Go like AR experience:

• Define the position of a Pokémon• Relative to the current user position using AR.RelativeLocation• Absolute to the user using AR.GeoLocation

//latitude, longitude, altitudevar location = new AR.GeoLocation(47.77317, 13.069929, 320.);

Page 29: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK

Pokémon Go like Geo AR

29

How to build a Pokémon Go like AR experience:

• Define the position of a Pokémon• Relative to the current user position using AR.RelativeLocation• Absolute to the user using AR.GeoLocation

• AR.Model object to render the 3D model of a Pokémon

//create a new Modelvar model = new AR.Model(“models/pokemon.wt3”);

Page 30: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK

Pokémon Go like Geo AR

30

How to build a Pokémon Go like AR experience:

• Define the position of a Pokémon• Relative to the current user position using AR.RelativeLocation• Absolute to the user using AR.GeoLocation

• AR.Model object to render the 3D model of a Pokémon• Can be animated through .fbx defined animations

//We want to start an animation inside a modelvar model = new AR.Model(“models/pokemon.wt3");var modelAnim = new AR.ModelAnimation(model, "hit"); //start hit-animation defined inside model

Page 31: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK

Pokémon Go like Geo AR

31

How to build a Pokémon Go like AR experience:

• Define the position of a Pokémon• …

• AR.Model object to render the 3D model of a Pokémon• …

• AR.GeoObject to actually bring the model onto the screen

// a GeoObject with drawables set on creation timevar geoObject = new AR.GeoObject(location, {  drawables : { cam: [model] //the drawable representing the GeoObject in the camera view  }});

Page 32: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK

Pokémon Go like Geo AR

32

How to build an even better Pokémon Go like AR experience:

• Use AR.GeoObjects onEnter/ExitFieldOfVision trigger• trigger custom events when the Pokémon becomes visible in the

camera

geoObject.onEnterFieldOfVision = function() {  /* your custom behaviour e.g. start a certain model animation */};geoObject.onExitFieldOfVision = function() {  /* your custom behaviour */};

Page 33: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK

Pokémon Go like Geo AR

33

How to build an even better Pokémon Go like AR experience:

• Use AR.GeoObjects onEnter/ExitFieldOfVision trigger• trigger custom events when the Pokémon becomes visible in the

camera• Define a direction indicator drawable for your Pokémon

• Helps users to find the Pokémon around them

var imageResource = new AR.ImageResource(“assets/pokemonIndicator.png”);var pokemonIndicator = new AR.ImageDrawable(imageResource, 0.2);

// a GeoObject with drawables set on creation timevar geoObject = new AR.GeoObject(location, {  drawables : { cam: [model], //the drawable representing the GeoObject in the camera view indicator: pokemonIndicator  }});

Page 34: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK

Pokémon Go like Geo AR

34

Page 35: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK

GeoObjects & HTML content

Menu barjQuery support

Geo-located POIs

Multiple Drawables- 3 images- 3 text labels

Web View

Augmented Reality View

Page 36: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK

Pokémon Go like Geo AR

36

How to build an even better Pokémon Go like AR experience:

• Use AR.GeoObjects onEnter/ExitFieldOfVision trigger• trigger custom events when the Pokémon becomes visible in the

camera• Define a direction indicator drawable for your Pokémon

• Helps users to find the Pokémon around them• Use the AR.Radar component to permanently show the position of

a Pokémon in the AR scene

Page 37: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK

Pokémon Go like Geo AR

37

Page 38: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK

Pokémon Go like Geo AR

38

How to build an even better Pokémon Go like AR experience:

• Use AR.GeoObjects onEnter/ExitFieldOfVision trigger• trigger custom events when the Pokémon becomes visible in the

camera• Define a direction indicator drawable for your Pokémon

• Helps users to find the Pokémon around them• Use the AR.Radar to permanently show the position of a Pokémon in

the AR scene• Capture a screenshot after you caught a Pokémon using the

corresponding ArchitectView Java API.

Page 39: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK

Image Recognition

Augment target images using JavaScript API capabilities (Drawables, Sounds, Animations,Videos, 3D Models)

var kitchenTracker = new AR.Tracker("Kitchen.wtc"); var overlay = new AR.VideoDrawable("howdens.mp4", 0.65); var trackable2DObject = new AR.Trackable2DObject(

kitchenTracker: "Kitchen", { drawables:

{ cam: overlay }});

Page 40: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK

Wikitude SDK Architecture

2D On-DeviceReco &

Tracking

2D Cloud Reco

3D Reco & Tracking

Unity3D

Unity Plugin

Plugins API

Custom CV plugin

Custom camera plugins

Your App

Your App Cordova

PluginTitanium Module

Xamarin Componen

t

Your App

Cordova

Titanium

Xamarin

Your App

Your App

JavaScript APINative API

Your App

3D Rendering

Augmen-tations LBS

Plugin Manager

OpenGL ES2 Rendering

ARMv7 NEON

optimizations

ARMv8 optimization

sCamera

GPU optimization

s

OpenCL

Metal APIIMU

Hardware Abstraction Hardware Optimizations

C++ Layer

Java/Obj C

Page 41: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK

41

Plugins API in detail

Custom Frame

Provider

Image Frame

Standard Frame

Provider

Wikitude CV EngineRendering

Custom CV Plugin

Plugins API

class Plugin { public: … void cameraFrameAvailable(const Frame&; cameraFrame_); void update(const vector<RecognizedTarget> recognizedTargets_); void startRender(); void endRender …};

1234567

class RecognizedTarget {   public:      const string&amp;    getIdentifier() const;       const Mat4&amp;      getModelViewMatrix() const;       const Mat4&amp;      getProjectionMatrix() const;      const float      getDistanceToCamera() const;};

Page 42: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK

Use-Cases for Plugins API

● Run own computer-vision related code in parallel to Wikitude SDK-OCR Engine-Barcode Reader-Face Recognition● Own the camera stream and the camera rendering-Custom effects and shaders-Remote camera stream with local processing

Page 43: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK

….architecture, real estate….. ….gaming, filming industry….. ….indoor navigation, retail…..

New algorithms and concepts

3D Tracking / SLAMObject Reco & Tracking

Page 44: Philipp Nagele (CTO, Wikitude) An Insider Deep-Dive into the Wikitude SDK

See more.At Booth B6