Google Tango APIPierre-Luc St-Charles

Software developer, Vision and Imaging Team, CRIM

24

IntroductionWhat is Tango?

● “Augmented Reality Computing Platform”● Provides high-level API for Android-based AR applications (C, Java, Unity…)

● Developed by Google (>2012) & device vendors (Lenovo, Asus, ...)

● Primary use case: track device in 3D environment for live apps (SLAM!)● Processing 100% done on device, in real time

○ Needs some pretty fancy computing hardware! (CPU+GPU)○ Also needs a custom camera array (more later)

Project Tango

25

Introduction‘Where’ is Tango? (1st generation)

● Two development platforms (‘Peanut’ phone, ‘Yellowstone’ tablet)

● One commercial device available now (Lenovo Phab 2 Pro)

● One commercial device available soon (Asus Zenphone AR)

Project Tango

26Sources: Google, Lenovo, Asus

Hardware & Device SpecsWhy so special?● Multi-Camera sensor array:

○ Custom for all devices (Tango provides abstraction layer)○ Typically: standard+fisheye cameras, and IR projector/sensor

● Processing power:

○ ‘Peanut’ = Qualcomm quad-core + Movidius Myriad 2 VPU (!!)○ ‘Yellowstone’ = ARM quad-core + Nvidia Tegra K1 (~300 GFLOPS GPU)○ Phab 2 Pro = Qualcomm Snapdragon 652 (octa-core w/ ~200 GFLOPS GPU)○ Zenphone AR = Qualcomm Snapdragon 821 (quad-core w/ ~500 GFLOPS GPU)

Project Tango

27

Source: hothardware.com

Software & APIThree main functionalities:

● Motion Tracking○ based on visual-inertial odometry (fisheye cam + accel + gyro)○ allows accurate 6DOF tracking of device (even w/o depth!)

● Area Learning○ based on large scale visual/depth feature extraction & matching○ for relocalization in known spaces, and drift correction

● Depth Perception○ based on structured light (or) time of flight (or) photogrammetry○ for surface intersection estimation, and 3D reconstruction

Project Tango

28Source: Asus

Software & APISimplified usage:● Connect to Tango service (already installed on device!)

● Setup configuration (permissions, I/O, parameters…)

● Setup data callbacks (for point clouds, images, position, …)

● Load Area Description File (if available, for relocalization)

● Run! (and feed data back to 3D reconstruction module, if needed)

Project Tango

29

DemosFirst Demo :

Raw Point Clouds

Second Demo :Wosomtech’s EasyBuild

(Plane Fitting using Raw Point Clouds for Room Modeling)

Third Demo :Dense 3D Reconstruction

Project Tango

30

3D Reconstruction ResultsYou might recognize this place… (CRIM 2nd floor!)

Project Tango

31

Total Scan time @ ~40 mm resolution = 20 mins

ConclusionIn short:● “Real-time” mobile SLAM now very “real”

○ Accurate point clouds & device localization at >30Hz● Unlocks tons of new consumer applications

○ Home modeling, furniture previews (IKEA/Amazon), AR gaming, ...

● Simple API for non-experts in computer vision○ Can now even automatically extract ‘floor plans’ from building reconstructions

● 1st generation devices out now!

Project Tango

For more info onWosomtech’s EasyBuild:

www.easybuild.io32