MCS3109 Advanced Computer Graphics and Gaming

State-of-the-Art in Mobile Augmented Reality Development tools/APIs

T. A. Makumburage Index No: 13440481

1

Mobile Augmented Reality Augmented reality (AR) is related to a more general concept called ‘mediated reality’, in which a view

is modified by a computer. As a result the technology enhancing one’s current perception of reality.

Augmented reality is used to overlay a synthesized world on top of the real world, which will add

meaningful information to help with day-to-day life.

Mobile augmented reality is a relatively young technology, but it has already attracted a great deal of

hype and skepticism in equal measure. Overlaying digital information onto the real world, viewed

through a camera phone, is technically impressive, but the business models and usage patterns are

still evolving.

Technology In the recent years smartphones are the most developed aspect of the mobile computing paradigm.

Latest smartphones are equipped with brilliant cameras, accelerometers, GPS devices, compasses,

barometers, step counters and many more sensors to capture contextual data. Thus the smartphones

and tables are become suitable AR platforms within few years.

Hardware

“Turning your phone into the small miracle it truly is” This is the motto of famous mobile chip

manufacturer, and yes those latest processes are expanded up to 8 cores and speed is up to 2.5Ghz

per core. Certainly over powering the standard desktop PCs. Touch inputs and various sensors making

the life much easier for AR mobile application developers.

A high-end smartphone will include a camera within 8-40 Mega pixels, which can be used as an input

to a mobile AR application. Mobile devices are now passed the HD era of the displays, 2K & 4K displays

are starting to appear on the market, which will display every bit of detail an AR application can output.

Figure 1: Versatility of the mobile chipsets Figure 2 : Qualcomm Snapdragon 800 chipset

2

Operating system

The newest version of Google Android operating system supports ‘OpenGL ES 3.1’ specification. Apple

iOS supports ‘OpenGL ES 3’ specification. Windows Mobile doesn’t supports OpenGL, but supports up

to ‘Direct 3D feature level 9_3’. So these operating systems are capable of running high end AR

applications without any trouble with the help of the hardware. Software development kits of these

operating systems will provide APIs to accesses above mentioned 2D and 3D graphic rendering

facilities with hardware acceleration.

Applications Augmented reality has long sounded like a wild futuristic concept, but the technology has actually

been around for years. It becomes more robust and seamless with each passing decade, providing an

astonishing means of superimposing computer-generated images atop a user’s view of reality, thus

creating a composite view rooted in both real and virtual worlds. Following are some of the best

mobile augmented reality apps and games out there.

Wikitude Browser

Yelp Monocle

Google Ingress

SpecTrek

SnapShot

Showroom

Theodolite

Augmented Car

Finder

Sun Seeker

Acrossair Browser

Google SkyMap

Augment

Layar

Spyglass

ARBasketball

Lookator

iOnRoad

AR Invaders

ColAR Mix

Anatomy 4D

Blippar

Aurasma

AR Defender 2

3

Tools The main purposes of an AR SDKs are:

Recognition – What to track

Tracking – Where it is

Content Rendering

Augmented toolkits can be mainly categorized as open source and proprietary.

Open source o Argon

Argon is a standards-based Augmented Reality (AR) web browser. The goal of

Argon is to make AR accessible to the widest possible group of developers and

users through a web-centric platform model. Argon2 Isotope is the current version

of Argon and is programmed using familiar web technologies: HTML5, CSS3,

JavaScript, as well as media content (images, audio, video). 3D graphics is provided by an integrated

version of the popular JavaScript framework: three.js.

o ARToolKit

ARToolKit is a computer tracking library for creation of strong augmented reality applications that

overlay virtual imagery on the real world. To do this, it uses video tracking capabilities that calculate

the real camera position and orientation relative to square physical markers in real time. Once the real

camera position is known a virtual camera can be positioned at the same point and 3D computer

graphics models drawn exactly overlaid on the real marker. So ARToolKit solves two of the key

problems in Augmented Reality; viewpoint tracking and virtual object interaction. This enables the

easy development of a wide range of Augmented Reality applications. Some of the features of

ARToolKit include:

Single camera position/orientation tracking.

Tracking code that uses simple black squares.

The ability to use any square marker patterns.

Easy camera calibration code.

Fast enough for real time AR applications.

SGI IRIX, Linux, MacOS and Windows OS distributions.

Distributed with complete source code.

o JavaCV

JavaCV uses wrappers from the JavaCPP Presets of commonly used libraries by researchers in the field

of computer vision (OpenCV, FFmpeg, libdc1394, PGR FlyCapture, OpenKinect, videoInput,

4

ARToolKitPlus, and flandmark), and provides utility classes to make their functionality easier to use on

the Java platform, including Android.

o DroidAR

DroidAR is a framework for Augmented Reality on Android. Location based AR and marker based AR

both are supported in DroidAR. This framework can be used for many different scenarios,

Footstep recognition & Indoor AR

Location based Gaming & Crowdsourcing Framework

o ATOMIC Authoring Tool

This was created as a front end (Graphic Interface) for use ARToolKit

library without having to know programming. It is Multi-platform

which allows you to use it in operating systems Microsoft Windows,

Ubuntu and Mac OS X. The main motivation of ATOMIC is to provide to the community an open source

tool that can be easily modified and that does not require too much technical knowledge to access

the augmented reality technology.

o Goblin XNA

Goblin XNA is a platform for research on 3D user interfaces, including mobile augmented reality and

virtual reality, with an emphasis on games. It is written in C# and based on the Microsoft XNA platform.

Goblin XNA inherits some of the goals of the earlier Goblin project, but with a shift in emphasis toward

core 3D user interface functionality, leveraging the existing functionality of DirectX 3D game engines

and development environments. The platform currently supports 6DOF (six degree of freedom)

position and orientation tracking using marker-based camera tracking through ARTag with OpenCV or

DirectShow, and InterSense hybrid trackers. Physics is supported through BulletX and Newton Game

Dynamics. Networking is supported through Lidgren library. Goblin XNA also includes a 2D GUI system

to allow the creation of classical 2D interaction components.

Proprietary

o AR development toolkits

PointCloud SDK

PointCloud SDK is a free library that lets you build iOS

applications that have advanced computer vision

capabilities like Simultaneous Localization and Mapping (SLAM) and image detection and tracking.

PointCloud Lab has been acquired by Facebook and is joining the Oculus team.

5

Vuforia Augmented Reality SDK

The Vuforia platform uses superior, stable, and

technically efficient computer vision-based image

recognition and offers the widest set of features and

capabilities, giving developers the freedom to extend

their visions without technical limitations. With

support for iOS, Android, and Unity 3D, the Vuforia platform allows you to write a single native app

that can reach the most users across the widest range of smartphones and tablets. Some of the new

features of new Vuforia 4.0 are,

o Object Recognition

Object Recognition, one of our most requested features, enables apps to recognize and track a wide

range of objects. This new capability is ideally suited to bring toys and consumer products to life.

o Vuforia Object Scanner

The Vuforia Object Scanner is an Android app that makes it easy to create object targets. As you scan

an object, the app provides real-time visual feedback on the target quality, coverage, and tracking

performance, allowing you to test and adjust the target even before starting app development.

o iOS 64-bit Support

Support for building 64-bit apps on iOS is now available, giving you the ability to create compelling

user experiences on compatible iOS devices that are more responsive and richer in graphics.

o AR content management systems

Metaio Creator

The metaio Creator is augmented reality software that allows

users to create a complete AR scenario without specialized

programming knowledge through a drag and drop interface.

BuildAR.com

Launched in 2009, buildAR was the world's first AR Content Management System

(AR-CMS). The first version launched as a test to see if people were interested in

being able to create their own augmented experiences. It turned out you were! So

then we got busy developing a production version and officially launched the new

and improved buildAR at the Augmented Reality Event in Silicon Valley in May 2011.

6

Catchoom CraftAR

The SDKs licensed by Catchoom Technologies offer tools to

connect your branded mobile application with the CraftAR

service in the cloud. There are two SDKs available for different

operating systems and devices. Following are some of the features of the CraftAR SDK.

Camera capture management

Cloud Recognition

3D Object Tracking

AR content rendering in AR view

Automatic download of AR content

stored online

AR content description parser

AR content type extension

Hoppala Augmentation

With more than 7000 registered developers Hoppala

Augmentation is the world’s largest content platform for geo

based augmented reality. It seamlessly publishes your content to

all three major AR browsers: Layar, Junaio and Wikitude. Create once and publish everywhere. The

graphical web interface makes it very easy to build geo referenced content with just some mouse

clicks, there is no software installation required and no coding needed at all.

Webcam Social Shopper

The Webcam Social Shopper, often referred to as virtual dressing room software, debuted online in

June 2009 and was created by Los Angeles based software company, Zugara. Cited initially as an

"Augmented Reality Dressing Room", The Webcam Social Shopper allows online shoppers to use a

webcam to visualize virtual garments on themselves while shopping online. The software also uses a

motion capture system that allows users to use hand motions to navigate the software while standing

back from their computer.

o End-to-end branded app solutions

Alive app

It uses the mobile device's video camera to recognize pre-trained

images and overlay an image, video or 3D content on the recognition

of image. Aliveapp is the only augmented reality application in the

world to be available on the four major platforms (Android, iOS, Windows, Blackberry Symbian and

Java) and is augmenting the complete newspaper. It is the first publicly available application designed

for a Newspaper Giant that used a location-based approach to Augmented Reality along with the

image based Augmented Reality.

7

Aurasma

Aurasma is HP Autonomy's augmented reality platform. It is available as an

SDK or as a free app for iOS and Android mobile devices. Aurasma's image

recognition technology uses a smartphone's or tablet's camera to recognize

real world images and then overlay rich media on top of them in the form of

animations, videos, 3D models and web pages. Both businesses and

individuals use Aurasma to create and share their own augmented reality experiences as well as to

discover hidden digital content around them. Teachers are among the most active group using the

platform.

Junaio

Junaio is an Augmented Reality browser designed for 3G and 4G

mobile devices. It is developed by Munich-based Company, Metaio

GmbH. It provides an API for developers and content providers to

generate mobile Augmented Reality experiences for end-users. The smartphone app as well as the API

are free to use. Currently, it is available for Android and iPhone platforms. Junaio is the first

augmented reality browser that has overcome the accuracy limitations of GPS navigation through LLA

Markers (latitude, longitude, altitude marker).

Layar

The layar browser makes use of the Accelerometer, built-in camera,

compass and GPS sensors. These are used together to identify the

user’s location and field of view. From the geographical position, the

various forms of data are laid over the camera view like inserting an

additional layer. Data in the browser comes in the form of layers.

Layers are REST web services serving geo-located points of interest in the vicinity of the user. Layers

are developed and maintained by third parties using a free API. Layar as a company is responsible for

their validation in the publication process.

Nokia City Lens

Nokia City Lens, is an augmented reality (AR) software that gives dynamic

information, through the phone’s camera display, about users' surroundings such

as shops, restaurants, and points of interest, shown as virtual signs overlaid on or

above buildings. A commercial Beta version was release in 2013. It also has free

of charge turn-by-turn voice guided navigation, Nokia Drive 3.0, which can also

be used without internet connection using preloaded maps. It's also possible to de-clutter the

8

surroundings: version 1.5 has a "Sightline" feature where the user can narrow her view to just what’s

in her direct line of sight, making it easier to spot interesting places.

Wikitude

Wikitude is a mobile augmented reality software which is

developed by the Austrian company Wikitude GmbH

(formerly Mobilizy GmbH) and was first published in October

2008 as freeware. It displays information about the users' surroundings in a mobile camera view,

including image recognition and 3d modeling. Wikitude was the first publicly available application that

used a location-based approach to augmented reality.

For location-based augmented reality the position of objects on the screen of the mobile device is

calculated using the user's position (by GPS or Wi-Fi), the direction in which the user is facing (by using

the compass) and accelerometer. The location-based augmented reality view provides a direct view

of reality enhanced with additional, computer-generated content, as opposed to a conventional map,

which provides an abstract view from above. A central element in the Wikitude World Browser is the

location. Compared to regular web pages, where the location of the user plays a minor role, Wikitude

World Browser is optimized for mobile location-based usage.

Content in the Wikitude World Browser is mostly user generated. Content can be added by a web

interface, by KML, and ARML. In addition, Web services are available to register the delivery of

dynamic data. Wikitude is a W3C member and OGC member and is working to develop ARML further

as part of a W3C ARML project.

Word Lens

Word Lens is an augmented reality translation application from Quest Visual.

Word Lens uses the built-in cameras on smartphones and similar devices to

quickly scan and identify foreign text (such as that found in a sign or a menu),

and then translate and display the words in another language on the device's

display. The words are displayed in the original context on the original

background, and the translation is performed in real-time without connection to the internet.

XARMEX

XARMEX (short for: eXtended Augmented Reality for Military EXercise) is

an augmented reality-aided close quarter’s combat simulation system,

combining motion detection hardware with image overlay/stabilization

software to create realistic military simulation environments and computer games.

9

Comparison SDK Purpose Tracking Platform Graphics Cloud GPS License

Qualcomm Vuforia

2D Images, Markers

NFT, Marker, Text

iOS, Android Unity3D & Alt.

yes no Free

Metaio SDK

2D Images, GEO, 3D, Anywhere

NFT, GPS, 3D, SLAM

iOS, Android Unity3D & Alt.

yes yes 0-$10,000

Wikitude SDK

2D Images, GEO

GPS, NFT iOS, Android, Blackberry

HTML, Proprietary

yes yes 0-$2,300

Layar SDK 2D images NFT, QR iOS, Android Proprietary yes no $3,250/app/yr. + $20/page

13th Lab Point Cloud

2D Images, 3D, Anywhere

NFT, 3D, SLAM

iOS, Android Unity3D & Alt.

no no 0-$5,000

AR Toolkit 2D Markers 2D Images

NFT, Marker

iOS, Android Unity3D & Alt.

no no $0 (GPL) - $2,495 + royalty

HP Aurasma SDK

VFX, Videos on 2D Images

NFT iOS, Android Proprietary yes yes Per click

Obvious Engine

2D images, Cylinders

NFT, Cyl.

iOS, Android Unity3D & Alt.

no no $8000/app/platform

String Framed 2D images

Framed Image

iOS Unity3D & Alt.

no no $499 -$8000/app/year

References 1. Wikipedia. [WWW] [Feb 2015, 20]

http://en.wikipedia.org/wiki/List_of_augmented_reality_software.

2. Augmented World Expo. [WWW] [Feb 2015, 23]

http://en.wikipedia.org/wiki/List_of_augmented_reality_software.

3. Slide Share - Mobile AR SDK Comparison and Tutorial [WWW] [Feb 2015, 25]

http://www.slideshare.net/patrickoshaughnessey/2013-0603augmented-

worldexpomobilearsdktutorial