18
DEPARTMENT OF INFORMATION TECHNOLOGY Augmented reality Display of possibilities of augmented reality using Java and C++ Marcus Agelii & Gösta Agerberg Supervisor: Maya Neytcheva & Sergio Courtade Project in computer sience: Report January 2013 PROJECT REPORT

Augmented reality - it.uu.se€¦ · What is augmented reality Augmented Reality (AR) is the process of detecting objects in the reality and adding information to that reality. The

  • Upload
    others

  • View
    37

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Augmented reality - it.uu.se€¦ · What is augmented reality Augmented Reality (AR) is the process of detecting objects in the reality and adding information to that reality. The

DEPARTMENT OF INFORMATION TECHNOLOGY

Augmented reality Display of possibilities of augmented reality using Java and C++

Marcus Agelii & Gösta Agerberg Supervisor: Maya Neytcheva & Sergio Courtade Project in computer sience: Report January 2013

Jan 2013

PR

OJE

CT

RE

PO

RT

Page 2: Augmented reality - it.uu.se€¦ · What is augmented reality Augmented Reality (AR) is the process of detecting objects in the reality and adding information to that reality. The

Abstract

The purpose of this project is to explore augmented reality and investigate its possibilities.

Another target was to have software which ran on a mobile device. In this report we

summarize the conducted study on the current progress in the field and describe two

applications, for Android, developed under the project work. The applications

demonstrate what augmented reality can be used for. The applications are written in Java

and C++.

Page 3: Augmented reality - it.uu.se€¦ · What is augmented reality Augmented Reality (AR) is the process of detecting objects in the reality and adding information to that reality. The

3

Contents

1. Introduction

What is augmented reality ............................................................................................... 4

Markers ............................................................................................................................ 4

Augmented reality - early history and current development ........................................... 5

Augmented reality for mobile devices ............................................................................ 6

Android applications ....................................................................................................... 6

2. Project description and aims

Problem description ......................................................................................................... 7

Limitations ....................................................................................................................... 7

Goals ................................................................................................................................ 7

3. Workflow and implementation ..................................................................................... 8

4. Results and conclusions

Logotype app ................................................................................................................. 12

Virtual buttons app ......................................................................................................... 13

Conclusions .................................................................................................................... 13

5. Discussion

Difficulties ..................................................................................................................... 14

Possible developments of our apps ................................................................................ 14

Possibilities of Augmented Reality ................................................................................ 15

6. Acknowledgements ...................................................................................................... 17

7. References ..................................................................................................................... 18

Page 4: Augmented reality - it.uu.se€¦ · What is augmented reality Augmented Reality (AR) is the process of detecting objects in the reality and adding information to that reality. The

4

1. Introduction

What is augmented reality

Augmented Reality (AR) is the process of detecting objects in the reality and adding

information to that reality. The best way to explain this further is by an example: Imagine

that you are looking at a screen; the screen shows a camera feed of today’s newspaper.

When the camera comes to a certain picture in the newspaper, instead of just showing that

newspaper with the picture, it instead shows a video taking the picture’s place in the

newspaper.

The effect is achieved by enabling the program that detects the picture to search through

its database looking for what actions to take when it finds that exact image. This specific

program has videos as actions, and uses it by overlaying the image by the video.

Markers

To detect something in reality an AR application needs certain input; this is usually from

a camera but could also be from other sources such as GPS or accelerometers. If the AR

application uses a camera feed as an input and recognizes an object using image

recognition software, this object is called a marker. Often specific images are used as

markers but it could also be real objects such as a car or a building. There is also an AR

technology called markerless based AR, instead of trying to recognize specific markers

from a camera feed, this technology tries to do a mapping of the surrounding environment

and uses this map to augment reality.

Figure 1: Picture in a newspaper being replaced with a video using augmented reality

Page 5: Augmented reality - it.uu.se€¦ · What is augmented reality Augmented Reality (AR) is the process of detecting objects in the reality and adding information to that reality. The

5

Augmented reality - early history and current development

AR as a concept and idea has been around for many years but it’s only in the recent years

that technology has been made small and fast enough to make AR available for everyday

use. The first AR system was developed in the 60’s but the term AR was not coined until

1990 by a Boeing researcher called Tom Caudell.1 One of the first uses of simple AR

technology was by pilots in fighter jets, where the pilot had a HUD (Heads Up Display)

showing altitude, velocity and other useful flight data in their main field of view. This can

however be discussed as being real AR but when this technology evolved into also

showing waypoints and cues over targets (Target designators) this becomes more of what

people see AR as today.

In recent years as mobile phones and handheld devices have been made cheaper and more

powerful, the potential of AR applications have increased significantly. Today there are

many apps available to download from both Google play and Apples App store. The

application field of these apps ranges from navigation to entertainment, home furnituring

etc. There is however still a lot of developing needed before these apps surpasses the

usefulness and the reliability of other methods and apps not using AR.

Figure 2: A HUD display in a fighter jet.

1 Kangdon, Lee; http://link.springer.com/article/10.1007%2Fs11528-012-0559-3

Page 6: Augmented reality - it.uu.se€¦ · What is augmented reality Augmented Reality (AR) is the process of detecting objects in the reality and adding information to that reality. The

6

Augmented reality for mobile devices

AR right now is optimal for mobile devices and especially for smartphones. The

smartphones have the ingredients needed for the AR to work, a camera, a CPU and a

screen, all in the same small package. Another feature that makes smartphones very

suitable for implementing AR apps is their accessibility both to consumers and to

developers. It is easy to use a smartphone and it is easy to develop software for a

smartphone. The smartphone is also very mobile and as such very easy to move around an

augmented object. Since the smartphones also have other detectors such as

accelerometers and GPS, these can also be used when handling AR.

Android applications

Android is the name of an operating system primarily used for mobile and tablet devices

but also in other devices such as cameras and watches.2 Android is based on Linux but

have been further developed by Google to better suit smartphones and tablets (such as

energy management). Android is open source and primarily developed by Google, but due

to its open source nature many carriers and smartphone manufacturers have adapted

Android to fit their products. For Android there is a wide variety of third-party

applications that can be installed on devices running Android. These applications (apps)

are either distributed by Google’s own store called Google play or by other available

stores such as Amazon store, or by third party websites since it’s possible to

download .apk files to an android device and install them manually.

To develop applications for Android, Google have released an Android Software

Development Kit (SDK) and the applications are developed using Java. This software kit

includes all that is needed to start developing applications for Android including a

debugger, software libraries and tutorials. There is also a native development kit available

for applications or extensions in C or C++. The official Integrated Development

Environment (IDE) supported is Eclipse3 using the Android Development Tools plugin.

2 http://www.android.com

3 http://www.eclipse.org

Page 7: Augmented reality - it.uu.se€¦ · What is augmented reality Augmented Reality (AR) is the process of detecting objects in the reality and adding information to that reality. The

7

2. Project description and aims

Problem description

The project description for our project was rather general and did not contain any specific

details and limiting conditions. Our supervisor gave us the “AR for mobile devices” topic

and ideas in this topic to investigate and let us set the goals and limitations by ourselves.

The ideas to focus on were basically three; the image analysis involved in reading the

images, the creation and reading of information from the database, or the projection of

information. Therefore our first task was to actually come up with a more specific project

description. An advice we got from our supervisor was to “not invent the wheel again”

and by having this in mind we decided to focus on already built up SDK:s and investigate

how these worked and what could be done with them.

Since we didn’t want this to become merely a literary study, we decided that if possible,

we wanted to create an application that would display the possibilities of AR in an easy

way. In the short description we got, it was preferred that we would work on mobile

devices so we decided that our applications target would be a smartphone of some kind.

Limitations

We had to set up our own limitations, but since neither we nor our supervisor had any

experience in the subject we had no idea how to set them up initially. We did some

research to find a suitable library to work in, and then some more research to see what

this library could do for us, and what we could do with it.

This resulted in that we limited ourselves to the third alternative for specialization – the

displaying of information. We also narrowed our project down to using a library called

NyARToolkit4 and working in Windows. The last two limitations proved to be

unnecessary, since we early in the project found out that the limitations of NyARToolkit

were too strict to fit what we were interested in doing.

More research paved way for a library called Vuforia; this library had more functions

built in, was up to date and offered the possibility to more easily write an application for a

mobile device. This meant that our new limitations sent us towards an Android

application, and also the new experience in programming for a mobile device.

Goals

To merge our project description together with the limitations we posed, we decided to

focus on the following three goals. First, do a literary research of AR together with

Vuforia5, to better know what is possible to do, and also how to do this. Second, to keep

our research from becoming merely a literary study we wanted to create an application for

Android that displayed some of the AR features. Third we decided to provide ideas for

further development, as an inspiration for other projects within the same area of research.

4 http://nyatla.jp/nyartoolkit/wp

5 https://www.vuforia.com

Page 8: Augmented reality - it.uu.se€¦ · What is augmented reality Augmented Reality (AR) is the process of detecting objects in the reality and adding information to that reality. The

8

3. Workflow and implementation

We started by using a library, called NyARToolkit, and managed to successfully

implement this on a PC. Though easy to implement, NyARToolkit didn’t contain all the

functionalities we needed, neither was it especially suitable for a mobile platform, and we

switched to Vuforia.

The library we have been using, Vuforia, is a library developed by Qualcomm, for AR

and smartphones. This library contains a lot of resources, along with a couple of tutorials

which makes for quick learning. After a period of researching Vuforia, setting up the hard

– and software, we started by successfully implementing all of the tutorials and making

sure we understood the code.

Next step was to create our own application. Since we decided to create an application for

Android, we used Vuforia together with the Android SDK and Eclipse.

We wanted to do something that involved both Uppsala University and Schlumberger, so

we decided to try to do something with their logotypes. To do this we had to first make

the application be able to recognize the logotypes. This is currently being done using a

web service that Qualcomm is providing. It works by uploading an image of the marker

you want to find, let their classifier find characteristics in the image, and finally it

presents the marker together with a grade of how easy the marker will be to spot. For

example our markers for Uppsala University and Schlumberger were not that good. This

depends on the markers not using the entire image space; there are a lot of gaps of plain

white which doesn’t help the image recognition. More so, the logotype of Uppsala

University is almost perfectly rotational symmetric, this is making it difficult for the

algorithm to find out what is up and what is down. As a last problem, both images don’t

have sharp edges, or differ greatly in contrast, which is also something that the image

analysis algorithm is looking for. When done you can download the markers as two files

and attach them to you application. You get an .xml file containing the names and sizes of

the markers and a .dat file which is containing the data.

Page 9: Augmented reality - it.uu.se€¦ · What is augmented reality Augmented Reality (AR) is the process of detecting objects in the reality and adding information to that reality. The

9

Figure 3: The Eclipse development environment.

As soon as the application can recognize the marker it’s time to decide what you want it

to do when it finds the marker. We wanted to produce something that was showing the

possibilities of AR, and that also was linked to the project and the companies. Therefore

we decided to project text strings and one rotating logotype, all done in 3D.

The creation of text is a bit tricky; this is because OpenGL, which is used in Android,

doesn’t support easy dynamical text creation and projection. It is possible to display text

on the screen relatively easy, but since we wanted the text to be produced on the marker,

we had to do find a way to do this.

We ended up having to create the text in Blender6, an open source 3D content creation

suite. This meant that the text ended up being in 3D, but at the cost of it taking much

more time than just writing a string like one would do for a normal JAVA or C++ program.

The logotypes were also created using Blender, together with Inkscape7, a program that

can automatically trace an image and turn the image into scalable vector graphics in

the .svg format.

6 http://www.blender.org

7 http://inkscape.org

Page 10: Augmented reality - it.uu.se€¦ · What is augmented reality Augmented Reality (AR) is the process of detecting objects in the reality and adding information to that reality. The

10

Figure 4: Rendering the Uppsala University logotype in Blender.

Finally to import the objects into our application, we had to use a Perl8 script that turned

the .obj file into OpenGL graphics.

The objects don’t contain any textures, simply for the reason that they were already large,

and were content because we could apply generic textures as long as we accepted that our

objects could only have one color.

To display the objects as we wanted them to be displayed, we had to rewrite and write a

lot of new code. There are a lot of smart functions inside the Vuforia library though, and

this has helped us a lot. We decided to let the logotypes spin, and be displayed above the

marker, while the text should be static and displayed in the marker’s plane – all of which

would help to show the possibility of walking around the object since it is locked to the

marker and not to the screen.

To make the object spin we created a function that uses the difference in time in between

two calls for the system time, and rotating the object before projecting it. Doing this often

makes it look like the object is spinning.

8 http://www.perl.org

Page 11: Augmented reality - it.uu.se€¦ · What is augmented reality Augmented Reality (AR) is the process of detecting objects in the reality and adding information to that reality. The

11

We created one more app as well, to show the possibilities of virtual buttons. Virtual

buttons are a function that creates buttons on the marker that when pressed sends a signal

to the application to react. In our example, we created a spaceship that can rotate and

move forward and backward. This also makes for a nice AR effect, where if you look at

the spaceship right on, and push the button to make it fly away, it goes out of screen

quickly. But, if you move the screen so that you see the space ship from behind, you can

see it fly all the way out to periphery and disappear.

Page 12: Augmented reality - it.uu.se€¦ · What is augmented reality Augmented Reality (AR) is the process of detecting objects in the reality and adding information to that reality. The

12

4. Results and conclusions

Two different applications for Android are implemented. One shows text and an

augmented 3D-logotype of Uppsala University or Schlumberger when the app detects

their respective logotype.

Logotype app

This application has five different markers. Three of the markers are different versions of

the Uppsala University logotype and two markers are different versions of the

Schlumberger logotype. Depending on whether the Uppsala University logotype or the

Schlumberger logotype has been found it does different but similar things. If the Uppsala

University logotype (in any of the three different versions) has been detected, a red 3D

augmented logotype is shown spinning above the marker. The position of this augmented

logotype will be in the middle of the found marker but a small bit above the plane of the

marker. In addition two pieces of text that lies in the plane of the marker will also be

visible. These text pieces are 3D-objects but since it is placed in the plane of the marker it

will give the appearance of being 2D. A third feature is also that when the Uppsala

University logotype has been detected an options-menu button is changed from linking to

google.com to instead linking to uu.se.

When the Schlumberger logotype has been detected, the red augmented Uppsala

University logotype is changed to a spinning Schlumberger logotype and the link is

instead to Schlumbergers homepage slb.com.

Figure 5: Our logotype app displayed on an Android device.

Page 13: Augmented reality - it.uu.se€¦ · What is augmented reality Augmented Reality (AR) is the process of detecting objects in the reality and adding information to that reality. The

13

Virtual buttons app

This app uses a generic image as a marker. This image consists of a wood cabinet

combined with four different buttons. The main feature of this image is the four buttons

with which the user can interact. The Image itself is taken from the Qualcomm examples

and is used because of its good properties as a marker.

When the marker has been found by the app a spaceship will be drawn on the screen

above the plane of the marker. The buttons are best described as virtual buttons and can’t

be interacted with on the screen of the smartphone, but must instead be interacted with as

if they existed for real on the image. To press and interact with the buttons the user must

cover the button that he wants to interact with from the view of the smartphone camera.

The buttons have different uses and you can accomplish four different things by pressing

the different buttons. The leftmost button makes the spaceship roll right in its flight path.

The second button makes the spaceship roll left and the third and fourth button makes the

spaceship travel either forward or backward. The buttons works as “press and hold-

buttons” meaning that when the first button is pressed, the spaceship will start rolling

right with a constant speed. When the button is released the spaceship will stop rolling

and maintain the angle at which it stopped. The same approach is used for all of the other

buttons as well.

Figure 6:A screenshot from an Android device using the virtual buttons app.

Conclusions

We think that we have succeeded in reaching our goals, by researching AR and creating

two applications. On the way we managed to present a couple of possible uses for AR,

that in our meaning, points in the direction that AR is a medium that is worth developing

and can be of good use.

Page 14: Augmented reality - it.uu.se€¦ · What is augmented reality Augmented Reality (AR) is the process of detecting objects in the reality and adding information to that reality. The

14

5. Discussion

Difficulties

The main difficulties within our project were first and foremost to decide what we were

going to focus on. AR is such a big area, with a lot of possibilities. The project course

plan practically only stated that we should do something within AR.

Further on, we had to do some research before we found Vuforia, and we also tried with

another library before Vuforia. We weren’t really content with the functionalities and the

demands of NyARToolkit and therefore we came to the conclusion that we might have to

do some more research to find a better suited library.

We also had some problems due to the fact that almost all new computers nowadays runs

on 64-bit systems, and the first library that we used (NyARToolkit) used some very old

components that demanded a 32-bit system. This was however quickly solved when we

found Vuforia, since it is updated and supports almost all platforms.

The biggest challenge perhaps was that we had to learn how to create objects in 3D, and

to convert these into OpenGL standard. This is due to the lack of easy support for writing

text within Vuforia and OpenGL. There are a few workarounds but due to the limitations

in time for this project we decided to do this the same way it was presented in one of the

tutorials we found. To do this we had to learn how to create 3D objects, and we chose a

program called Blender. This was completely new to us and therefore took some time to

get a hold of.

Possible developments of our apps

As a part of this project, we also decided that we would present a couple of possible

improvements, so that further developing could be done by new groups. Since the project

course only is 15 credits, this means that no group will be able to do all of the proposed

tasks, but our propositions should instead be considered as inspiration to new project

works.

First of all we would like to see improvement being made in the image analysis. This is a

part where we haven’t done anything and as a result we ended up relying solely on the

built-in image analysis in Vuforia. Though it works fine, it has some restrictions that keep

us from using whichever picture we would like, and this is something we see as a

potentially big improvement.

The creation of markers is the second area we have thought of. Right now, markers are

being made using the Vuforia homepage, by uploading the desired image, and then

downloading one .dat and one .xml file. We would like to see the creation of markers

being so simple that the user should be able to take a photo with a camera, and then

choose to use it as a marker. This together with the improved image analysis could make

it possible for any marker at all to be used.

The third proposition is to work on simplifying the creation and displaying of text. This

should be one of the simpler tasks, and make a big difference. In the end a lot of

information is written, thus you can easily imagine that text will be the most projected

object. If it would be possible to create the information dynamically, one could also have

Page 15: Augmented reality - it.uu.se€¦ · What is augmented reality Augmented Reality (AR) is the process of detecting objects in the reality and adding information to that reality. The

15

updated information being projected. Together with internet access it would be possible to

use updated information from homepages, process these and display the info. One can

also use the example above and add, to the newly taken photo of a marker, a couple of

sentences describing the marker – or maybe even a link to the homepage where the

information should be gathered.

The probably biggest and most important improvement however might be to be able to

use some kind of cloud-based storage system. If all the markers and information can be

kept online, it would dramatically decrease the size of the application, and in the

meantime making it possible to store almost unlimited amount of data. Again using the

previous example, one can think of a database that isn’t local anymore, but expanded by

users worldwide. This would also make for a database that expands very quickly and at

almost no cost at all. The application has to be connected to the internet at all times, but

together with some smart search algorithm, should be quick in finding if the current

picture is a marker, and if so – what information is connected to this marker. The search

algorithm also has to keep a check so that two users don’t upload two versions of the

same image and by doing so making it hard for the image analysis to detect which of the

pictures is most similar to the camera feed.

Possibilities for Augmented Reality

We can see a lot of areas where AR could be used, some of them now and some in the

future.

Video games is a market that is constantly developing and pushing its frontiers forwards

toward more realistic gameplay and new and spectacular gaming experience. With that in

mind, we think that AR could be a huge success in certain video games. With the use of

an arena and gaming glasses, one could create football, war or adventure games. You

would need to keep track of the players, their movements and actions, but since this is

already accomplished using for example the Kinect9 tool for Xbox 360; this shouldn’t be

a main concern. To be able to offer a Battlefield or Call of Duty10

experience where

you’re playing using your body to move, having a 360 degree of sight and the feeling of

being inside the game, would dramatically improve the experience.

One can also think of more practical possibilities, such as furniture fitting. Here you can

imagine persons that are thinking of buying a shelf; they want to know whether or not it

fits in their home – by size, color and style. Produce a simple marker and put it where you

want the shelf, and by using your mobile device you can see the actual size and style of

the object, and of course change the color – all by using AR.

AR could also be used as installation walkthrough, whether we’re talking furniture

assembling with IKEA products or for example electronic devices. Recently an American

show produced an example of how to use AR as a tool to install a router, with the mobile

device recognizing the type of router and also which cables should be inserted in which

9 http://www.xbox.com/sv-SE/Kinect/GetStarted

10 Battlefield and Call of Duty are two popular video games, in the FPS (First Person Shooter) genre. They

exist in numerous editions but all of them are in a war setting.

Page 16: Augmented reality - it.uu.se€¦ · What is augmented reality Augmented Reality (AR) is the process of detecting objects in the reality and adding information to that reality. The

16

order and position.11

Another idea is to use AR as a presentation tool. This can for example be used for

companies who want to demonstrate their latest product, and to make it possible for

potential buyers to see the actual product in 3D and be able to move around it using a

more physical approach than just watch the product on a computer screen. It can also be

thought of as accompanying other information, and extending its possibilities. Since May

2012 there is an app for iPhone that together with the Metro newspaper lets the user see

video clips of news that the paper otherwise would be restricted to describe in text and

maybe publish a picture of.12

Since most papers already feature a web version where they

have video contents, this is a simple and cheap way to offer this functionality to

customers who prefer to read a physical newspaper but still are interested in additional

material.

The best example of how highly rated AR is today is the looming introduction of Google

glasses. That such a big company has invested a lot of resources into an AR project

indicates that they really have confidence in the technique. Google glasses are believed to

have a huge impact on the way and speed at which we will retrieve information. Without

the obligation of using your hands, they can let you find out which bus that will take you

home, who built the building across the street or what groceries is on sale in the local

shop and which recipes that goes hand in hand with these.

11

Mills, Matt;

http://www.ted.com/talks/matt_mills_image_recognition_that_triggers_augmented_reality.html 12

http://www.metro.se/nyheter/i-dag-far-din-metro-tidning-liv/EVHleB!n64olTSHiS1VQ

Page 17: Augmented reality - it.uu.se€¦ · What is augmented reality Augmented Reality (AR) is the process of detecting objects in the reality and adding information to that reality. The

17

6. Acknowledgements

We would like to thank Maya Neytcheva for her help and support with our project. She

gave us valuable information and set up our office were we worked on this project.

We would also like to thank Sergio Courtade, and Schlumberger, who came up with the

idea to the project. It’s been a really interesting and challenging project, and we have

really enjoyed working together with Schlumberger.

At last we also would like to thank Data-Mats for setting us up with computers, other

hardware and the software that we needed.

Thank you all!

Page 18: Augmented reality - it.uu.se€¦ · What is augmented reality Augmented Reality (AR) is the process of detecting objects in the reality and adding information to that reality. The

18

7. References

Kangdon, Lee. “Augmented Reality in Education and Training.” Tech Trends 56, no. 2 (March

1, 2012): 13–21. doi:10.1007/s11528-012-0559-3.

Mills, Matt. “Image Recognition That Triggers Augmented Reality | Video on TED.com,” July

2012.

http://www.ted.com/talks/matt_mills_image_recognition_that_triggers_augmented_realit

y.html.

“Android.” Accessed January 8, 2013. http://www.android.com/.

“Blender.org - Home.” Accessed January 8, 2013. http://www.blender.org/.

“Eclipse - The Eclipse Foundation Open Source Community Website.” Accessed January 8,

2013. http://www.eclipse.org/.

“Get Started with Kinect - Xbox.com.” Accessed January 8, 2013. http://www.xbox.com/en-

US/Kinect/GetStarted.

“I Dag Får Din Metro-tidning Liv – Metro,” May 29, 2012. http://www.metro.se/nyheter/i-

dag-far-din-metro-tidning-liv/EVHleB!n64olTSHiS1VQ/.

“Inkscape. Draw Freely.” Accessed January 8, 2013. http://inkscape.org/.

“NyARToolkit Project | The NyARToolkit Project Website.” Accessed January 8, 2013.

http://nyatla.jp/nyartoolkit/wp/.

“The Perl Programming Language - Www.perl.org.” Accessed January 8, 2013.

http://www.perl.org/.

“Vuforia | Augmented Reality Unleashed.” Accessed January 8, 2013.

https://www.vuforia.com/.