SmartReality: Integrating the Web into Augmented Reality

SmartReality: Integrating the Web into Augmented Reality

Lyndon Nixon1, Jens Grubert2, Gerhard Reitmayr2, and James Scicluna3 1 STI International, Neubaugasse 10/15, 1070 Vienna, Austria

[email protected] 2 Graz University of Technology, Inffeldgasse 16c, 2. Floor, 8010 Graz, Austria

{grubert, reitmayr}@icg.tugraz.at 3 Seekda GmbH, Grabenweg 68, 6020 Innsbruck, Austria

[email protected]

Abstract. This poster and accompanying demo shows how Semantic Web and Linked Data technologies are incorporated into an Augmented Reality platform in the SmartReality project and form the basis for enhanced Augmented Reality mobile applications in which information and content in the user’s surroundings can be presented in a more meaningful and useful manner. We describe how things of interest are described semantically and linked into the Linked Open Data cloud. Relevant metadata about things is collected and processed in order to identify and retrieve related content and services on the Web. Finally, the us-er sees this information intuitively in their Smart Reality view of the reality around them.

Keywords. Augmented Reality, mobile, Semantic Web, Linked Data, Web ser-vices, Web APIs, Web content, Things of Interest.

1 Introduction

SmartReality is a project which began in October 2010 as a nationally funded project in Austria. The participating partners are STI International, Technical University of Graz, Seekda GmbH and play.fm GmbH. Together they represent expertise and inno-vation in Augmented Reality (AR), semantic technology, Web services and online media. In the context of the project, they explore how people may be able to access relevant information, content and media about things of interest in their vicinity via AR. The AR is enhanced dynamically by the Web of data and the Web of services. This enhancement is made possible by a SmartReality platform which mediates be-tween the client device and the Web-based data and services, focused on using avail-able metadata to select the most appropriate content and services for display to the user. This poster and accompanying demonstrator will present the first results and prototypes of the project. We will explore and demonstrate how semantic technology and Linked Data can be integrated with AR in order to make a user’s reality “smart-er”, based on a scenario with street club posters.

Proceedings of the I-SEMANTICS 2012 Posters & Demonstrations Track, pp. 48-54, 2012.Copyright © 2012 for the individual papers by the papers' authors. Copying permitted onlyfor private and academic purposes. This volume is published and copyrighted by its editors.

48

2 Sm

The valuereflected searchersplored usinformatiGPS posimake useobjects inof pointsfound in selection flexible A

For oucommon likely to btypically Together cess to mour goal around uInternet wtReality u

martRealit

e of the ideas in recent pos

s [2]. A mobise of Linked ion and culturitioning and ae of new data sn the AR focus of interest (Linked Data. and presentat

Augmented Re

ur initial appliand increasin

be early adopinterested in with our part

more than 18 0is to use semas in our currewhich enhancuse case, for e

ty Vision an

of Semantic Wsition papers, ile applicationData and fac

ral artefact meare designed tosources or ada

us. The vision (POIs) interlin

This forms thtion at the AReality experien

Fig. 1. Sm

ication domaingly shared eters of Smart listening to m

tner Play.fm G000 DJ mixesantics and Linent reality to vce our experieexample, we c

nd Scenario

Web and Linknot only fromn, ‘Mobile cuceted browsinetadata. Theseo work in speapt the presenof SmartReal

nked with thehe basis of enR client. Thisnce.

martReality post

in, SmartRealexperience viaReality solutimusic, discovGmbH, whoses and live recnked Data to dvirtual informence of musicconsider a mu

o

ked Data to thm SmartRealitultural heritagng in combinae apps make ucific domainstation of resullity is to levere broad sourcnabling dynams results in a m

ter scenario

lity is focusina the Internetions are youngvering artists ae web site and

cordings to 15dynamically l

mation, contenc in our currenusic-conscious

he AR domainty [1] but alsoge guide’ [3]ation with deuse of (often is, with limitedlting content aage a richer d

ce of concept mic and relevanmore useful, a

g on music. Mt. The user grg professionaland attendingd mobile apps50 000+ usersink referencest and servicesnt reality. In s person wand

n has been o other re-, also ex-

evice GPS imprecise) d ability to around the description t metadata ant content aware and

Music is a roup most ls who are

g concerts. s offer ac-s a month, s to music s from the the Smar-

dering city


49

streets anNow, at band can bsupposes link togetfind out avenue whfor the coposter wievents, ve

3 Sm

A SmartRand servilow (Fig.image recThis identore we tein the Wservices fdisplay in

1 1 http://w

nd seeing the best, if this pebegin to searcthey can iden

ther the informabout the artishere the artist oncert. Howevith related conenues and oth

martRealit

Reality server ices. A simpli. 2). First, thecognition servntifier is linkeerm a “TOI Reb of Data, isfrom the Webn the AR view

www.kooaba.co

ubiquitous sterson is interech on aspects ntify these aspmation they at, find some ais playing, fin

ver, with Smarntent automatier things. Thi

ty impleme

handles the iified illustratioe object in thevice (we use Kd with a desc

Repository”. Ts processed inb. The resulti

w.

Fig. 2. Smart m

treet posters aested in the colike the artist

pects from the are interested audio to listen nd out how to rt Reality theyically overlaidis is illustrated

ntation

interaction beton of the stepe mobile devi

Kooaba1), whiccription of a “The TOI descrn order to seleing content is

tReality workflo

advertising coontent they mat, event or venposter and thin across varito from the arget there, fin

y can view an d over the posd in Figure 1.

tween the clieps taken in Smces camera vch returns an i

“Thing of Inteription, enrichect the most a packaged an

ow illustration

oncerts and cluay have mobilnue. However

hey still must gious Web searrtist, check whd out how to genriched verster references

ent and the WmartReality is iew is identifiidentifier for trest” (TOI) i

hed by links toappropriate cond sent to the

lub nights. le Internet r, this pre-gather and rches, e.g. here is the get tickets sion of the s to artists,

Web of data given be-

fied via an the object. in a datas-o concepts ontent and

client for


50

3.1 An

To ease twe impleports seleregions fodisplay ththe selecfrom an efor play.fentry andconcept).When ednotation fevent poruntime. Tand is pub

nnotation

the process ofemented a Weecting street pfor triggering he content ovted regions, uexisting concefm (artists, evd Ajax-based The user is a

diting is finishfor the poster ster is uploadThe TOI data blished at http

f generating theb based annoposters from the appearancer the poster. users rather septual schemevents, clubs) concept selec

also free to ushed, the annota

and stores it ided to Kooabmodel has be

p://smartreality

Fig. 3. Sm

he initial metaotation tool (Fplay.fm’s im

ce of content Instead of ad

select a Linke. In this casand support

ction (automase a full URI ation tool genin a TOI repoba to make ieen created in y.at/rdf/toi.

martReality anno

adata about ThFig. 3). The tage database.as well as reg

dding a concreedData URI e we use the their addition

atically fillingfrom any oth

nerates a Thinsitory. Additioit possible toRDF/S specif

otation tool.

hings of Interool currently Here, the usgions where tete link to conrepresenting Linked Data i

n by allowingin the full U

her Linked Dang of Interest onally, the im

identify the fically for Sm

rest (TOI), only sup-

ser selects to actually ntent from a concept identifiers

g free text URI of the ata source. (TOI) an-

mage of the poster at

martReality


51

3.2 Server

The server is developed as a set of components which interchange messages to realize the SmartReality functionality expressed in the above workflow (Fig. 2). The platform has been developed in Ruby on Rails and exposes a RESTful API to clients. The re-positories and APIs used by the components to retrieve data into the workflow are separated from the code of the core components so that different storage and remote API solutions (including cloud) could be used as required. After parsing the TOI’s metadata (the TOI being identified via the Kooaba identifier which is included in its description in the repository), it provides an initial response to the client which identi-fies the TOI’s regions of interest for display in the AR view (see below, Fig. 4 left). Two further functional steps are realized on the server to provide the content bundle for the enrichment of the TOI’s regions of interest in the AR view with links to con-tent:

• Linked Data consumption. The (Linked Data) concepts used to annotate the TOI’s regions and extracted from the TOI’s metadata are crawled and further, related concepts extracted as defined in a set of Linked Data crawling rules. The rule syntax makes use of LDPath2. As a result, a local repository of relevant structured metadata about the concepts of interest in the TOI has been created. We use mainly play.fm Linked Data3 in the current demo, while the approach is vocabulary-independent, i.e. any Linked Data could be used in the annotation and supported in this step. For the Linked Data step, we make use of the Linked Media Framework (LMF4), which pro-vides on top of a RDF repository the means to directly cache Linked Data resources and synchronize their metadata with the description on the Web. This means for a new annotation the LMF will automatically use the locally cached resource metadata in available rather than repeatedly retrieve it from the Web which can lead to latency in the platform response.

• Service selection and execution for content retrieval. Based on this local metada-ta, a service description repository is queried. Services or APIs are described in terms of their conceptual inputs and outputs so that, for the given class of a concept in the annotation, appropriate services can be found to provide content for the enrichments of the TOI in the AR view. In the current demo, we use an API provided by play.fm to access audio streams of recordings by artists as well as an API provided by Seekda to link an event to a booking interface for nearby hotels with rooms available on the night of the event. Service execution will require querying the concept descriptions to extract the necessary input values – e.g. for the hotel booking interface, the service API needs the event’s data and its location’s longitude and latitude to be passed in the input request. Likewise, the service response needs to be parsed to return to the Smar-tReality platform the content inks which can be used to enrich the TOI with respect to the original concept. For this, we use the concept of “lowering” and “lifting” in the Linked Services approach [4] where the semantic concept is ‘lowered’ to datatype values for the input request to the service, and the datatype values from the service 2 http://code.google.com/p/ldpath/wiki/PathLanguage 3 http://data.play.fm 4 http://code.google.com/p/lmf/


52

output reMaps of t

The “lJSON.

3.3 Cl

We builtAugmenttered on smartphoinitializatAugmentnow poinright). Thobject in

The Sment on Anselves.

Fig. 4. Smare indicat

the artisbooking auser is at

5 http://ww

sponse are ‘lithe place). lifted” respon

lient

t a client appted Reality in

the physicaone on the phytion, segmentted Reality intnt towards indhe rendering o3D space is bmartReality dndroid smartp

martReality vieted. Middle: onst Fabio Almeria hotel room fort a hotel room b

ww.opensceneg

ifted’ to new s

nses are collec

plication protonterface visualal object via ysical thing of ts containing terface on the

dividual segmeof content thatased on Open

demo will use phones to give

w in the client.

n pressing a conia is associated r after his next booking screen

ev

raph.org

semantic conc

cted and sent

otype for Anlizing relevan

Natural Feaf interest to inirelevant info

e physical objents and obtat is spatially renSceneGraph5

two real clube visitors the e

Left: poster is ntent region, the

with an audio concert from S(date and locat

vent’s descriptio

cepts (e.g. fro

as a content

ndroid smartpnt informationature Trackinitialize a TOI

ormation are hject (Fig.4 be

ain detailed inegistered to se. b event posterexperience of

recognized ande available contstream from plaeekda. Right: fotion is not inputon)

m a Place to

bundle to the

hones. It levn that is spatiang. A user p

query. After shighlighted thlow left). Theformation (Fiegments on th

rs with the insSmartReality

d the regions wient items are shay.fm and a webfollowing the wet, as it is known

images of

e client in

verages an ally regis-points her successful hrough an e user can ig.4 below

he physical

stalled cli-y for them-

ith content hown. Here b link to eb link the

n from the


53

4 Future Work

The SmartReality project has focused on a proof of concept with club event posters and enrichment via LOD from mainly the play.fm database. The infrastructure devel-oped has been deliberately designed to separate distinct data and content sources from the workflow which realizes a SmartReality experience, i.e. the use of other objects as “Things of Interest”, the annotation with other LOD sources, or the linkage to content from other providers for display in the AR view, should be feasible as a configuration issue and not require any changes to the SmartReality platform or client.

5 Acknowledgements

This work has been performed in the Austrian project SmartReality (http://www.smartreality.at) which is funded by the FFG.

6 References

1. Nixon, L., Grubert, J. and Reithmayer, G.: Smart Reality and AR Standards, at the 2nd in-ternational Augmented Reality Standards meeting, Barcelona, Spain, 2011.

2. Reynolds, V., Hausenblas, M., Polleres, A., Hauswirth, M., Hegde, V.: Exploiting Linked Open Data for Mobile Augmented Reality. In: W3C Workshop on Augmented Reality on the Web, Barcelona, Spain, 2010.

3. Van Aart, C., Wielinga, B. and van Hage, W.: Mobile cultural heritage guide: location-aware semantic search. In: Proceedings of the 17th International Conference on Knowledge Engineering and Knowledge Management (EKAW ‘10), Lisbon, Portugal, 2010.

4. Pedrinaci, C., Liu, D., Maleshkova, M., Lambert, D., Kopecky, J., Domingue, J.: iServe: a linked services publishing platform. In: Ontology Repositories and Editors for the Seman-tic Web Workshop, Heraklion, Greece, 2010.


54

Documents

SmartReality: Integrating the Web into Augmented Reality