Designing interactive narratives for mobile augmented reality

  • Published on

  • View

  • Download

Embed Size (px)


  • Cluster ComputDOI 10.1007/s10586-014-0354-3

    Designing interactive narratives for mobile augmented reality

    Yanghee Nam

    Received: 15 November 2013 / Revised: 22 January 2014 / Accepted: 31 January 2014 Springer Science+Business Media New York 2014

    Abstract An increasingnumber of personalmobile devicessuch as smartphone are playing an important role in our dailylives. Among constituing technologies for such pervasivecomputing environment, mobile augmented reality (mobileAR) is a technique that extends physical world with virtualobjects or information in truly mobile settings. That is, awayfrom the carefully conditioned environments of research lab-oratories and special purpose work areas, general people canengage in location aware or physical object related contentusing their portable devices. Deploying attractive mobile ARservices, however, has been regarded quite difficult becausecomputer-vision based relevant techniques are very com-plex. Because of that, discussions on the practical possi-bilities of AR have tended to stay in location-aware infor-mation delivery service or as a consumption platform ofdeveloper-supplied content. Against those popular researchtrends pursuing technical advances ofAR, this paper attemptsto explore the potential of AR as a general peoples cre-ativemedium. It can provide a good storytelling environmentbecause a specific location or real-world objects can easilybecome story subject matters and stimulate peoples imagi-nation. There exist, however, some barriers preventing usersto actively participate in onsite creative activity construct-ing their own story. In mobile situation, it could be cumber-some to create a narrative at site, requiring to author somesequences of events. Therefore, the careful design regard-ing such situational characteristics of mobile AR is essentialfor realizing real-time mobile interactive AR narrative. Thispaper thus presented design factors for interactivemobile ARstorytelling systems, and applied narrative theory to design

    Y. Nam (B)Division of Digital Media, Ewha Womans University,SK-Telecom Building #402, Seoul, Koreae-mail:

    and explore actual possibilities of interactivity levels achiev-able using mobile AR medium. Technical difficulties areruled out as possible so that the design could be focused onreal-time onsite story creation activity. Proposed design con-cepts are developed as three kinds of prototypes, each of themreflecting different level of narrative interactivity. They arealso deployed in public exhibition and the suggested designfactors are evaluated in terms of user experience and inter-views. The result of evaluation shows that even simple ARinteractive narrative setup can have strong power to allowusers playful experience of in-situ narrative creation.

    Keywords Interactive narrative Digital storytelling Augmented reality Mobile media

    1 Introduction

    Augmented reality (AR) is a technique that extends physi-cal world with virtual objects or information. It can providea good storytelling environment because a specific locationor real-world objects can stimulate peoples imagination andcontribute to formation of a story. In particular, when weregard users participatory generation of AR content, it hasgreat potential to produce user-created stories associatedwithreal things or locations. As the supporting technology, more-over, there has been recent growth onmobile platform includ-ing network infrastructure [1], location-aware mechanisms[2], and storage platforms for pervasive computing [3]. Theseadvances in mobile network, storage and positioning mech-anisms also greatly encourage practical and commercial useof AR, potentially as users content generation and sharingplatform [4].

    However, previous works on AR have mainly stayed indiscussions on development of computer vision or graph-


  • Cluster Comput

    ics techniques for visual composition of real and virtualscenes and their correct alignment. There also have beena few researches having regarded AR in relation to sto-rytelling, most of them, however, have actually presentedAR as an interactive story simulation environment with pre-determined stories created by developer or designers of thesystem in advance.

    Comparatively, this article pays attention toARas an inter-active storytelling medium, that is, a creative space for gen-erating a users own narrative. Based on a literary theoryon digital narrative texts, the design aspects enabling usersin-situ augmented storytelling is discussed in this paper.Accordingly, prototype AR systems are implemented reflect-ing various interactivity levels of narrative and deployed in aninternational art festival for a month and let visitors interactwith them during the exhibition period. Based on the evalu-ation lasted for a month and also thereafter have shown thateven simple interactiveAR setup for user-generated narrativecould have strong power to allow users playful experience ofin-situ storytelling.

    The rest of this paper is organized as follows. Section 2summarizes the previous research on AR content authoringwith comparison to the proposed goal here. Section 3 is themain part introducing design considerations and the catego-rization of narrative interactivity, and based on those aspectsthree kinds of design goals are suggested. Section 4 presentsthe actual implementation of the proposed types of interactiveAR narrative and the results of empirical studies are brieflymentioned. Finally, conclusions are given in Sect. 5.

    2 Related work

    Existing works on AR content authoring have mainly beenclassified into one of the three categories. The lowest levelapproaches among those categories provide complex toolk-its, libraries or scripting frameworks for application devel-opers. Software toolkits such as ARToolkit [5], and morerecent libraries such as Layar SDK [6] or Vuforia [7], andmoreover HTML/KML based AR scripting [8] are examplesin this category. On the other hand, the higher level authoringframeworks have mainly included graphical user interfacesthat enable content designers to build AR applications with-out coding or scripting. Non-developers can build AR sceneswithout knowing how to code using, for instance, BuildARtool [9]. The aforementioned authoring systems in both cate-gories, however, have been built for the usage on the desktopor the web.

    More recently, the third type authoring approaches haveconcerned allowing the user in-situ authoring of AR con-tent. In-situ modeling or authoring means a creation processwhereby content or scenes are constructed using AR systemwithin which they will be used [10].

    Since the purpose of our research is to provide interac-tive creation of users own narrative in AR environments, wefurther investigated existing works on real-time in-situ con-tent creation which are in real time applicable to end userscarrying their mobile devices.

    So far, exiting research on in-situ AR authoring havemostly focused on letting users define visual shape of vir-tual objects or mixed scene in a simple manner. Hengelswork [10], for instance, introduced in-situ geometric model-ing from real time video and inserting it back into the samelive video stream. Hagbi et al. [11] presented in-place sketch-ing method for AR content that allows users sketching gamearena using visual rules. Lee et al. [12] attempted tangibleauthoring of 3D virtual scenes using marker, RFID, cameraand tangible miniatures, while Langlotz et al. [13] suggestedreal time composition of video content in mobile augmentedreality. All these in-situ modeling or sketching approachesare targeting intermediate role of users creating visual aspectof AR content to serve it for other end users. Also, user inter-actions are not relevant with creating narrative but relatedonly to visual augmentation of content with regard to physi-cal site.

    Only fewworks have concerned end-users involvement ofin-situ AR content creation. Zhu et al. [14] attempted pro-viding an authorable AR system to assist the maintenancetechnicians by enabling them to superimpose the mainte-nance instructions virtually on the real equipment. Wetzel etal. [15] presented TidyCityAR gamewhere users can add in-situ content in addition to web-based pre-built content. Thisallowed users scavenger hunt type mission creation basedon simple rules to construct place-oriented riddles. In Wet-zels work, it seemed that users could have augmented nar-rative experience while having power to create some riddlesin-situ. While both Zhu and Wetzels approaches were thesame in the aspect that they all utilized partly pre-developedAR environment and users could add some variation to that.However, either work has only remained a case study with-out further discussion on the types and possibilities of usersin-situ augmented narrative creation in broader sense.

    When we ponder on why existing works on AR authoringhardly considered users narrative-level creativity, we couldfind several reasons. First, real time object tracking and reg-istration techniques for AR have remained so far most urgentand difficult problems in AR scene rendering. In-situ author-ing of 3D objects is still also a challenge. However, usersexperience and motivation towards the use of digital sys-tems are not encouraged only by visual aspects but also bynarrative and interactive experience given to the users [16].Secondly, in-situ content creation process should be simpleenough for mobile users to carry out creative task while stop-ping at or hovering around some physical location, moreoverin standing or walking situation. It is therefore not feasibleto ask users to do complex composition of content in-situ.


  • Cluster Comput

    Table 1 Comparison ofproposed work against relatedworks

    Comparable aspects Related works

    Onsite narrativeauthoring capability

    Onsite authoringcomplexity

    Offline authoringeffort

    Support for gen-eral guidelines

    [10,13] Out of concern(focus: visualshape modeling)

    Complex(segmentingvideo, etc.)

    Required General (forshape creation)

    [14] Limited (totechniciansillustration)

    Simple (text +trajectorysketch)

    Required General (fortechnicalillustration)

    [15] Limited (byfilling-up riddletemplate)

    Simple Required A case study

    Proposed work Full narrative Simple Mostly none General (fornarrativestructure)

    Table 1 summarizes a comparison of the proposed workagainst the aforementioned related works. Qualitative nov-elty is shown in the aspect that while most previous workshaving focused on visual content authoring, this article aimsto support users in-situ narrative creativity. Our distinctiveapproach is that the proposed designs of augmented narra-tive systems are based on narratology. Accordingly, severaltypes of users story creation involvement are differentiatedand prototyped in terms of user-narrative interaction levels.

    3 System design

    This section describes our consideration on design factorsand goals, and then the properties of resulting designs forusers participatory narrative in AR.

    3.1 Design factors

    User activities for in-situ narrative creation are prone to tire-some, in particular in users standing and moving situationwhile carrying their smartphones. Also, composing a storyinstantly might cause cognitive pressure of users and theymight thus avoid using the system. With the above in mind,we set up a set of design factors that guide our design processfor interactive AR narrative systems.

    3.1.1 Factor 1: Level of interactive narratives

    Many literary scholars havementioned there exist several lev-els of users narrative involvement [1618]. From branchingnarratives such as hypertext to highly interactive computergames, users influence levels on resulting digital story canvary diversely. Exploring such diversity will be consideredin our designs of AR narrative interactivity.

    3.1.2 Factor 2: Affordable interface

    Interface affordance also matters to effectively induce usersvoluntary participation for plot creation. While the number

    of meaningless or subsidiary actions such as menu selectionshould beminimized or reduced, users should be able to jumpdirectly into the storytelling process. The degree of freedomgiven to the users should be carefully be constrained whilestill allowing user experience of real time plot generation.

    3.1.3 Factor 3: Flow between physical and digital world

    The design is driven by the aim to support a fluid interactionacross the physical and the digital domains, which constitutethe digitally augmented space. Afluid interaction perspectivetreats the digitally augmented space as a whole and aims ata continuous experience. To reduce possible discontinuitiesthat can arise from cross-space interaction, the approach heretries to choose adequate interaction metaphors that supportcross-space flow and also integrated narrative.

    3.1.4 Factor 4: Joy and playfulness

    Besides exerting physical and intellectual activity, it wouldbe nice if the system could meet peoples desire for joy andplayfulness. Joy, fun and gameplay is inherent in both chil-dren and adults. The designs proposed here try to considera discovery by chance and game-like playful events. Sen-suous modalities for interaction is also considered to helpemotional and joyful AR experience.

    3.2 Layers of narrative interactivity

    As mentioned in our design factor 1, several layers of possi-ble user interactivity in digital narrative text have been dis-cussed in literary context. This paper mainly refers a promi-nent literary scholar M. L. Ryans digital narrative theory[17] that adapted Aarseths user functions in cybertext [18].As if peeling the onion, there exist the outer shallow layerwhich allows only of users exploratory functions and theinner layer for ontology-level user involvement. That is, onthe outer layer, interactivity affects only the presentation ofthe story, and the story itself pre-exists to the running of


  • Cluster Comput

    the system. On the inner layer, however, the story is createddynamically through the interaction between the user and thesystem.

    This article takes into account these kinds of users mul-tiple level engagement through different layers of interac-tivity in augmented narratives. To do that, this paper beginswith characterizing both layers with regard to mobile ARcontext.

    3.2.1 Outer layer: exploratory AR narrative

    First, in the outer layer of user interactivity in digital text, theuser is able to navigate the database of the pre-determinedstory segments deciding the order to take, but she has noimpact on the virtual story world itself. Therefore, the sim-ple interactionmechanism is enough for users only to controlnarrative discourse and the presentation of the story. Whilestory refers to the actual chronology of events in a narra-tive, discourse refers to the manipulation of that story inits presentation. Figure 1a shows the concept of this layersnarrative. The important thing in designing this layers inter-activity is to provide users some way of controlling the orderor part of the pre-defined story, or determining whether aug-menting it or not...


View more >