ISMAR 2014 AR Workshop: Designing Location Based Experiences

Location Based Experience Design Workshop

Mark Melnykowycz

ISMAR 2014 Sept. 9th 2014

1

Mark Melnykowycz [email protected]

Story Structure

Story Location

User Experience

A/MR Technology

Location

Hero’s Journey Changing

structures

Books, movies, ebooks, games

IxD / UXFundamentals

of IxD

Prototyping

Consumption Patterns

User Journey

Game Mechanics Metaio

SDK

Tracking / Positioning

AR / MR Fundamental differences

Technology

Teaching Users How

to Play

World Building

Game Space Rules

2014 ISMAR Workshop:Location based experience design

Lost In RealityMobile storytelling

Ghosts of VeniceMixed-reality film

2

mailto:[email protected]

Mark Melnykowycz [email protected] 3

Location



Generic Location App Use Case

User goes to a location

Content or interaction is

triggered



Designing Location Experiences

Why is the person in that

location?

How are they interacting with the location?

Why are they motivated to engage in this

experience?

How are experiences

planned?

What technologies are available?



Key Physical Elements

LocationPosition /

Orientation

InteractionContent



User motivations / immersion

Sketching out user flows / prototyping / product definition

Building interaction / defining app world

Technology trends

Communication patterns



IoT/M2M Devices

Device Fragmentation Evolving Communication

Tech and Society Trends



Technology and Device Trends



50 Billion!By 2020*!

Quantified User!

Quantified Self!

IoT!

http://share.cisco.com/internet-of-things.html

Connected City

Location Interaction

Large connected ecosystem

Connected Device Evolution


http://share.cisco.com/internet-of-things.html


Location

Time

Heart Rate

Sensor Fusion User State

Quantified User



Defined UX in Closed SystemsD

evic

eIn

tera

ctio

n

Interaction and consumption patterns are learned making the transition to new devices easy



Ubiquitous C

omputing

Defined UX Space

Distributed UX

Fragmented UX Patterns



How are users trained to interact with locations?

Maps

Location as a Device Experience



Determine orientation in location using navigation aids

Maps

Identify physical anchor points to define position

Use technology to identify information

around you

Future actions based on understanding where you are

and what you can explore



Communication Trends



Short Evolution of Communication

Face to Face

Web

Email Static HTML Blogging Instant Messaging

Social Networks

Location ServicesWriting

PrintingCopier

TelephoneMobile SMS

MMS

Fragmented Lines!Push/Pull!

Transmedia



Content Container Consumption Velocity

The User

User Experience Design

18

Consumption Concepts



Content

The User

Information transferred between containers and consumed by people

Consuming Media



Physical Container

The User

A real, tangible thing used to hold, and through which content can be consumed

Has been designed, has structure and fills up physical space



The User

A virtual thing used to hold, and through which content can be consumed

Has structure, fills up space and can scale

Defined through graphic design and UI engineering

Virtual Container



Consumption

The User

Understand/Retain information Make it part of your knowledge base How fast is this?

Velocity

Content

Vocabulary



Landing Page GridContent Home

Print Web Mobile

Pull out bits of information

Scan - look for anchors

Save an idea and move on

Linear progression

Consumption Patterns



Push/Pull Consumption Patterns

24



Digital Consumption

App/Webpage Social Media Video Augmented Reality

Audience

CuratorPush/Pull



SnowFall

New York Times / http://www.nytimes.com/projects/2012/snow-fall/

Scroll page, easy to pull out bits of the story and be immersed in graphics, story, characters.


http://www.nytimes.com/projects/2012/snow-fall/


Text, animations, videos, different themes, consistent narrative

Pull-based consumption and reading of content



Parallax, easy to explore Pull consumption Easy to design/prototype

https://www.scrollkit.com


https://www.scrollkit.com


Easy to integrate, drag/drop images, embedded media (video/audio/iframes)

Custom URL or publish on scrollkit.com



Social Media Curation

More powerful than Tumblr, connect to and search social media channels, present content in sequence

Flow/steam Slideshow



Connect to social media channels

Arrange elements to group together Add text and links



Publish as landing page on BrickFlow, share link to social media channels, remix

Embed ex te r na l l y as iframe element, view as slideshow experience



Connect to and search social media channelsAdd content, text, etc.



Mixed Video/Interaction

Non-linear video navigation Clickable elements Easy assembly

Combine multi-media elements View in video/slide format Videos, audio, animations, text, etc.



http://www.raptmedia.com/product

Add clickable elements to videos, seamlessly jump from one video to another.

Allow choices for jumping between storylines/elements


http://www.raptmedia.com/product


Interactive layer allows clicking/hyperlinks

Different video clips can be linked together

User can make decisions at plot points

Easy to create alternative story lines or allow exploration



Connect to channels, ability to upload

Arrange elements Share

www.zeega.com


http://www.zeega.com


Designing Experiences



Empathy Define Ideation Prototype Test

Discovery High-Level Solution Product Solution

Product/Experience Design PhasesUser Needs

Focu

s / F

lare

App Prototype

39

Experience Planning Interactivity Prototyping

Story/Location Technology



Story Consumption Container

What is the story, the world it takes place in, the character, etc.

How will this story world be consumed by the audience or users?

How will the consumption be realized, physically/virtually designed, etc?

Know your User!Wireframe/prototyping Webpage/App Video/ebook

Know your Audience!Story boarding Pre-Visualization

User Experience/Interaction



Creative Product

Creates stories, art, movies, etc.

Builds physical/virtual products

The UsersThe Audience


Find your audience

Prototype/Test/Explore



How will the consumption be realized, physically/virtually designed, etc?

Wireframe/prototyping Webpage

Video ebook

ContainerConsumption


User Journey

Paper/Digital Mockup

Interactive Prototype

Final Experience



How does the user get into your story and engage in your experience?

User Journey

User sees meme via Facebook/Tumblr

Searches for Zurich Dada

Game on Google

Starts reading/learning about the Dada ghost

in Zurich

Need to have good SEO base on landing page

Need fast consumption on desktop/mobile

Parallax page design

User is push/pull fragmented reader



Design representation of your app/container to communicate and test the user experience

Paper/Digital Mockup/Prototype

Communicate between design / development

Communicate with potential partners / investors / users

Testing tool for fast user feedback before development

Pape

rM

ocku

pD

emo

App



Web/App Prototyping

Keynote Powerpoint OmniGraffle GUIToolkits

HotGloo JetStrap

AppCooker BluePrint iMockups

Stencils

Visio Balsamiq

Axure iRise MX

Photoshop Illustrator

iPad

OnlineDesktop



Storytelling

46



Narrative

•Story, a recounting of a sequence of events* •An account of imaginary or real people and events

told for entertainment: "an adventure story"**

•A narrative (or story) is any account that presents connected events***

•A spoken or written account of connected events; a story: "a bare narrative of the details"**

*Wikipedia **Google dictionary ***Oxford English Dictionary (online)

Story


http://en.wikipedia.org/wiki/Sequence_of_events

http://en.wikipedia.org/wiki/Report


Emotions

Know

ledg

eStory

Society



...storytelling is a vehicle used to combine emotions and information

...and communicate those feelings and information in a structure which gives us comfort or instills feelings of fear,

optimism, etc.

Shared emotions help form the fabric of society



Emotions

Memories

Connections

Structure



Storytelling gives emotional context to our linear lives

51



Birth Death

Physical time is linear Emotional memory is non-linear



Connect current events to good feelings from the past

Play out emotions we can’t explore in reality

Hope in the future and our confidence to meet challenges

Story



...makes me feel good about my chances of surviving a dramatic terrorist attack

...statistically it will probably never happen, but I’m sort of mentally prepared if it does

Die Hard, 20th Century Fox

Die Hard



What is traditional story structure? Does it conflict with our fragmented communication

patterns?

55

Story Structure


Many different models of story structure for different media

56


Linear Story Basics



Villain Conflict

Generic Story Ingredients



TransformationChallenge

Over-coming extreme odds...to succeed in the end…



http://www.musik-therapie.at/PederHill/Structure&Plot.htm

60

http://wandertones.deviantart.com/art/Standard-Story-Structure-178276496

Rising drama leading to an apex with essential plot points and local challenges/resolutions

Linear Story Structure


http://www.musik-therapie.at/PederHill/Structure&Plot.htm

http://wandertones.deviantart.com/art/Standard-Story-Structure-178276496


“Campbell explores the theory that important

myths from around the world which have

survived for thousands of years all share a

fundamental structure, which Campbell called

the monomyth.”

https://en.wikipedia.org/wiki/The_Hero_with_a_Thousand_Faces

Joseph Campbell


https://en.wikipedia.org/wiki/Monomyth

https://en.wikipedia.org/wiki/The_Hero_with_a_Thousand_Faces


THE HERO IS INTRODUCED IN HIS ORDINARY WORLD THE CALL TO ADVENTURE

THE HERO IS RELUCTANT AT FIRST THE HERO IS ENCOURAGED BY THE WISE OLD MAN OR WOMAN

THE HERO PASSES THE FIRST THRESHOLD THE HERO ENCOUNTERS TESTS AND HELPERS

THE HERO REACHES THE INNERMOST CAVE THE HERO ENDURES THE SUPREME ORDEAL

THE HERO SIEZES THE SWORD THE ROAD BACK RESURRECTION

RETURN WITH THE ELIXIR

http://www.cs.uu.nl/docs/vakken/b2go/docs/hero_journey.pdf

Joseph Campbell


http://www.cs.uu.nl/docs/vakken/b2go/docs/hero_journey.pdf


The Hero’s Journey repeated in countless media…



Suspension of disbelief or willing suspension of disbelief is a term coined in 1817 by the poet

and aesthetic philosopher Samuel Taylor Coleridge, who suggested that if a writer could infuse a "human interest and a semblance of truth" into a fantastic tale, the reader would

suspend judgement concerning the implausibility of the narrative.

http://en.wikipedia.org/wiki/Suspension_of_disbelief

Suspension of Disbelief



Why is the Hero’s Journey structure so popular in movies and media?

Fairy Tales, myths, Star Wars, The Matrix, etc.

Do we like it because we expect it and are conditioned for it?

Is it inherent in our minds and DNA?

Does it hold us back in life? Must we be chosen to do great things by someone else?



Games and Story



Confront a challenge

Understand your progress

Play

Interactive with the world

Interact with other players

Actions have consequences

Conquer small quests

Games



Story

Players

Game world / field of play

Conflict / ChallengeMechanics

Resources

Game Elements



Players understand the rules

Players overcome challenges

Players understand their place in the

context of the game

Points System

Levels

Time

Reason for the experience to exist

Know how to interact with the experience

Game Mechanics


Game Enjoyment

70

The StoryThe Challenge The Game Play

http

://en

.wik

iped

ia.o

rg/w

iki/D

oom

_(19

93_v

ideo

_gam

e)

http

://z2

.com

/gam

e/m

etal

stor

m-o

nlin

e/

http

://en

.wik

iped

ia.o

rg/w

iki/P

uzzl

e_Bo

bble

http

://en

.wik

iped

ia.o

rg/w

iki/W

orld

_of_

War

craf

t


World Building



Character evolves

Character overcomes challenges

Character presented with challenge

Generic HeroGeneric Threat

Finds courage... Triumphs!

A Generic Story



Character evolves

A character evolves/changes due to conflict inherent in the story

Relationships

Environment

Character Change Through Conflict



Is the user a participant or an outside viewer in a location game or experience?

Who is the Main Character?



Story Development

Character Driven World Driven

How will the world affect character development?

Generic environment, used as a setting for character interaction to take place.

The story of a character in a given environment, which forces action / conflict

...become a different person

...character traits stay the same



Contemporary super hero

Traditional action hero

Die Hard Arnold Schwarzenegger

Ironman, Thor, Spiderman, etc.

Watchmen Dark Night

Man of Steel X-men

Simple dialogue, hero never dies, always

triumphs

Question their actions and place in the story, supporting characters

die, like real life.

Good vs. Evil Good? vs. Grey Zone



Location Time Physics

Where is the world? Is it a city, a new planet, a coffee house?

Take a character from another time

or use a world from a different

time?

If it’s a new world, does it respect the physics of Earth as

we understand them?

World


78

X | Media | Lab Switzerland 2013: Transmedia Alex McDowell – Transmedia’s Renaissance Man: Narrative Designer for Minority Report; Fight Club; Man of Steel; Associate Professor, USC – School of Cinematic Arts; Visiting Artist, MIT Media Lab (Hollywood)

79


80



How does a character move around the world?

Character Journey

Character becomes sick Must go to

hospital

How would she die? Never invented

gasoline, all hydrogen

Transportation logistics



Physical Virtual World Building / Prototyping

Unity 3D!• Multi-platform Game

development • Terrain building • Visualization • Integration of buildings,

people, etc.



Sketch out worlds

The Hobbit Maphttp://tolkiengateway.net/wiki/File:Christopher_Tolkien_-_Map_of_Wilderland.jpg

Physical story worlds define logic of story progression



Paint terrain and topography features !Add textures for grass, rock, water, etc. !Add buildings, people, tanks, etc.


Mark Melnykowycz [email protected] 85Load new models

Paint on topography

Load textures

Use different brushes

Unity User Interface



Augmented and Mixed Reality Technologies

86



Generic AR Experience

Marker Marker-less Point-cloud Edge-based

App tracks feature in reality Content positioningUser engaged with

app

Virtual content

87



AR Software

Creator / StudioSDKs

AR Browsers

88



Depth Cameras

Desktop

LaptopMobile

Stru

ctur

e.io

Proj

ect T

ango


90

Ghosts of Venice

ghostsofvenice.it

http://ghostsofvenice.it


Augmented Reality Ghost Stories of Venice

ghostsofvenice.it



92

Marco Gatti

Film Maker / Director

Sara Mautino

AR Glass Expert

Mark Melnykowycz

UX / AR coding

Visual Effects

Story Tech

Production

ghostsofvenice.it


93

Place Ghosts in the Environment

ghostsofvenice.it

Take users on a ghost story through Venice, mixing story elements and 3D animations into the local environment


Film vs. App

94

C o n s u m e d b y watching/listening while stationary

L o c a t i o n a n d con ten t can be mixed to provide more immers ive experience

ghostsofvenice.it


Augmented Reality

95

Technology used to slightly modify the perception of the environment of the user

GPS location Media overlay

Image recognition

Env i ronment or user triggers new actions in the software

Feature tracking

Image Video Audio Text1

2

3

4

ghostsofvenice.it


Story - Technology Development

96


Transmedia Narrative

Online Story Hard Copy

Mobile App

Educate users of the background story before starting

Give users something tangible, objects from t h e g h o s t s , t h e i r letters, etc.

Main point of contact between user, story, and location

ghostsofvenice.it97




Story Development

App Design

App Development

Media Production

How do all of these key elements come together in the transmedia

development process?ghostsofvenice.it98




Story/App Workflow

Story App Design

App Development

User Experience User Interaction

Background stories of ghostsHow stories are

delivered

Implement UX/UI in App Development

Test/Iterate

Prototype

Production App

ghostsofvenice.it99




Story/Media CreationStory Locations Ghosts

Choose locations Choose trigger

elements

Background story for each ghost

Visual Design Casting Filming

Post-production

Ready to distribute on

mobile device

Mobile Media

Test App

Code into data model for

viewing/testing

Production App

ghostsofvenice.it100




Sets the stage, gives tangibility to the story experience, being in the same physical place as the ghosts

App is used to view the ghosts, to connect with the “being” the user has learned about

The impact of seeing the ghosts comes from the emotional connection to the story behind them

How can the story and location be interactive with the participants? This increases the emotional connection to the experience

Transmedia LandscapeA transmedia project integrates a larger narrative across different distribution

channels, we need to develop a plan for how this would be realized

Location

Mobile AppStory Channels

Interaction

Real book/diary of ghosts Wikipedia page each individual Artifacts of ghosts in the city/museums...what else? Establish emotional connection and empathy to ghost characters ghostsofvenice.it101



Augmented Reality Technology Needs

102


User Walks Through Area

Location Aware

Image recognition

Event Playing

User sees they are in the area where

a trigger is

User searches for physical

trigger

Location triggers haptic/vibration on

device

Content plays through device

ghostsofvenice.it

User Flow



ghostsofvenice.it 104

Alpha Mask

Transparent background

Overlay in video feed


Image Recognition for Location?

Software recognizes specific features of the environment

Image pre-programmed in to the dev ice o r queried from a server

Recognition may not be automatic, depends on hardware speed



Install Junaio

Access GOV channel

View content on image trigger

106ghostsofvenice.it


Changing light conditions will impact

tracking ability

Image target not reliable enough for spatial location use

107ghostsofvenice.it


Feature RecognitionSoftware recognizes specific features of the environment

P o i n t - c l o u d m a p created to aid in feature recognition

CAD or edge-based tracking can be tested as well



Metaio Edge-based Tracking

CAD model overlay on environment

Better in low-light tracking scenarios

Where do I get building CAD?

109

www.metaio.com

https://dev.metaio.com/sdk/documentation/tracking-config/optical-tracking/cad-model/

http://www.metaio.com

App Development Flow

110

Layar/Junaio Metaio SDK Unity Beta App

Test the concept of playing audio o r v i d e o fi l e s based on GPS location with AR Browser

Initial Test Functional Prototype Initial Release

I m p l e m e n t v i d e o overlay concept in a na t i ve iOS /Andro id application, testing with users to understand the UX better

With a developed s t o r y l i n e / p l o t , producing the full m i x e d r e a l i t y exper ience and broad testing with users

ghostsofvenice.it



Use 3D scan of objects to create CAD for model

positioning

Experiment with 3D mapping for

model positioning

Mobile 3D Scanning



Ghosts as model scans

Match to environmental

features

Combine Person and Environment Models


Marco Gatti

Film Maker / Director

Sara Mautino

AR Glass Expert

Mark Melnykowycz

UX / AR Development

Visual Effects

Team

www.ghostsofvenice.it113

http://www.ghostsofvenice.it

Lost In Reality

114

Lost In Reality

Mobile App Web App

Search Consume Create Share

Create Edit

Storytelling Social Network

115lostinreality.net [email protected]


Francisco Campelo José Rodrigues Mark Melnykowycz Ismail Cimen

App DeveloperPartner and

Story Development

Project Lead UX/UI

Design UX/UI

Team

Lisbon, Portugal Zurich, Switzerland


https://plus.google.com/107348059563772986779


Emo8onsLoca8ons

People

How can we use story to add to the information context of locations?



Stories saved

For Storytellers

Authors can write loca8on stories on the web app

Users find stories based on loca8on

and tags

Users can write stories on the mobile app

Story server



Tourists have more unique experiences in

new ci8es

For Authors

Authors can write loca8on stories on the web app

Users can download stories

Locals have new perspec8ves on

their city

Loca8on used as integral element of

stories



Users search and access loca8on-‐stories on

demand with mobile app

For Readers

Story elements triggered based on loca8on in

ci8es

Context of stories gives more value to loca8ons



Location Stories vs. Social Media

Location Services

Context?

Social Networks

Location stories give structure and context to information and combats the social media noise

Social media evolved from blogging as faster way to publish ideas

Social media updates are only useful for a short time frame

“Facebook/Twitter shows people by who they are, what they have done and said”

Location services give an idea of what a person can do in a place (irregardless of when)

“Locat ion s tor ies show people the experience they’ve had”

“Location stories let people follow experiences and have their own”



Users can create, search, and follow location stories

Focus on location-based storytelling

Emotional engagement to location

Primary Goals of LIR

Smart phone application

Dashboard Story Library

Local Story Search Story Element

Story elements delivered to user at GPS location

(triggered)

122lostinreality.net

Mobile App Flow

[email protected]


Maker Web App

lostinreality.net [email protected] 123


Maker Web App

lostinreality.net [email protected] 124


125

User can follow different stories in a city as they

walk around.

lostinreality.net [email protected]



User motivations / immersion

Sketching out user flows / prototyping / product definition

Building interaction / defining app world

Technology trends

Communication patterns

126
