Storied Navigation

Edward ShenHector BeltranDonyun Kim12/15/06

Story Analogy

　 character A character B

character

typestudent teacher

situationnot knowing where

to go for the future

willing to give

guidance to character

A

emotion /

attitude

anxious, lost, hoping

for help, pessimistic

caring, insightful,

appreciatory, advisory

　 character A character B

character

typeStudent teacher

situation

Thought that she was

clear where to go for

the future

willing to give

suggestion to

character A

emotion /

attitudeConfused, hesitated

encouraging, insightful,

advisory

Idea

Build system that allows users to tell different stories from sequences of video clips with meaningful relationships between clips (using analogies to other stories)

Original System DesignWhat’s the story about? Presidential election

1. Who were the candidates?

2. When was the election?

3. What was the outcome of the election?

4. What were the issues?

Video Corpus

1. How do you feel about him?

2. What has he done in the past?

Challenges for Q/A scenario

How to derive context-relative questions based on provided information

How to derive plausible annotations for a video sequence based on the context of using it

How to design representation for video sequence such that the above can be conducted more easily

Generating Questions

Assume video sequences annotated with Roger Schank’s form of indexing stories (theme,goal,plan,result,lesson)

Questions come from:Generic question templatesCommon sense reasoningStory analogies

Curiosity

Ask questions people tend to be more curious about

Introduce new pieces of information that produce effect of new direction of navigation

Curiosity

Relationship between people Why do people do what they do?

External forces Values/Beliefs Internal desire

New System Design

Different usersUsers that know video corpus very wellUsers that are seeing videos for first time

Help new users create stories by using information first set of users have already entered (system developers)

New system design

1Q1Q2Q3

System Developer

End User

23Q4

Q5

Q5 Q6

44

5Q7Q8Q9

2Why is Mary sad?

Using semantic similarity and common sense44

3

View suggested questions

Q7Q8Q9

555

System Developer1. System Developer types in story

Gustave is one of the dancers. He likes to suggest ideas for the dance. He gets very

frustrated that he can only do what he is told.

2. Sentence/Sentences attached to clips

1 2 3

3. Question set developed for each sequence

1

2

3

Q1Q2Q3

Q1Q2Q3

Q1Q2Q3

4. Generate Q/A sets between video sequences

12Q1

Q2

Q3 Q4

4

Steps 1 – Segment StoriesGustave is one of the dancers.

He gets very frustrated that he is only allowed to do what he is told.

At one point he was crying. This happened during the first week.

The second week was different.

Dancers were encouraged to design their own dance.

Gustave had a wonderful time.

He likes to suggest ideas for the dance.

1. Always include topic sentence2. Try topic sentence with new sentence, if sentence does not return 4/5 same clips as previous sentence/segment, start new segment. Otherwise, add to previous sentence/segment

3/5

4/5

4/5

3/5

4/5

Step 2 – Assign clips

He gets very frustrated that he is only allowed to do what he is told.

At one point he was crying. This happened during the first week.The second week was different.

Dancers were encouraged to design their own dance.

Gustave had a wonderful time.

He likes to suggest ideas for the dance.

1

2

3

Gustave is one of the dancers.

Gustave is one of the dancers.

Gustave is one of the dancers.

Step 3 – Question GenerationGustave is one of the dancers.

He gets very frustrated that he is only allowed to do what he is told.At one point he was crying. This happened during the first week.The second week was different.

Dancers were encouraged to design their own dance.

He likes to suggest ideas for the dance.1

2

3

Gustave is one of the dancers.

Gustave is one of the dancers.

Gustave had a wonderful time.

1. General templates What,when,how,why

How does Gustave dance? What are Gustave’s ideas?

Why did Gustave have a wonderful time?

2. Emotions – why…? Ever feel the opposite?

Why was Gustave frustrated?

Is Gustave ever happy?

3. Conflict – ask for resolution

How was the conflict resolved?

Common Sense