Human Computation & ESP Game

Human Computation & ESP Game

2008/12/19

Presenter: Lin, Sin-Yan

1

References

• Ahn, L. V. & Dabbish, L. (2004) Labeling images with a computer game, paper presented at the Proceedings of the SIGCHI conference on Human factors in computing systems.

• Ahn, L. V., Liu, R. & Blum, M. (2006) Peekaboom: A game for locating objects in images, paper presented at the Proceedings of the SIGCHI conference on Human Factors in computing systems.

• Ahn, L. V. (2007) Human computation, paper presented at the Proceedings of the 4th international conference on Knowledge capture.

• Ahn, L. V. & Dabbish, L. (2008) Designing games with a purpose. Communications of the ACM, 51(8), 58-67.

• Chen, L.-J., Wang, B.-C., Chen, C.-Y., King, I. & Lee, J. (2008) An analytical study of puzzle selection strategies for the esp game. Report for Institute of Information Science, Academia Sinica (Taipei, Taiwan, ROC).

• Weber, I., Robertson, S. & Vojnovi´C, M. (2008) Rethinking the esp game. Report for Microsoft Research, Microsoft Corporation.

2

Outline

• Human Computation

• Human Computation System and Application

• Human Computation in E-Learning

• The ESP Game

• Mathematical Model in ESP Game

• System Gain

• Image selection

• Automatic playing the ESP Game

• Problems of ESP Game

• Conclusion and Comments

3

Human Computation

• Concepts are started at 2001: The Open Mind Initiative

• Some task are trivial for humans, but continue to challenge computer algorithms

• Treat human brains as processors in a distributed system, each can perform a small part of a massive computation

• But humans require some incentive to become part of a collective computation

• By Games (Ahn and Dabbish, 2004) (Ahn and Dabbish, 2008)

Ahn, L. V. & Dabbish, L. (2004) Labeling images with a computer game, paper presented at the Proceedings of the SIGCHI conference on Human factors in computing systemsAhn, L. V. & Dabbish, L. (2008) Designing games with a purpose. Communications of the ACM, 51(8), 58-67.

4

Human Computation System- Peekaboom

5Ahn, L. V., Liu, R. & Blum, M. (2006) Peekaboom: A game for locating objects in images, paper presented at the Proceedings of the SIGCHI conference on Human Factors in computing systems.

Human Computation System- PhotoShoot (1/2)

6Cheng, K.-Y. (2006). PhotoShoot: A Web-Game for User Assisted ROI Labeling. Information Management. Taipei, Taiwan, National Taiwan University. Master.

Human Computation System- PhotoShoot (2/2)

7

Human Computation System- ImageHunter

8Chen, L.-H. (2008). Accuracy Analysis in Productivity Games: A Case Study on Landmark Annotation. Department of Computer Science and information Engineering. Taipei, Taiwan, National Taiwan University. Master: 62.

Human Computation System- PhotoSlap

9Chang, T.-H. (2007). Productivity Game Design and Gameplay Analysis: Play and Annotate with PhotoSlap. Department of Computer Science and Information Engineering. Taipei, Taiwan, National Taiwan University. Master: 52.

Human Computation System- reCAPTCHA

10

• Proposed by Ahn (2008)

• Gathering the characters that optical character recognition (OCR) software has been unable to read

Human Computation in E-Learning

• 人力計算具有利用人類直覺的特性• 動機

• 傳統英文閱讀學習及評量無法達成的地方• 目的

• 用 TAGs表達英文閱讀學習過程中最直覺的想法• 方法

• Human Computation and Game Design

11

Human Computation in E-Learning- why tag

• Free, uncontrolled vocabulary

• Social relations of TAGs

12

photophoto

artart

desigdesignn

imagimagee

colorcolor

taggintaggingg

researcresearchh

Agenda

• Human Computation

• Human Computation System and Application

• Human Computation in E-Learning

• The ESP Game

• Mathematical Model in ESP Game

• System Gain

• Image selection

• Automatic playing the ESP Game

• Problems of ESP Game

• Conclusion and Comments

13

The ESP Game

• The first human computation system to take advantage of people’s desire to be entertained and provide useful metadata

Ahn, L. V. & Dabbish, L. (2004) Labeling images with a computer game, paper presented at the Proceedings of the SIGCHI conference on Human factors in computing systems

Randomly selected image

Input Keywords Input Keywords

Keyword Match

14

Mathematic Model in ESP Game

• There are many parameters in ESP Game

• We can use those parameters to modeling many things that we want to know

• Ex.

• Number of players

• Number of labels per game

• Number of guesses to reach a consensus

• Good and bad word assumption

• (Human Computation) System Gain

• Image selection

• User scoring model

15

Mathematic Model in ESP Game- System Gain Modeling (1/2)

• The performance of human computation system

• The purpose of ESP Game tries to collect images’ tags

• Two aspect of thinks

• Take as many tags as possible for each image

• Playing the maximum number of image

• A metric to evaluate the system gain

16

N

SlnNlnG

Mathematic Model in ESP Game- System Gain Modeling (2/2)

17

C2

k)ln(Vln(T)ln(r)

k)ln(Vln(T)ln(r)k))ln(V(ln(T)(ln(r))

ln(r))k)(ln(Vln(r))(ln(T)

r)kln(Vln(T/r)G

2

2

N

SlnNlnG

2

k)ln(Vln(T)

er

Mathematic Model in ESP Game- Image Selection (1/3)

• Different image selection method

• Random

• Fresh-first

• Maximize ln(N)

• Optimal (OPSA)

• Largest possible system gain

• P0: all images that have not been played

• P1: all images that have been played at least once, but less than r times

• P2: other images

18

Mathematic Model in ESP Game- Image Selection (2/3)

19

T / N

Mathematic Model in ESP Game- Image Selection (3/3)

20

M: Total number of images in the system

Wang, B.-C., C.-W. Lin, et al. (2008). An Analytical Study and Modeling of the ESP game. Taipei, Taiwan, Institute of Information Science, Academia Sinica.

Mathematic Model in ESP Game- Automatic Playing the ESP Game (1/4)

• Given enough instances of labeled images as training data

• Similar with Language model

• Assign probability to the next label to be added

• Plays the ESP Game without looking at the image

• Limitation: at least one “off-limits” in the image

21

Weber, I., Robertson, S. & Vojnovi´C, M. (2008) Rethinking the ESP game. Report for Microsoft Research, Microsoft Corporation.

Mathematic Model in ESP Game- Language Modeling (1/2)

• Model the sequence of word usage of a certain language

• Given a Language Model: M

• P(I love this game) = P(I) P(love|I) P(this|I love) P(game|I love this)

22

Mathematic Model in ESP Game- Language Modeling (2/2)

• P(I love this game) = P(I) P(love|I) P(this|I love) P(game|I love this)

• Markov assumption:

• To give reasonable predictions, only the prior local context (the last few words) affects the next word

• n-gram word model

• Bi-gram:

• P(I love this game) = P(I) P(love|I) P(this|love) P(game|this)

23

Mathematic Model in ESP Game- Automatic Playing the ESP Game (2/4)

• Given enough instances of labeled images as training data

• At least one “off-limits” in the image

• t: next label

• T: off-limits set

• P(“t is next label” | “set T already present”)

24

Weber, I., Robertson, S. & Vojnovi´C, M. (2008) Rethinking the ESP game. Report for Microsoft Research, Microsoft Corporation.

)(

)()|()|(

TP

tPtTPTtP

)()|( tPtTP

Mathematic Model in ESP Game- Automatic Playing the ESP Game (3/4)

• P(“t is next label” | “set T already present”)

• Conditional independence assumption

• The problem of zero probability

25

Weber, I., Robertson, S. & Vojnovi´C, M. (2008) Rethinking the ESP game. Report for Microsoft Research, Microsoft Corporation.

)()|()|( tPtTPTtP

Tt

i

i

ttPtTP )|()|(

Mathematic Model in ESP Game- Automatic Playing the ESP Game (4/4)

• The problem of zero probability

26

Weber, I., Robertson, S. & Vojnovi´C, M. (2008) Rethinking the ESP game. Report for Microsoft Research, Microsoft Corporation.

)()|()1()|(~

iii tPttPttP

Tt

i

i

ttPtTP )|()|(

C

tPttP

TtP Tti

i

)()|(

)|(

~

Mathematic Model in ESP Game- Others

• User scoring model

• More informative tags

27

Problems of ESP Game

• There is a lot of redundancy in the tag sets

• Even when tags are not exactly synonyms, they are often “to be expected” given the other tags

• There is a tendency to match on colors

• People tend to add more generic labels

Weber, I., Robertson, S. & Vojnovi´C, M. (2008) Rethinking the ESP game. Report for Microsoft Research, Microsoft Corporation. 28

Conclusion and Comments

• The first thing of Human Computing research is finding a clear PURPOSE

• How your game or method to achieve the purpose

29