Type Attack - Crowdsourcing OCR Correction With a Human Computation Game

Crowdsourcing OCR CorrectionThrough Game Playing

By:

Lin Tingji JovianNational University of Singapore (NUS)

Olivier AmprimoDigital Resources & Services, National Library Board (NLB)

http://apps.facebook.com/typeattack/ © Lin Tingji Jovian, Olivier Amprimo

Overview

1. Problems with Digital Archiving


Overview


2. Solutions


Overview


2. Solutions

3. TypeAttack


Overview


2. Solutions

3. TypeAttack

4. How Does TypeAttack Work


Overview


2. Solutions

3. TypeAttack

4. How Does TypeAttack work

5. Where We Are Now

Problems


Problems in Digitizing Archives

• National Library Board (NLB) digitizes The Straits Times (SPH) articles.




• However, when the articles are older, digitization of the content is prone tomany inaccuracies.





• For example:

Source: ReCaptcha




Some of NLB’s OCRResult

NLB’s OCR Translation


IS NOT GOOD ENOUGH !Some of NLB’s OCRResult



• National Library Board (NLB) digitizes Straits Times (SPH) articles.


• For example:

• In fact, the NLB needs to employ people to double check and rectifyerrors.



• National Library Board (NLB) digitizes Straits Times (SPH) articles.


• For example:

• In fact, the NLB needs to employ people to double check and rectifyerrors.

• This leads to extra cost and inefficiency.

Solutions


Solution to Improve Digitization Process

1. Many tasks challenges even sophisticated computer programs.


Solution to Improve Digitization Process

1. Many tasks challenges even sophisticated computer programs.

2. They are trivial for humans


Options to Improve Digitization Process

1. Employ a large number of people, dedicated full time.

+ + NLB has experience in doing this.

- - Resource Allocation and Co-ordination Cost


2. Enroll and Support Volunteers

+ + National Library of Australia > http://www.nla.gov.au/ndp/get_involved/

- - Singapore specifics:• Copyrights• Computer Literacy of Elders• Partial Retirement



3. Make it a mandatory part of popular processes

+ + A Turing Machine to separate bots from humans online > ReCaptcha

- - ReCaptcha turned out to be very proprietaryNo plans for a change, even with GoogleWord, not sentence > decontextualization = poor meaning



4. Make it part of something attractive

+ + Tap into popularity of games – Human Computational GamesMore than 200 million hours are spent each day playing

computer games in U.S. alone.The World Financial Crisis makes it bigger!

>> TypeAttack



TypeAttack

• TypeAttack is a Human Computational Game on Facebook that helpsdigitize archives for National Library Board.

• Being built on Facebook, TypeAttack can:– Harness Facebook’s 200 million active users worldwide.– Utilize Facebook’s viral techniques:

• Friend invites to the game.• Publishing typing scores onto a user’s Facebook wall.• Utilizing Facebook’s newsfeeds to expose TypeAttack to more

Facebook users.– Most importantly, perform Human Computation efficiently.


TypeAttack






TypeAttack






TypeAttack






Luis von AhnCarnegie Mellon University

Since people spend so much time on computer games…

Let us make use of them to perform tasks that computershave difficulty performing.

BIRTH OF Human Computational Games1.These are games played by humans that produce useful computationas a side-effect.

2.People play not because they want to solve computational problems,but because they want to be entertained.

3.It combines human brainpower with computers to solve problemsthat neither could solve alone.

Step-back:Motivation behind Human Computational Games

















How Does TypeAttack Works


How TypeAttack works

• Flow:


• Flow:

Entire collection of Straits Timesin the year 1938 (worth 10 GB)

XML files representing data per (newspaper)page



image

XML


1. TypeAttack “cuts” the differentarticles in a page.

2. Within an article, it cuts out shortsnippets of text.


1. TypeAttack “cuts” the differentarticles in a page.

2. Within an article, it cuts out shortsnippets of text.



• Flow:

Entire 1938 Straits Times+

Respective XML files

TypeAttack+

Facebook users+

Computational Algorithms

Digitized versions withover 99.0% transcription

accuracy at the word level.



How does TypeAttack digitize content?

TypeAttack uses 2 kinds of information:

1. Output from Facebook userss.

2. NLB’s OCR Translation Results– accuracy rate @ word level

3. Bi-gram (text prediction)– Looks at the probability of two-word sequence– E.g. Given Word A, what is the probability of Word B?





2. NLB’s OCR Translation Results– accuracy rate @ word level






2. Results from NLB’s OCR Translation.– accuracy rate @ word level



How does TypeAttack digitize content?(1) Output from Facebook users



Today is nice.

Today iz nice.

is nize.

Today



Today is nice.

Today iz nice.

is nize.

Today



Today is nice.

Today iz nice.

is nize.

Today



Today is nice.

Today iz nice.

is nize.

Today is



Today is nice.

Today iz nice.

is nize.

Today is



Today is nice.

Today iz nice.

is nize.

Today is nice.



Today is nice.

Today iz nice.

is nize.

Today is nice.



• Say for example, the paragraph to be typed is “Today is nice.".

• Based on all the players' output on this particular paragraph, it will get aprobability of each word.

“Today" = 96.4% of users typed."is" = 95.1% of users typed."nice." = 97.3% of users typed.

• Thus since each word in this paragraph is at least 95% probable, wedetermine that “Today is nice." is the correct output.














How does TypeAttack digitize content?(2) NLB’s OCR Translation Result

• Method previously simply compares the output between players.

• To speed things up, we will compare the players' output with the OCRResult.

• Once every word in the paragraph is >95% probable, the paragraph'sstatus is set to 'complete' and will not be displayed in the game anymore.




• To speed things up, we will compare the players' output with theOCR Translation Result.





• To speed things up, we will compare the players' output with theOCR Translation Result.



Innovativeness and Uniqueness

1. Uses Game Elements and Social Networking to channel humanbrainpower through computer games.




2. Utilizing probabilities that are determined from different areas (fromFacebook userss and OCR) to ensure that we can extract the correcttext content with minimum user output.




2. Utilizing probabilities that are determined from different areas (fromFacebook userss and OCR) to ensure that we can extract the correcttext content with minimum user output.

3. Working SystemSeamlessly integrates Game Elements, Computational Algorithms andSocial Networking aspects to solve problems that neither humans norcomputers can solve individually.

Where We Are Now


Where We Are Now

• Evaluating the project : Sustainability• User growth• Volume of contribution• Average user contribution trend• Automating the process further (snippet selection)• Expansion beyond FaceBook


Where We Are Now

• Evaluating the project : Sustainability• User growth• Volume of contribution• Average user contribution trend• Automating the process further (snippet selection)• Expansion beyond FaceBook

• Impacts• Product Design• Marketing and Communication• NLB Digitization Process• NLB Culture


Where We Are Now

• Evaluating the project : Economic Rationale• Performance (Words per minute) TypeAttack (60) vs Standard (33)• Cost of TypeAttack (operations + development) vs Part timers


Where We Are Now

• Evaluating the project : Economic Rationale• Performance (Words per minute) TypeAttack (60) vs Standard (33)• Cost of TypeAttack (operations + development) vs Part timers

• Impacts• Finance• IT and Digital Services


Where We Are Now

• Evaluating the project : Scope of Activity• What about content with low word confidence (< 95%) ?• Only the Straits Times +70 years old?• Other languages? Mandarin, Bahasa Melayu, Tamil


Where We Are Now

• Evaluating the project : Scope of Activity• What about content with low word confidence (< 95%) ?• Only the Straits Times +70 years old?• Other languages? Mandarin, Bahasa Melayu, Tamil

• Impacts• NLB Digitization Process• Singapore Copyright Law: 70 years > 20 years• NLB / SPH Partnership (derogatory agreement on copyright)

What You Can Do


What YOU can do

• Be part of the crowd: join and play! http://apps.facebook.com/typeattack/


What YOU can do


• Spread the word!


What YOU can do


• Spread the word!

• Feel free to contribute!

• After which, NLB will decide whether they will want to use the system tofurther digitize more contents that have poor OCR accuracies.

Thank You

Documents

Type Attack - Crowdsourcing OCR Correction With a Human Computation Game