Embed Size (px)
DESCRIPTION
Presented at BJUG, 5/8/2012 by Ivan Portilla IBM Watson is a reasoning system with a question and answer front end that processes natural language coming from both structured and unstructured data. Watson additionally incorporates analytics from which the system learns to derive answer confidence and scoring. We will discuss the Watson System and some of its key foundations that came from the Open Source Apache Software Foundation. We will share the lessons learned of using Open source technologies including UIMA, Derby, Hadoop and Tomcat in Watson. We will explain how the primary (shallow) search was built with Apache Lucene and how the team followed Agile best practices for its Software development efforts.
Citation preview
If I have seen further it is by standing on the shoulders of giants.
Isaac Newton, Letter to Robert Hooke, February 5, 1675
2Sunday, May 20, 12
3
ObjectivesBy the end of this session, you
should be able to:üDescribe the main characteristics
of Watson QA system.ü Identify the key open source SW
used in Watson.üRecognize examples of Agile
development best practices.
3Sunday, May 20, 12
Disclaimers
4Sunday, May 20, 12
Disclaimer 1ü This presentation represents the view of the
author and does not represent the view of IBM. ü All opinions expressed in this presentation are
strictly of the speaker, and do NOT represent those of IBM, IBM management, or anyone else.
ü IBM and IBM (logo) are trademarks or registered trademarks of International Business Machines Corporation in the United States and/or other countries.
5Sunday, May 20, 12
Disclaimer 2I (We) do not work for the Watson team.
6Sunday, May 20, 12
Let’s Play Jeopardy
7Sunday, May 20, 12
Let’s Play JeopardyBEFORE & AFTER: The Jerry Maguire star who automatically maintains your vehicle’s speed.
7Sunday, May 20, 12
Let’s Play JeopardyBEFORE & AFTER: The Jerry Maguire star who automatically maintains your vehicle’s speed.
7Sunday, May 20, 12
Let’s Play JeopardyBEFORE & AFTER: The Jerry Maguire star who automatically maintains your vehicle’s speed.
COMMON BONDS: trout, loose change in your pocket, and compliments.
7Sunday, May 20, 12
Let’s Play JeopardyBEFORE & AFTER: The Jerry Maguire star who automatically maintains your vehicle’s speed.
COMMON BONDS: trout, loose change in your pocket, and compliments.
Diplomatic Relations: Of the four countries in the world that the United States does not have diplomatic relations with, the one that’s farthest north.
7Sunday, May 20, 12
Let’s Play JeopardyBEFORE & AFTER: The Jerry Maguire star who automatically maintains your vehicle’s speed.
COMMON BONDS: trout, loose change in your pocket, and compliments.
Diplomatic Relations: Of the four countries in the world that the United States does not have diplomatic relations with, the one that’s farthest north.
7Sunday, May 20, 12
Let’s Play JeopardyBEFORE & AFTER: The Jerry Maguire star who automatically maintains your vehicle’s speed.
COMMON BONDS: trout, loose change in your pocket, and compliments.
Diplomatic Relations: Of the four countries in the world that the United States does not have diplomatic relations with, the one that’s farthest north.
Geography: Chile shares its longest land border with this country
7Sunday, May 20, 12
Natural Language Processing
Understanding natural language is hard!
8Sunday, May 20, 12
Watson educa@on
hAp://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=6177717
9Sunday, May 20, 12
A Brief History of Watson
10Sunday, May 20, 12
A Brief History of Watson§ Deep Blue Ended in 1997§ Looking for a new research challenge§ 2004, IBM Research manager Charles Lickel,
§ Ken Jennings§ Started in 2005
• David Ferrucci• DeepQA in 2007• Won Jeopardy Match, Feb 2011
hAp://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=6177717
11Sunday, May 20, 12
A Brief History of Watson§ Deep Blue Ended in 1997§ Looking for a new research challenge§ 2004, IBM Research manager Charles Lickel,
§ Ken Jennings§ Started in 2005
• David Ferrucci• DeepQA in 2007• Won Jeopardy Match, Feb 2011
hAp://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=6177717
11Sunday, May 20, 12
Baseline
12/2007
8/2008
5/2009
10/2009
11/2010
12/2008
5/2008
4/2010
Prec
isio
n
hAp://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=617771712Sunday, May 20, 12
What is Watson?
üUnderstands natural language.
üGenerates & evaluates hypothesis for beAer outcomes.
üAdapts & learns from user selec@ons and responses.
hAp://www.ibm.com/innova@on/us/watson/
13Sunday, May 20, 12
Watson metricsDevelopment Team: 25 peopleProject Duration: 4 years
Hardware: 90 IBM Power-750 servers 2880 Power7 cores @ 80+ TFLOPS20 TB Disk, 16 TB RAM (memory)10 Gbps network
hAp://na11.apachecon.com/talks/19932
hAp://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=617771714Sunday, May 20, 12
Open Source Software
15Sunday, May 20, 12
OSS -‐ Linux
hAp://video.linux.com/videos/linuxcon-‐vancouver-‐day-‐2-‐1/
16Sunday, May 20, 12
Open Source too
hAp://@.arc.nasa.gov/opensource/hAp://ocw.mit.edu/index.htm
17Sunday, May 20, 12
How does it work?
18Sunday, May 20, 12
Learning To Rank (Basic Architecture)
HypothesisEvidenceScoring
Keywords
Watson by R.Yates
19Sunday, May 20, 12
Watson Architecture
hAp://en.wikipedia.org/wiki/Watson_(computer)
20Sunday, May 20, 12
Who is the 44th President of the United States?
21Sunday, May 20, 12
Who is the 44th President of the United States?
Who is the 44th President of the United States?
Watson by R.Yates
22Sunday, May 20, 12
Who is the 44th President of the United States?
Who is the 44th President of the United States?
'Who' is the '44th' 'President' of the 'United States'?
Lexical Answer Type
→ Person
Focus
Can be replaced by the correct answer to make a true statement
Keywords
23Sunday, May 20, 12
Who is the 44th President of the United States?
Question
HypothesisGeneration
QuestionAnalysis
Keywords:44th President United States
Watson by R.Yates
24Sunday, May 20, 12
Who is the 44th President of the United States?
Question
HypothesisGeneration
QuestionAnalysis
Keywords:44th President United States
25Sunday, May 20, 12
Question
PrimarySearch
HypothesisGeneration
QuestionAnalysis
Who is the 44th President of the United States?
26Sunday, May 20, 12
Question
PrimarySearch
HypothesisGeneration
QuestionAnalysis
Barack ObamaGeorge W. BushHarvard Law SchoolIllinois
Who is the 44th President of the United States?
Watson by R.Yates
27Sunday, May 20, 12
ScoringScoring
Question
Scoring
HypothesisGeneration
QuestionAnalysis
Who is the 44th President of the United States?
Who is the 44th President of the United States?Barack Obama
Who is the 44th President of the United States?George W. Bush
Who is the 44th President of the United States?Harvard Law School
Who is the 44th President of the United States?Illinois
AnswerScoring
28Sunday, May 20, 12
AnswerScoringAnswer
Scoring
ScoringScoring
Question
ContextualAnswer Scoring
Scoring
HypothesisGeneration
QuestionAnalysis
Who is the 44th President of the United States?
Who is the 44th President of the United States?Barack Obama
Who is the 44th President of the United States?George W. Bush
Who is the 44th President of the United States?Harvard Law School
Who is the 44th President of the United States?Illinois
AnswerScoringAnswer
Scoring
Who is the 44th President of the United States?
→ Person
Is Barack Obama a Person? .90Is George W. Bush a Person? .90Is Harvard Law School a Person? .10Is Illinois a Person? .15
Watson by R.Yates
29Sunday, May 20, 12
ScoringScoring
Question
Unstructured Informa@on Management Applica@ons -‐ UIMA
Scoring
HypothesisGeneration
QuestionAnalysis
ContextualAnswer Scoring
Who is the 44th President of the United States?
Barack Obama is the 44th President of the United StatesGeorge W. Bush is the 44th President of the United StatesHarvard Law School is the 44th President of the United StatesIllinois is the 44th President of the United States
30Sunday, May 20, 12
ScoringScoring
Question
Unstructured Informa@on Management Applica@ons -‐ UIMA
Scoring
HypothesisGeneration
QuestionAnalysis
Who is the 44th President of the United States?
Barack Obama is the 44th President of the United StatesGeorge W. Bush is the 44th President of the United StatesHarvard Law School is the 44th President of the United StatesIllinois is the 44th President of the United States
Barack Hussein Obama II (i/bəәˈrɑːk huːˈseɪn oʊˈbɑːməә/; born August 4, 1961) is the 44th and current President of the United States.
George Walker Bush (born July 6, 1946) is an American politician who served as the 43rd President of the United States from 2001 to 2009 and the 46th Governor of Texas from 1995 to 2000.
Barack Obama .95George W. Bush .80Harvard Law School .05Illinois.10
Watson by R.Yates
31Sunday, May 20, 12
ScoringScoring
QuestionTrained Models
EvidenceRetrieval
Unstructured Informa@on Management Applica@ons -‐ UIMA
Scoring
HypothesisGeneration
QuestionAnalysis
Who is the 44th President of the United States?
Candidate Answer Answer Scoring
Evidence retrieval & scoring
Confidence
Barack Obama 0.90 0.90 .95
George W. Bush 0.90 0.80 .65
Harvard Law School 0.10 0.05 .05
Illinois 0.15 0.10 .10
Watson by R.Yates
32Sunday, May 20, 12
. . .
Models
Answer & Confidence
Question
Evidence Sources
Models
Models
Models
Models
ModelsPrimarySearch
CandidateAnswerGeneration
HypothesisGenera@on
Hypothesis and Evidence Scoring
Final Confidence Merging & Ranking
Answer Sources
Ques@on & Topic Analysis
Answer Scoring
Evidence Retrieval & Scoring
Learned Modelshelp combine and weigh the Evidence
HypothesisGenera@on
Hypothesis and Evidence Scoring
Ques@onDecomposi@on
Balance& Combine
Merging &Ranking
Synthesis
DeepQAMassively Parallel Probabilistic Evidence-Based Architecture
ApacheCon 2011, Watson, a Reasoning System: based on Apache Inside!, David Boloker
33Sunday, May 20, 12
OSS in Watson 1.0
34Sunday, May 20, 12
OSS in Watson
üUIMA, UIMA-‐ASüHadoop, Map ReduceüLucene, Indri
ApacheCon 2011, Watson, a Reasoning System: based on Apache Inside!, David Boloker
35Sunday, May 20, 12
UIMA-‐Asynchronous Scaleout
UIMA AS provides more flexible and powerful scale out capability.
Innovate 2011, How Does It Work? The Architecture of Watson. Grady Booch
37Sunday, May 20, 12
Think Hadoop
A framework for storing & processing big data.
üUp to 4,000 machinesüUp to 20 PB
High reliability done in soiware:üAutomated failover for data & computa@onüImplemented in Java
hAp://hadoop.apache.org/mapreduce/
38Sunday, May 20, 12
Map reduce
hAp://hadoop.apache.org/mapreduce/
39Sunday, May 20, 12
UIMA pipelines in Hadoop
hAp://blogs.apache.org/founda@on/entry/apache_innova@on_bolsters_ibm_s41
HadoopDistributedFileSystem
Input
.
.
.
~5000 “splits”
Mapper
Mapper
Mapper
Mapper
.
.
.
~50 -‐ 100 “mappers”
Mul@ple threads, each runing a UIMA pipeline
ThreadThread
ThreadThread
. . .
ThreadThread
ThreadThread
. . .
ThreadThread
ThreadThread
. . .
ThreadThread
ThreadThread
. . .
Reducer
Reducer
.
.
.Shuffl
e/Sort
HadoopDistributedFileSystem
Output
~400-‐800 threads
40Sunday, May 20, 12
Architecture
hAp://lucene.apache.org
41Sunday, May 20, 12
Indri &Lemur
Indri is a text search engine developed at Umass & CMU. Indri is part of the Lemur project.
hAp://lemurproject.org/indri/
42Sunday, May 20, 12
© 2011 IBM Corporation
Watson answers in 2-‐6 seconds
Ques@on100s Possible Answers
1000’s of Pieces of Evidence
Mul@ple Interpreta@ons
100,000’s scores from many simultaneous Text Analysis Algorithms100s sources
. . .
HypothesisGenera@on
Hypothesis and Evidence Scoring
Final Confidence Merging & RankingSynthesisQues@on &
Topic AnalysisQues@onDecomposi@on
HypothesisGenera@on
Hypothesis and Evidence Scoring Answer &
Confidence
ApacheCon 2011, Watson, a Reasoning System: based on Apache Inside!, David Boloker
43Sunday, May 20, 12
Other OSS in Watson
hAps://www.ibm.com/developerworks/mydeveloperworks/blogs/InsideSystemStorage/entry/ibm_watson_how_to_build_your_own_watson_jr_in_your_basement7?lang=en
44Sunday, May 20, 12
J-‐Archive data
hAp://www.j-‐archive.com/showgame.php?game_id=3577
45Sunday, May 20, 12
Development Process
üWar room setting with continuous collaboration.
üWeekly integration.üResults driven with E2E
regression testing.
ü About 8,000 experimentsü10 GBs of test data/wk.üAgile development
Innovate 2011, How Does It Work? The Architecture of Watson. Grady Booch
46Sunday, May 20, 12
Other OSS
hAp://www.apache.orghAp://manning.com/
47Sunday, May 20, 12
Take AwayOSS is powerful and scalable enough for the Watson team, what about your project?
48Sunday, May 20, 12
ResourcesIBM Journal of Research and DevelopmenthAp://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=6177717IBM WatsonhAp://www.ibm.com/innova@on/us/watson/hAp://www.research.ibm.com/deepqa/index.shtml
NovahAp://www.pbs.org/wgbh/nova/tech/smartest-‐machine-‐on-‐earth.html
49Sunday, May 20, 12
51
Review of Objectives
Now that you have completed this session, you are able to:
ü Describe the main characteristics of Watson QA system.ü Identify the key open source tools used in Watson.ü Recognize examples of Agile development best practices.
50Sunday, May 20, 12
52
…any final questions ?
51Sunday, May 20, 12
