1 Extracting Relations from Large Text Collections Eugene Agichtein, Eleazar Eskin and Luis Gravano Department of Computer Science Columbia University

1

Extracting Relations Extracting Relations fromfrom

Large Text CollectionsLarge Text Collections

Eugene Agichtein, Eugene Agichtein, Eleazar Eskin and Luis Eleazar Eskin and Luis GravanoGravano

Department of Computer ScienceDepartment of Computer ScienceColumbia UniversityColumbia University

Combining Strategies for

Eugene Agichtein Columbia University•2

Extracting Relations From Extracting Relations From DocumentsDocuments

•Extract all tuples that appear in the document collection•Require minimal training•Resolve conflicting information•Exploit redundancy of information in documents

Text Documents hide valuable structured information


Example Task: Example Task: Organization/LocationOrganization/Location

Apple's programmers "think different" on a "campus" in

Cupertino, Cal. Nike employees "just do it" at what the company refers to as its "World Campus," near Portland, Ore.

Microsoft's central headquarters in Redmond is home to almost every product group and division. Organization Location

Microsoft

Apple Computer

Nike

Redmond

Cupertino

Portland

Brent Barlow, 27, a software analyst and beta-tester at Apple Computer headquarters in Cupertino, was fired Monday for "thinking a little too different."


Related WorkRelated Work

• Traditional Information ExtractionTraditional Information Extraction– MUCMUC

• BootstrappingBootstrapping– Riloff et al. (‘99), Collins & Singer (‘99)Riloff et al. (‘99), Collins & Singer (‘99)

– Blum & Mitchell (co-training) (‘98)Blum & Mitchell (co-training) (‘98)

– Brin (DIPRE) (‘98)Brin (DIPRE) (‘98)


Extracting Relations from TextExtracting Relations from Text

ORGANIZATION LOCATIONMICROSOFT REDMONDIBM ARMONKINTEL SANTA CLARABOEING SEATTLE

Seed tuples:



Computer servers at Microsoft'sheadquarters in Redmond...Exxon, Irving, said it will boost itsstake in the...In midafternoon trading, shares ofIrving-based Exxon fell…The Armonk-based IBM has introduced anew line ...The combined company will operate fromBoeing's headquarters in Seattle.Intel, Santa Clara, cut prices of itsPentium...

Occurrences of seed tuples in text:




Tag Entities:




GeneratePatterns:

• <ORGANIZATION>’s headquarters in <LOCATION>

•<LOCATION> -based <ORGANIZATION>

•<ORGANIZATION> , <LOCATION>


• <ORGANIZATION>’s headquarters in <LOCATION>

•<LOCATION> -based <ORGANIZATION>

•<ORGANIZATION> , <LOCATION>


Generate new seed tuples:



• Potential Pitfalls:Potential Pitfalls:

– Inaccurate Generated PatternsInaccurate Generated Patterns

– Conflicting InformationConflicting Information

• Snowball Solutions:Snowball Solutions:

– Pattern representationPattern representation

– Pattern generationPattern generation

– Automatic patterns and tuples evaluationAutomatic patterns and tuples evaluation

– Different context representationsDifferent context representations


SnowballSnowball Systems Systems

• Snowball-VSSnowball-VS (Vector Space) (Vector Space)– Original system (presented in DL’00)Original system (presented in DL’00)

• Snowball-SMTSnowball-SMT (consider order of (consider order of terms)terms)– estimating probability based on the estimating probability based on the

sequence of terms surrounding entities.sequence of terms surrounding entities.

– Uses Sparse Markov Transducers for Uses Sparse Markov Transducers for probability estimation.probability estimation.


Snowball-VS: Pattern RepresentationSnowball-VS: Pattern Representation

ORGANIZATION 's central headquarters in LOCATION is home to...

Snowball-VS : Vector space model. Text context is represented as vectors of terms


Snowball-VS: Pattern RepresentationSnowball-VS: Pattern Representation

ORGANIZATION 's central headquarters in LOCATION is home to...

LOCATIONORGANIZATION{<'s 0.5>, <central 0.5>, <headquarters 0.5>, < in 0.5>}

{<is 0.75>, <home 0.75> }

Snowball-VS : Vector space model. Text context is represented as vectors of terms


Snowball-VS: Pattern GenerationSnowball-VS: Pattern Generation


Snowball-VS: Tuple ExtractionSnowball-VS: Tuple Extraction

• Represent each text segment as the Represent each text segment as the context 5-tuple:context 5-tuple:

Netscape 's headquarters in Mountain View is near




LOCATIONORGANIZATION{<'s 0.5>, <headquarters 0.5>, < in 0.5>}

{<is 0.75>, <near 0.75> }





• Find Most Similar Pattern:Find Most Similar Pattern:


{<is 0.75>, <near 0.75> }





• Find Most Similar Pattern:Find Most Similar Pattern:


{<is 0.75>, <near 0.75> }


LOCATIONORGANIZATION{<'s 0.5>, <central 0.5>, <headquarters 0.5>, < in 0.5>}

{<is 0.75>, <home 0.75> }


Snowball-VS: Pattern EvaluationSnowball-VS: Pattern Evaluation

Conf(Pattern) = Positive/(Positive + Negative) Conf(Pattern) = Positive/(Positive + Negative)

e.g., Conf(Pattern) = 2/3 = 66%e.g., Conf(Pattern) = 2/3 = 66%

Exxon, Irving, said +Intel, Santa Clara, cut prices +invest in Microsoft, New York-basedanalyst Jane Smith said

-

Pattern P, “ORGANIZATION, LOCATION” in action:

Pattern Confidence


Snowball-VS: Tuple evaluationSnowball-VS: Tuple evaluation

Conf(T) = 1 - Conf(T) = 1 - (1 -Conf(P(1 -Conf(Pii))))

A tuple T will have high confidence if it is extracted A tuple T will have high confidence if it is extracted by multiple high-quality patterns, scaled by theby multiple high-quality patterns, scaled by thesimilarity of actual text segments to patterns. similarity of actual text segments to patterns.


Snowball-SMT Snowball-SMT : Tuple Extraction: Tuple Extraction


Conf(<Netscape, Mountain View>) = P( Positive | ‘s , headquarters, in )

Use probabilistic model that best describes the positive examples(seed tuples) and negative examples (conflicting tuples)

Exxon, Irving, said +Intel, Santa Clara, cut prices +invest in Microsoft, New York-basedanalyst Jane Smith said

-


Combining Combining Snowball-VSSnowball-VS and and Snowball-Snowball-SMTSMT

Initial seed tuples



Run each system in parallel















Combination Strategies: •Union•Intersection•Weighed Mixture



Select tuples to use as new seed



Repeat the process


Task Evaluation MethodologyTask Evaluation Methodology

• Data: (300,000 newspaper articles)Data: (300,000 newspaper articles)

• Methodology:Methodology:– IdealIdeal Set of tuples Set of tuples

– Automatic Recall/Precision estimationAutomatic Recall/Precision estimation

• Estimated precision using samplingEstimated precision using sampling(see paper)(see paper)


TheThe Ideal Ideal metric metric

All tuples mentioned in the collection

Hoover’s directory(13K+ organizations)

Ideal

• Precision:Precision:Fraction of Fraction of IdealIdeal tuples that the system tuples that the system extracted that have correct values.extracted that have correct values.

• Recall:Recall:Fraction of Fraction of IdealIdeal tuples extracted from tuples extracted from

collection.collection.


Experimental Results (Individual Experimental Results (Individual Systems) Systems)

Precision(a) and Recall(b) of Snowball-VS, Snowball-SMT and DIPRE.

(a) (b)

Extracted tables contain more than 80,000 tuples


Experimental Results - CombinationExperimental Results - Combination

Precision and Recall (b) of the Intersection, Union and Mixture strategies compared to Snowball-VS

(a) (b)


ConclusionsConclusions

• System works after minimal training System works after minimal training (handful of examples)(handful of examples)

• Able to extract 80% of test tuplesAble to extract 80% of test tuples

• Combining Models results in better Combining Models results in better tabletable


Future WorkFuture Work

• EfficiencyEfficiency

• Evaluate on other extraction tasksEvaluate on other extraction tasks– non-binary relationsnon-binary relations

– relations with no keyrelations with no key

HTML documentsHTML documents

– link structurelink structure

– document structuredocument structure


ReferencesReferences

• Blum & Mitchell. Combining labeled and unlabeled Blum & Mitchell. Combining labeled and unlabeled data with co-training. Proceedings of 1998 Conference data with co-training. Proceedings of 1998 Conference on Computational Learning Theory.on Computational Learning Theory.

• Brin. Extracting patterns and relations from the World-Brin. Extracting patterns and relations from the World-Wide Web. Proceedings on the 1998 International Wide Web. Proceedings on the 1998 International Workshop on Web abd Databases (WebDB’98)Workshop on Web abd Databases (WebDB’98)

• Collins & Singer. Unsupervised models for named Collins & Singer. Unsupervised models for named entity classification. EMNLP 1999entity classification. EMNLP 1999

• Riloff & Jones. Learning dictionaries for information Riloff & Jones. Learning dictionaries for information extraction by multi-level bootstrapping. AAAI’99extraction by multi-level bootstrapping. AAAI’99

• Yarowsky. Unsupervised word sense disambiguation Yarowsky. Unsupervised word sense disambiguation rivaling supervised methods. ACL’95rivaling supervised methods. ACL’95

38

Snowball: Snowball: Extracting Relations from Extracting Relations from Large Plain-Text CollectionsLarge Plain-Text Collections

Eugene AgichteinEugene Agichtein and Luis Gravano and Luis GravanoDepartment of Computer ScienceDepartment of Computer ScienceColumbia UniversityColumbia University


Snowball patterns incorporate Snowball patterns incorporate Named Entity tagsNamed Entity tags

Pattern 1 <ORGANIZATION>'s headquarters in <LOCATION>

Pattern 2 <LOCATION>-based <ORGANIZATION>}

Pattern 3 <ORGANIZATION>, <LOCATION>

Entities tagged using MITRE’s Alembic Workbench namedentity tagger.


Experimental results: trainingExperimental results: training

(a) (b)

Recall (a) and precision (b) using the Ideal metric (training collection)


Snowball: ContributionsSnowball: Contributions

• Pattern RepresentationPattern Representation

• Pattern GenerationPattern Generation

• Patterns and Tuples EvaluationPatterns and Tuples Evaluation


Snowball patterns incorporate Snowball patterns incorporate Named Entity tagsNamed Entity tags

Entities tagged using MITRE’s Alembic Workbench named entity tagger.

Microsoft 's central headquarters in Redmond is home to almost every product group and division.

Today's merger with McDonnell Douglas

positions Seattle -based Boeing to make major money in space.

<ORGANIZATION>’s central headquarters in <LOCATION>

…, a producer of apple-based jelly, ...

<LOCATION>-based <ORGANIZATION>


Snowball-VS: Pattern Similarity Snowball-VS: Pattern Similarity MetricMetric

{ Lp . Ls + Mp . Ms + Rp . Rs if the tags match

0 otherwise

Match(P, S) =

P = <Lp, T1, Mp, T2, Rp>

S = <Ls, T1, Ms, T2, Rs>


Approximate MatchingApproximate Matching

• Use Whirl Use Whirl

Microsoft Corp. = MicrosoftMicrosoft Corp. = Microsoft = Microsoft = Microsoft CorporationCorporation


Collections used in Experiments Collections used in Experiments

More than 300, 000 real newspaper articles

Collection Source Year

TrainingThe New York TimesThe Wall Street JournalThe Los Angeles Times

199619961996

TrainingThe New York TimesThe Wall Street JournalThe Los Angeles Times

199519951995,’97


Sparse Markov TransducersSparse Markov Transducers

• Computes conditional probability: P(Positive|WComputes conditional probability: P(Positive|W11,W,W22,W,W33,W,W44) ) where W where Wii is the i-th word in the context between entities. is the i-th word in the context between entities.

• SMT’s compute mixture of sparse prediction trees.SMT’s compute mixture of sparse prediction trees.

Eskin, Grundy, and Singer ISMB-2000.