Results of the 2000 Topic Detection and Tracking Evaluation in Mandarin and English Jonathan Fiscus...

Results of the 2000 Topic Detection and Tracking Evaluation

in Mandarin and English

Jonathan Fiscus and George Doddington

What’s New in TDT 2000

• TDT3 corpus used in both 1999 and 2000– 120 topics used in the 2000 test: 60 1999 topics, 60 new topics

• Of 44K news stories, 24% wereat least singly judged YES or BRIEF

• 1999 and 2000 topics are very different in terms of size and cross-language makeup

– Annotation of new topics using search engine-guided annotation:“Use a search engine with on-topic story feedback and interactive searching techniques to limit the number stories read by annotators”

• Evaluation Protocol Changes– Only minor changes to the Tracking (Negative example stories)– Link Detection test set selection changed in light of last year’s experience

0100020003000400050006000700080009000

10000

TD

T3

To

pic

s

19

99

To

pic

Se

t

20

00

To

pic

Se

tCo

un

t of O

n T

op

ic S

tori

es

MandarinEnglish

Search-Guided Annotation: How will it affect scores?

• Simulate search-guided annotation using 1999 topics, 1999 annotations and 1999 Systems

Effectiveness of Seach Guided Annotation

0.001

0.01

0.1

1

10

100

1000

0.1 1 10ROF (#Read On-topic/#Read Off-Topic)

SG (FB=10) Pjudged

SG (FB=10) #Reads

Probability of a human reading a

judged story

Number of read stories

Stability in“Region ofInterest”

TDT Topic Tracking Task

training data

test data

on-topicunknownunknown

7 Participants:– Dragon, IBM, Texas A&M Univ., TNO, Univ. of Iowa, Univ. of Massachusetts, Univ. of

Maryland

System Goal:– To detect stories that discuss the target topic,

in multiple source streams.• Supervised Training

– Given Nt sample stories that discuss a given target topic

• Testing– Find all subsequent stories that discuss the target topic

Topic Tracking Results

0.1

1

Nor

mal

ized

Tra

ckin

g C

ost

English Translation Native Orthography

0.1

1

Nor

mal

ized

Tra

ckin

g C

ost English Translation, Nt=4,Nn=0

Native Orthography, Nt=4 Nn=2

Native Orthography, Nt=4, Nn=0

With Negative

Without Negative

Basic Condition: Newswire + BNews, reference story boundaries, English training: 1 On-topic

Challenge Condition: Newswire + BNews ASR, automatic story boundaries, English training: 4 On-topic, 2 Negative training

Topic Tracking Results(Expanded Basic Condition DET Curve)

Topic Tracking Results (Expanded Challenge Condition DET Curve)

With Negative

Without Negative

Effect of Automatic Story Boundaries

• Evaluation conditioned jointly by source and Language– Newswire, Broadcast News, English and Mandarin

Degradation due to story

boundaries source for ASR

Test Condition: NWT+Bnasr, 4 English Training Stories, Reference Boundaries

IBM1 UMass1

0.093 0.119

0.010

0.100

1.000

1999

Top

icT

rain

ing

2000

Top

icT

rain

ingN

orm

aliz

ed T

rack

ing

Cos

t

0.01

0.1

1

0.01 0.1 1

1999 Topic Training

20

00

To

pic

Tra

inin

g

Variability of Tracking Performance Based on Training Stories

• BBN ran their 1999 system on this year’s index files:– Same topics, but different

training stories– One caveat: these results based

on different “test epochs”, 2000 index files contain more stories

• There could be several reasons for the difference

• …needs future investigation

NIST Speech Group

TDT Link Detection Task

One Participant: University of Massachusetts

System Goal:– To detect whether a pair of stories discuss the same topic.

(Can be thought of as a “primitive operator” to build a variety of applications)

?

2000 Link Detection Results• A lot was learned last year:

– The test set must be properly sampled • “Linked” story pairs were selected by randomly sampling all

possible on-topic story pairs• “Unlinked” pairs were selected using all on-topic stories as one

of the pair, and a randomly chosen story was chosen as the second

– This year, the task was made multilingual

– More story pairs were used

0

10000

20000

30000

40000

50000

60000

70000

Overall Eng-to-Eng Eng-to-Man Man-to-Man

Num

ber

of S

tory

Pai

rs Linked Pairs

UnLinked PairsLink Detection Test Set Composition

Link Detection Results

• Required Condition– Multilingual texts– Newswire + Broadcast News ASR,– Reference story boundaries– 10 file decision deferral

0.31

0.17

0.330.41

0.10

1.00

UMass1

Nor

mal

ized

Lin

k D

etec

tion

Cos

t

OverallEng-to-EngMan-to-ManEng-to-Man

Overall

TDT Topic Detection Task

Three ParticipantsChinese Univ. of Hong Kong, Dragon, Univ. of Massachusetts

System Goal:– To detect topics in terms of the (clusters of) stories

that discuss them.

• “Unsupervised” topic training

• New topics must be detected as the incoming stories are processed.

• Input stories are then associated with one of the topics.

a topic!

• Required Condition (in yellow)

– Multilingual Topic Detection– Newswire+Broadcast News ASR– Automatic Story Boundaries

• Performance on the 1999 and 2000 topic sets are different

2000 Topic Detection Evaluation

0.1

1

CUHK1 Dragon1 UMass1

No

rma

lize

d D

ete

ctio

n C

ost

1999 Topics 2000 Topics 1999+2000 Topics

0.1

1

CUHK1 Dragon1 UMass1

Nor

mal

ized

Det

ectio

n C

ost

1999 Topics 2000 Topics 1999+2000 Topics

Using English Translations for Mandarin Using Native Orthography

Effect of Topic Size on Detection Performance

• The 1999 topics have more on-topic stories than the 2000 topics

• Distribution of scores are related to topic size– Bigger topics tend to have higher scores. – Is this a behavior induced by setting a topic size parameter in

training?

0.0001

0.001

0.01

0.1

1

1 10 100 1000

Number of On-Topic Stories

Nor

mal

ize

d D

etec

tion

Co

st 1999 Topics

2000 TopicsDragon1ResultsNWT+BNasr,

Reference Boundary, Multilingual Texts

Fractional Components of Detection Cost

• Evaluations conditioned on factors (like language) are problematic

• Instead, compute the additive contributions to detection costs for different subsets of data.

00.005

0.010.015

0.020.025

0.03

Nor

mal

ized

Det

ectio

n C

ost

English

Mandarin

0.00001

0.0001

0.001

0.01

0.1

1E-05 1E-04 0.001 0.01 0.1

Fractional Cost of English Errors

Fra

ctio

nal

Co

st o

f Man

dar

in E

rro

rs

1999 Topics2000 Topics

Dragon1ResultsNWT+BNasr,

Reference Boundary, Multilingual Texts

InterestingReversal

Effects of Automatic BoundariesOn Detection Performance

– Multilingual Topic Detection– Newswire+Broadcast News ASR– Reference Vs. Automatic Story Boundaries

0.1

1

Dragon1 Dragon2 UMass1

Norm

aliz

ed D

ete

ctio

n

Cost

Reference Boundaries

Automatic Boundaries

19%, 21% and 41%relative increase in

cost respectively

TDT Segmentation Task

Transcription:

text (words)Story:Non-story:

One Participant: MITRE(For TDT 2000, Story segmentation is an integral part of the other tasks, not just a separate evaluation task)

System Goal:– To segment the source stream into its constituent stories, for all audio sources.

Story Segmentation Results

• Required Condition: – Broadcast News ASR

MITRE Segmentation Results

0.1

1.0

1999System

2000System

No

rma

lize

d S

eg

me

nta

tio

n C

os

t

English - 10KW Deferral

Mandarin - 15KC Deferral

TDT First Story Detection (FSD) Task

Two Participants:National Taiwan University and University of Massachusetts

System Goal:– To detect the first story that discusses a topic,

for all topics.

• Evaluating “part” of a Topic Detection system, (i.e., when to start a new cluster)

First Stories on two topics

Not First Stories

= Topic 1= Topic 2

UMass Historical PerformanceUsing Reference Story Boundaries

0.76 0.64

0.10

1.00

10.00

UMass

Norm

aliz

ed F

SD

Cost 1999 System

2000 System

First Story Detection Results

• Required Condition: – English Newswire and Broadcast News ASR transcripts– Automatic story boundaries– One file decision deferral

0.1

1

10

NWT_Bnasr,Ref. Bnd

NWT+Bnasr,Auto Bnd

NWT+Bnman,Ref. Bnd.

Nor

mal

ized

FS

D C

ost

NTU1

UMass1

RequiredCondition

1999 and 2000 Topic Set Differences in FSD Evaluation

For UMass there is a slight difference,but a markeddifference for the NTU system

Summary

• Many, many things remaining to look at– Results appear to be a function of topic size and topic

set in the detection task, but it’s unclear why.• The re-usability of last year’s detection system outputs enable

valuable studies• Conditioned detection evaluation should be replaced with a

“contribution to cost” model

– Performance variability on tracking training stories should be further investigated

– …and the list goes on

• When should the annotations be released?• Need to find cost effective annotation technique

– Consider TREC ad-hoc style annotation via simulation