Recent Advances in Automated Fact Checking...Claims Marked Up Hundreds of pages, Thousands of claims ... Veriﬁcation Takes Weeks! Fact Checking @ IEA Claim Veriﬁed/Refuted Data

Recent Advances in Automated Fact

CheckingImmanuel TrummerCornell University

Automation & Fact Checking


Lorem Ipsum ...

Lorem Ipsum ...

Lorem Ipsum ...

Lorem Ipsum ...

Lorem Ipsum ...

Lorem Ipsum ...

Lorem Ipsum ...

Lorem Ipsum ...

Claim

IdentifyingCheck-Worthy

Claims


Lorem Ipsum ...

Lorem Ipsum ...

Lorem Ipsum ...

Lorem Ipsum ...

Lorem Ipsum ...

Lorem Ipsum ...

Lorem Ipsum ...

Lorem Ipsum ...

Claim Check

Lorem Ipsum ...

Lorem Ipsum ... Lorem

Ipsum ...

Lorem Ipsum ...

Lorem Ipsum ...

Lorem Ipsum ...

Lorem Ipsum ...

Lorem Ipsum ...


Claims

MatchingClaims toChecks

Verification


Lorem Ipsum ...

Lorem Ipsum ...

Lorem Ipsum ...

Lorem Ipsum ...

Lorem Ipsum ...

Lorem Ipsum ...

Lorem Ipsum ...

Lorem Ipsum ...

Claim Check

Lorem Ipsum ...

Lorem Ipsum ... Lorem

Ipsum ...

Lorem Ipsum ...

Lorem Ipsum ...

Lorem Ipsum ...

Lorem Ipsum ...

Lorem Ipsum ...


Claims

MatchingClaims toChecks

Talk Focus

Data-Driven Fact Checking

Claim

Verified/Refuted

Data Formula


Claim

Verified/Refuted

Data Formula(aka. SQL Query)


Claim

Verified/Refuted

Data Formula


Claim

Verified/Refuted

Data Formula

(In Natural Language)


Claim

Verified/Refuted

Data FormulaWhich data?



Claim

Verified/Refuted

Data FormulaWhich data? Which formula?


International Energy Agency


Paris-based intergovernmental organization

Established in 1974, 30 member countries

Mission: serve statistics on energy sector


Paris-based intergovernmental organization

Established in 1974, 30 member countries

Mission: serve statistics on energy sector

Claims Marked Up

Claims Marked Up

Hundreds of pages, Thousands of claims ...

Claims Marked Up

Hundreds of pages, Thousands of claims ...

Verification Takes Weeks!

Fact Checking @ IEAClaim

Verified/Refuted

Data Formula


Verified/Refuted

Data Formula

In 2017, global electricity

demand grew by 3% ...


Verified/Refuted

Data Formula



Electricity/Global/2017



Verified/Refuted

Data Formula




Electricity/Global/2016 D17/D16=1.03


Verified/Refuted

Data Formula






Verified/Refuted

Data Formula





The "Infodemic"

Fighting the Infodemic



Fighting the InfodemicClaim

Verified/Refuted

Data Formula

France has

more cases than US

F>UCDC/Confirmed/France

CDC/Confirmed/US


Verified/Refuted

Data Formula

France has

more cases than US

F>UCDC/Confirmed/France

CDC/Confirmed/US


Verified/Refuted

Data Formula

France has

more cases than US

CDC/Confirmed/France

CDC/Confirmed/USF>U

Other Use Cases

• Data journalism

• Business reports

• Scientific papers

• ...

(Demo)Verifying text summaries of relational data sets.

SIGMOD 2019S. Jo, I. Trummer, W. Yu, X. Wang, C. Yu, D. Liu, N. Mehta.

Challenges

Challenges

Text-Data Inconsistency... ...

American ...

Challenges

Multi-Claim Sentences


American ...

ChallengesContext

Multi-Claim Sentences


American ...

Fully Automated CheckingClaim

Translation

Formula

Evaluation

Fully Automated CheckingClaim

Translation

Formula

Evaluation

May go Wrong!

Semi-Automated CheckingClaim

Translation

Formula

Evaluation

Prob

abili

ty

Analyze Data Structure

country beer_servings spirit_servings ...... ... ... ...

Germany 346 117 ...... ... ... ...

USA 249 158 ...... ... ... ...



Germany 346 117 ...... ... ... ...

USA 249 158 ...... ... ... ...



Germany 346 117 ...... ... ... ...

USA 249 158 ...... ... ... ...

CountryGermany

USABeer Serving Spirit

... Keywords



Germany 346 117 ...... ... ... ...

USA 249 158 ...... ... ... ...

United States

America

CountryGermany


...

U.S.

Keywords

Synonyms......



Germany 346 117 ...... ... ... ...

USA 249 158 ...... ... ... ...

United States

America

CountryGermany


...

U.S.

Keywords

Synonyms......

Matches

Analyze Sentence Structure




Consider Text Structure

Claim Sentence

Paragraph


Claim Sentence

Section

Paragraph


Claim Sentence

Chapter

Section

Paragraph


Claim Sentence

Chapter

Section

Paragraph


Claim Sentence

Integrate Surrounding Keywords

Understand the Author

Translate Text


Translate Text

Infer Topic

ClaimTranslationHypothesis


Translate Text

Infer Topic

DocumentTopic

Hypothesis

ClaimTranslationHypothesis

(Demo)Verifying text summaries of relational data sets.

SIGMOD 2019S. Jo, I. Trummer, W. Yu, X. Wang, C. Yu, D. Liu, N. Mehta.

System OverviewLorem Ipsum ...

Data Analysis

Text Analysis

Topic Analysis

Automated AccuracyC

orre

ctne

ss C

hanc

e

0

25

50

75

100

Nr. Proposed Formulas1 2 3 4 5 6 7 8 9 10

Automated AccuracyC

orre

ctne

ss C

hanc

e

0

25

50

75

100


(Billions of possible formulas)

Automated AccuracyC

orre

ctne

ss C

hanc

e

0

25

50

75

100



Want Auto-Suggestions

Automated AccuracyC

orre

ctne

ss C

hanc

e

0

25

50

75

100



Need Human Feedback

Want Auto-Suggestions

User Study ResultsC

laim

s pe

r Min

ute

0

0.4

0.8

1.2

1.6

Verification Method

With Tool Without Tool

6x Speedup!

Mistakes Discovered

11% of claims were incorrect

7% average error of claim value

Analyzed 50 articles from major data journalism venues

Scaling It UpLorem Ipsum ...

AggChecker


AggChecker

Lorem Ipsum ...Lorem

Ipsum ...Lorem Ipsum ...Lorem

Ipsum ...Lorem Ipsum ...


AggChecker





AggChecker





Scrutinizer




Scaling It UpLorem Ipsum ...Lorem



Ipsum ...




Ipsum ...




Ipsum ...

Learning Data Mapper




Ipsum ...Ask!

Learn





Ipsum ...





Ipsum ...


Interface Optimization

Interface OptimizationWhich Question?


Which Options?


Which Options?


Which Options? Which Order?




Ipsum ...


Interface Optimizer




Ipsum ...


Interface Optimizer




Ipsum ...


Interface Optimizer

$$

$$

$$ $

$

$$

$ $

$$ $

$ $ $

$$

$

$

$

$

$

Verify Cheapest Claims First?

$$

$$

$$ $

$

$$

$ $

$$ $

$ $ $

$$

$

$

$

$

$


?

???? ?

?

?

??

?

?

?

??

?

? ?

?

?

Verify Interesting Claims First?

$$

$$

$$ $

$

$$

$ $

$$ $

$ $ $

$$

$

$

$

$

$


?

???? ?

?

?

??

?

?

?

??

?

? ?

?

?

Verify Interesting Claims First?

Consider Text Structure?




Ipsum ...


Interface Optimizer

Claim Ordering

User Study @ IEAC

laim

s pe

r Min

ute

0

0.3

0.6

0.9

1.2

Verification Method

With Tool Without Tool

> 2x Speedup!

(Demo)Scrutinizer: fact checking statistical claims.

VLDB 2020G. Karagiannis, M. Saeed, P. Papotti, I. Trummer.

CoronaCheck Impact

CoronaCheck Impact

12,000 Users

Team

I. Trummer S. Jo G. Karagiannis N. Mehta D. Liu

W. Yu C. Yu X. Wang P. Papotti M. Saeed

Cornell Faculty PhD @ Cornell PhD @ Cornell Ugrad @ Cornell Ugrad @ Cornell

Ugrad @ Cornell Scientist @ Google Scientist @ Google Eurecom Faculty PhD @ Eurecom

Conclusion• Data-driven fact checking

• Various use cases

• Presented two tools:

• AggChecker

• Verifying text summaries of relational data sets.SIGMOD 2019 (ArXiV 2018).

• Scrutinizer

• Scrutinizer: fact checking statistical claims. VLDB 2020.

Questions?

www.itrummer.org

[email protected]

Documents

Recent Advances in Automated Fact Checking...Claims Marked Up Hundreds of pages, Thousands of claims ... Veriﬁcation Takes Weeks! Fact Checking @ IEA Claim Veriﬁed/Refuted Data