VizByWiki: Mining Data Visualizations from the Web to Enrich News Articles

AllenYilunLin1,JoshuaFord2,EytanAdar3,andBrentHecht1

1NorthwesternUniversity;2AppleInc.(workdoneattheUniversityofMinnesota);3UniversityofMichigan allen.lin@eecs.northwestern.edu,ford0420@umn.edu,eadar@umich.edu,bhecht@northwestern.edu

ABSTRACT Datavisualizationsinnewsarticles(e.g.,maps,linegraphs,barcharts)greatlyenrichthecontentofnewsarticlesandresultinwell-established improvements to reader comprehension.However,existingsystemsthatgeneratenewsdatavisualiza-tionseitherrequiresubstantialmanualeffortorarelimitedtovery specific typesofdatavisualizations, therebygreatly re-strictingthenumberofnewsarticlesthatcanbeenhanced.Toaddressthisissue,wedefineanewproblem:givenanewsarti-cle, retrieve relevant visualizations that already exist on theweb.Weshowthatthisproblemistractablethroughanewsys-tem,VizByWiki,thatminescontextuallyrelevantdatavisualiza-tionsfromWikimediaCommons,thecentralfilerepositoryforWikipedia.Usinganovelgroundtruthdataset,weshowthatVizByWikicansuccessfullyaugmentasmanyas48%ofpopu-lar online news articles with news visualizations. We alsodemonstratethatVizByWikicanautomaticallyrankvisualiza-tionsaccording to theirusefulnesswithreasonable accuracy(nDCG@5of0.82). Tofacilitatefurtheradvanceson“newsvis-ualizationretrievalproblem”,wereleaseourgroundtruthda-tasetandmakeoursystemsourcecodepubliclyavailable.

KEYWORDS: news articles; Wikimedia Commons; user-generated content; data visualizations; peer production; Wikipedia

1 INTRODUCTION Datavisualizationshavebecomeanincreasinglyprominent

partof thenewslandscape[2,10,26]. Indeed,TheNewYorkTimes,TheWashingtonPost,TheWall Street Journal andTheGuardianallnowoperateentireteamsthatdesignandpublishdata visualizations [33]. Recent research suggests that thistrendisquitebeneficialforthereader:empiricalevidencehasshownthatdatavisualizationscanmakecomplexrelationshipsinnewsarticleseasiertocomprehendandcanprovidecriticalcontextfornewsnarratives[33].Moregenerally,couplingtext

with data visualizations has been shown to improve under-standing and recall over either text or visualizations alone[5,8,15](inaccordancewiththewell-establisheddual-codingtheoryfromcognitivescience[31]).

Whilenewsdatavisualizationsarehighlybeneficialandingreatdemand,creatingthemrequirestime,moneyandexper-tise.Thismeansthatlocalnewsorganizationsusuallycannotafford to make data visualizations, and even large nationalnewsoutletscanonlycreatethemforatinyfractionofarticles.Researchers from information visualization and data-drivenjournalism [2] have developedmany tools to help automatethisprocess.However,aswediscussbelow,thesesystemsei-therstillrequiresubstantialhumanintervention(e.g.,[23,30,31,41])oronlyworkwithveryspecifictopicaldomains(e.g.,financialreports[11])orvisualizationtypes(e.g.,maps[8]).

Inthispaper,weproposeanalternativeandtractableap-proachtoautomaticallyaddcontextually-relevantdatavisuali-zationstonewsarticleswithnohumanintervention.Moreover,unlike existing techniques, our approach has very few con-straintswithrespecttotopicaldomainorvisualizationtype.

Ourapproachisbasedonanovelinsight:thereisoftennoneedtocreatevisualizationsfromthegroundupbecauselargenumbersofdatavisualizationsalreadyexistinWikimediaCom-mons.WikimediaCommons,or“theCommons”[40],istheme-diarepositoryusedbyWikipediaeditorsandhasover41mil-lionimages,anon-trivialpercentageofwhicharedatavisuali-zations created to supportWikipedia articles.Moreover, thevisualizationsintheCommonshaveredistribution-friendlyli-censes,makingtheCommonsaparticularlyappealingresourcefornewspublishers.Tothebestofourknowledge,thispaperisthefirsttorecognizeandleveragetherichcorpusofdatavisu-alizationsthatexistsintheCommons.Aswediscussbelow,wehopeourresearchcansupportfurtherinquiryintothevalueofthisimportantrepository,bothwithinthenewsvisualizationcontextandbeyondit.

Inmoreformal and general terms, thispaperdefinesanewproblemthatwecallthenewsdatavisualizationretrievalprob-lem.Givenanarbitrarynewsarticle,thegoalofthenewsdatavisualizationretrievalproblemistoautomaticallyretrieverel-evant, pre-existing data visualizations to support the articlefromagivenrepository(inourcase,theCommons).

This paper is published under the Creative Commons Attribution 4.0International(CC-BY4.0)license.Authorsreservetheirrightstodisseminatethework on their personal and corporate Web sites with the appropriateattribution.WWW2018,April23-27,2018,Lyon,France.© 2018 IW3C2 (International World Wide Web Conference Committee), published under Creative Commons CC BY 4.0 License. ACM ISBN 978-1-4503-5639-8/18/04. DOI: https://doi.org/10.1145/3178876.3186135

Wedemonstratethatthenewsdatavisualizationretrievalproblemistractablethroughanew,end-to-endsystemcalledVizByWiki1.Asanexample,considerFigure1,whichshowsanarticleaboutadropinoilpricesthatincludesalinechartcre-ated by journalists (Figure 1a). For this article, VizByWikiminedtheCommonsandretrievedalinechartandamapandrankedthemaccordingtotheirusefulnesstothisarticle(Fig-ure1b).Comparedtothedatavisualizationsintheoriginalar-ticle,thelinechartpresentsthesamevariables(BrentCrudeoilpriceovertime,althoughforalongertimerange).Additionally,thethematicmapwasratedasuseful to thearticleby studyparticipants,butwasnotincludedbythenewspublisher.

ToformallyevaluateVizByWikiandtolearnthefactorsthatpredictusefuldatavisualizations,wedesignedacrowdsourc-ingtasktocollecthumanratingsontheusefulnessofthevisu-alizationsretrievedbyVizByWiki(whichresultedintheuseful-nessscoreforthemapinFigure1).WedemonstratethatViz-ByWikicanenricharound50%ofpopularonlinenewsarticleswithatleastonesomewhatusefuldatavisualization.Moreover,using this ground truth dataset, we gained an initial under-standing of the factors that determine useful data visualiza-tionsfornewsarticlesandtrainedVizByWikitorankvisualiza-tionsaccordingtothisunderstanding.Asweshowbelow,Viz-ByWiki’srankingaccuracyisgood,withannDCG@5of0.82.

Overall, thispaperboth contributesanewproblem (thenews data visualization retrieval problem) and demonstrates

1 Link to the system demo and source code repository: http://www.psagroup.org/pro-jects/vizbywiki.

thattheproblemistractablewithanend-to-endsystemcon-tribution(VizByWiki).Wealsomaketwoothercontributionswhose impactmaygeneralizebeyond thenewsvisualizationcontext.First,thispaperhelpstodemonstratethetremendouspotentialofWikimediaCommons,aresourcethat–unlikeitssister project Wikipedia –remains largely untapped by thecomputingresearchcommunity.Second,akeycomponentofVizByWiki involvesautomaticallydistinguishingvisualiza-tionsfromnon-visualizations,anewchallengethatcouldberelevanttootherdomains(e.g.,imageclassification).Inthispa-per,weshowthatthischallengecanbeaddressedwithaF1-scoreof0.91bysimplyusingapre-trainedConvolutionalNeu-ralNetwork(CNN).

Thestructureofthispaperfollowsbestpracticesinthesys-temsresearchgenre(e.g.,[8,11,14,32])byfirstprovidinganoverviewofoursystem’scomponentsandthenoutliningthenovelchallengesfacedineachcomponentanddescribinghowthechallengeswereaddressed.Priortothis,however,webeginbelowbyhighlightingrelatedwork.

2 RELATED WORK Inthissection,wereviewthethreeresearchareasthatmost

directlymotivatedthiswork:1)automatedtextillustration,2)othersystemsthatgeneratenewsdatavisualizations,and3)methodsfordatavisualizationclassification.

Figure 1: (a) A BBC article about the oil price drop after producers failed to agree on an output freeze and (b) the data visual-izations retrieved by VizByWiki. In (b), the retrieved data visualizations are ranked in descending order according to readers’ perceived usefulness.

2.1 Automated Text Illustration Oneareaoftheliteraturethatprovidedkeymotivationfor

thisworkistextillustration.Textillustrationisaconstrainedimageretrievalproblemthatfocusesonretrievingillustrativeimagesforlongtexts.Someofthisworkhasfocusedspecificallyonillustratingnewsarticleswithmultimediafiles.Forinstance,LiandHai[17]illustratenewsarticleswithimagesfromFlickr.NewsMap [19] on fusing image search results frommultipleshortqueriesthataregeneratedfromnewsarticles.Similarly,Delgado and Joao’s system [5] constructed a visual story ofnews articles by finding sequence of images. BreakingNews[28]leveragedCNNtolearntomatchtheoriginalimagesandthetextofthenewsarticle..

SometextillustrationsystemsalsouseimagesonWikipe-dia,butfocusonasmall-scale,self-curatedsampleoftheseim-ages(e.g.,[37]).TheonlyworkthatconsidersallWikipediaim-agesisbyAgrawaletal.[1],whodesignedabookillustrationsystem to enrich textbooks for developing countries. TheirmethodessentiallyranksWikipediaimagesaccordingtoscorescomputedfromthetokenoverlapbetweenthekeywordsofthetextandthedescriptionsoftheimage.Duetoitssimilaritytoourproblem(usingWikipediaimagesforlongtext),weimple-mentedtheiralgorithmanduseditasabaselineforcompari-sonintheevaluationsection,showingthatweoutperformthisapproachbyasolidmargin.

However, thenews data visualization retrieval problem isalsofundamentallydifferentfromtextillustrationproblem.Forexample,consideranewsarticleaboutairpollutioninBeijing.Inthetextillustrationproblem,theidealsolutionwouldbeaphotodepictingsmoginBeijing.However,inthenewsdatavis-ualizationscenarioanappropriateimagewouldbeabarchartshowingthenumberofdaysineachmonththathaveaPM2.5readinggreaterthanthesafe level.Moregenerally, thenewsvisualizationretrievalproblemisinterestedinaddingnewin-formationtocontextualizeastory,notfindingaphotothatwilldepictexactlywhatiswritteninthetext.Similarly,visualfea-turesplayamuchdifferentroleinthedatavisualizationprob-lem;datavisualizationsofthesametypecouldshareverysim-ilarvisualfeaturesbutcoversubstantiallydifferenttopicsanddata.Ourmethodologicalchoicesreflectourconsiderationsofthese importantdifferences,and isonereasonourapproachoutperformsthebaselinemethodfortextillustration.

2.2 Generating Data Visualizations for News Several research projects in the information visualization

anddata-drivenjournalismdomainshavesoughttoautomatethe ground-up generation of news data visualizations. How-ever,thesesystemseither(1)stillrequireasubstantialdegreeofmanualeffortor(2)focusonanarrowdomainwithinnewsarticles.AnexampleoftheformercaseistheMuckRakersys-tem[23],whichprovidesauserinterfacetohelpfindandvisu-alizestructureddatafromdatabasesthatisrelevanttoagivennewsarticle.Researchsystemsthatautomatethecreationofspecificcategoriesofnewsvisualizations includeContextifier

[11],whichproducesannotatedstockvisualizationsforfinan-cialnewsarticles.However,thesystemislimitedtofinancialnewsandonlygenerateslinecharts.NewsViews[8]isbuilttoautomatically generate geovisualizations for news articlesthroughapipelinethatinvolvesidentifyingtoponymsandtop-ics, findingrelevanttabulardatasetsandcreatinga thematicmap.LikeContextifier,NewsViewsfocusesonlyonaspecifictypeofdatavisualization(thematicmaps)andaspecifictypeofarticle(thosewithstronggeographicelements).NewsViewsisfurtherlimitedbythesmall,manually-curatedstructuredda-tasetsfromwhichthesystemgeneratesmaps.Incomparison,VizByWikidoesnotrequirehumaninterventionandoperateswithoutthelimitationsonthetypesofdatavisualizations,thetypesofnewsarticles,andthediversitiesofcurated datasets.

2.3 Classifying Data Visualizations Asdescribedbelow,anecessarystepinVizByWikiisdiffer-

entiatingbetweendatavisualizations(e.g.,barchartsandpiecharts)andotherimages(e.g.,photos,engineeringdiagrams).Thisproblemisrelatedtotheproblemofidentifyingdifferentvisualizationmark types,e.g., separatingbar charts frompiecharts.Thisisataskaddressedinseveralpapersthattrytore-coverthedatabehindstatisticalcharts.Forinstance,ReVision[32]classifiesvisualizationtypesusingacombinationof tex-tualfeaturesfromOCRandlow-levelvisualfeatureswhichcap-ture prominent patterns in the images. More recently, deeplearning has been used in this problem space. For example,ChartSense[14]includesamarktypeclassifierthatwastrainedfromscratchusingtheGoogLeNetarchitecture.Similarly,Heeretal. [27]builtavisualizationtypeclassifierbyfine-tuningapre-trained CNN and, using the same fine-tuning technique,FigureSeer[35]successfullytrainedaclassifiertodistinguishdifferenttypesofresultsfiguresinresearchpapers.

While VizByWiki faced a different problem than the re-searchdescribedabove(i.e.determiningwhetheranimageisadatavisualizationvs.distinguishingbetweentypesofvisualiza-tions), previousworks’ success using pre-trained CNNs pro-videdkeymethodologicalguidanceforourapproach.

3 SYSTEM OVERVIEW Inthissection,wefirstprovideanoverviewofVizByWiki’s

high-leveluserexperience.WethendescribeVizByWiki’sbe-hind-the-scenes system architecture. Finally, we discuss thedifferenttypesofdatasetsthatareusedtobuildVizByWiki.

3.1 User Experience TheprimaryaudienceforVizByWikiisthemillionsofpeo-

plewhoreadnewsonline.Forthisaudience,wehavebuiltaworkingbrowserplug-in(demonstratedasawebapplicationintheURLonpage2)thatimplementsallthetechniquesbelow.The plug-in processesunstructured text from a newsarticleandpresentsuserswithdatavisualizationsthatarerankedbytheirusefulnessalongsidethenewsarticle(asshowninFigure1b).ApotentialsecondaryaudienceforVizByWikiisthosein

thedata-drivenjournalismcommunity[26].Forthisaudience,VizByWikicouldbeusedasanexploratorytooltoinformthedesignofcustomizeddatavisualizations(aswellasperhapsinthejournalisticdiscoveryprocess).

3.2 System Architecture Inthissection,weprovideahigh-leveloverviewofthesys-

temarchitectureofVizByWiki.AsshowninFigure2,VizByWikiconsistsofathree-stagepipeline:1)topicfiltering,2)datavis-ualization identificationand3)datavisualization ranking. InStage1,VizByWikiusesWikipediaarticlesthatarerelevanttothenewsarticle,andthatcontainWikimediaCommonsimagesasaproxytofilterouttopicallyirrelevantimages.Tofindrele-vantWikipediaarticles,VizByWiki firstappliesentity linkingtechniques to identify Wikipedia concepts/articles that arementionedinthenewsarticletext.TheresultingWikipediaar-ticlesetisthenexpandedbyfindingthemostsemanticallyre-lated Wikipedia articles using semantic relatedness (SR)measures.AllimagesintheexpandedsetofWikipediaarticlesareextractedanddeliveredtoStage2.

InStage2,theextractedimagesarefilteredtoisolatedatavisualizations from non-data visualizations. VizByWiki em-ploys a two-step filtering approach that consists of bothstraightforward heuristics and CNN-based transfer learningtechniques.TheoutputofStage2–asetoftopicallyrelevantdatavisualizations–isthenprocessedbythedatavisualizationrankerinStage3.Therankerwastrainedusinganewground

truthdatasetthatwecollected.Thedatasetconsistsofhumanjudgmentsdescribingwhichdatavisualizationsareuseful towhichnewsarticles.

3.3 Dataset 3.3.1 Wikimedia Commons and English Wikipedia

VizByWikiaugmentsnewsarticleswithdatavisualizationsfromWikimediaCommons.WikimediaCommonsisthecentralmediafilesrepositoryforalltheWikipedialanguageeditionsandisthelargestrepositoryoffreelylicensededucationalcon-tent in theworld [40].We choseWikimedia Commons overotherpopularmediarepositories(e.g.,Flickr)becauseitcon-tainsalargenumberofdatavisualizationsduetoitsencyclo-pedicfocus.UsingahighlyaccuratedatavisualizationclassifierdevelopedaspartofVizByWiki(seebelow),weestimatedthatthe Commons contains roughly threemillion data visualiza-tions.Wewerealso attracted to the Commons’ licensing re-gime,asthismitigatesanyreal-worldlegalbarrierstoanycom-mercialuseofVizByWiki.Asdiscussedbelow, theCommonsprovedtobeapowerfulrepositoryofdatavisualizationsthatwebelievecanbeleveragedincontextsbeyondthisproject.

Anobstacletodirectly leveragingWikimediaCommonsisitspoormetadata.TheWikimediaCommonscommunityitselfhasstatedthatimagesare“describedonlybycasualnotation,makingitdifficulttofullyexploreandusethisremarkablere-source”andeventhis“casualnotation”doesnotexistforallim-ages [40].Moreover,while the Commonsbegana long-termprojecttostandardizeitsmetadatain2017,thisprocessisfarfromfinished[40].

Toaddressthisobstacle,weusedtheEnglishWikipediatoaugmenttheknowledgethatwehaveaboutCommonsimages.TheCommonsisthedominantimagerepositoryformostWiki-mediaprojects,includingWikipedia[42].Assuch,EnglishWik-ipediaarticlesandthetextaroundCommonsimagesthatap-pearinthesearticlescanprovidenaturalsemanticembeddingsforCommons images.Wenote that thismaybeauseful ap-proachtoseedthemetadatastandardizationprocessinthefullCommonsaswell.However,asideeffectofthisapproachisthatitdoeslimitourpoolofpotentialdatavisualizationstothosethatappearinatleastoneEnglishWikipediaarticle.

To process the full English Wikipedia, we leveragedWikiBrain[34],asoftwareframeworkthatprocessesWikipe-diaXMLdumpsandprovidesaccesstoarangeofWikipedia-based algorithms (including the semantic relatedness algo-rithmsweused).Forourstudies,weusedtheJune23,2017EnglishWikipediadump.

3.3.2 News Articles Datasets ToperformrealisticexperimentsonVizByWiki,wesampled

twodatasetsofpopularonlinenewsarticlesfrommajornewsoutlets.Thefirstnewsarticledataset(whichwecalltheadhocdataset)wasoriginallycollectedduringApril2016(foranear-lier project) and consists of 40 popular online newsarticlessampledarbitrarilyfromthehomepagesofvariouslargenews

Figure2.OverviewofVizByWiki’sarchitecture

outletsincludingCNN,FoxNews,BBC,andTheNewYorkTimes.Someofthesenewsarticlesareaccompaniedwithdatavisual-izationsdesignedbyexperts.Weusedthisdatasetprimaryforearlyfeasibilitytesting.

Ourcoredataset(calleduniform)wascollectedduringJuly2017andconsistsof60articlessampledfromthedifferenttop-icalcategoriesprovidedbypopularnewsoutlets.ThesearticlesweresampledthroughtheRSSfeedsofFoxNewsandCNN,bothof which are organized into topics including “World”, “U.S.”,“Business”,“Politics”,“Technology”,“Health”,“Entertainment”,and“Travel”.Werandomlysampledthesamenumberofarti-cles fromeachcategory.Different fromthepreviousdataset,thesepopularonlinearticleswereselectedwithoutconsidera-tionofwhethertheyalreadycontainedvisualizations.Thepur-poseofthisdatasetistoevaluatethegeneralutilityofthesys-teminanecologicallyvalid fashion,i.e.howwell itcouldre-trievedatavisualizationsforarbitrarynewscontent.

Insomeevaluations,wealsocombinedthesetwodatasetsintoasingle100-articledataset.Wecallthisdatasetcombined.

4 STAGE 1: TOPIC FILTERING ThefirststageintheVizByWikipipelineisidentifyingtopi-

callyrelevantWikimediaCommonsimagesforaninputnewsarticle.Here,weusedWikipediaarticlesthatcontaintheWiki-mediaCommonimagesasproxiesforthetopicoftheimages.VizByWikideploysatwo-stepprocessthatleveragesbothwik-ification[4,24]andqueryexpansiontechniquestoidentifytheappropriateWikipediaarticles.

4.1 Wikification Wikificationinvolvesdisambiguatingnamedentitiesinun-

structuredtexttoWikipediaarticles[24].Forexample,thesys-temcanrecognizethat“travelban”intherecentnewsarticlerefersto“ExecutiveOrder13769”(andlinkedtotheassociatedWikipedia page) and not two separate words (“travel” and“ban”).Weusea techniquedevelopedbyNoraset et al. [25],whichuseshybridrule-basednamedentityrecognitiontodis-covertermsandalearnedmodeltodisambiguatetheircorre-spondingWikipediaentities.

Theoutputofthewikificationprocesscontainsmanyenti-tiesthataretriviallyrelatedtothemaintopicofthearticle(e.g.,countriesthatwerementionedinthenewsarticlebutnotin-cludedinthelistofbannedcountries).Tofilterouttheseenti-ties,wecomputedthesemanticrelatedness(SR)betweenthese“wikified”entitiesandthenewsarticlecontentandonlykeptentitiesthatarehighlyrelated to thearticle content. Specifi-cally,weleveragedWikiBrain’simplementationoftheExplicitSemantic Analysis semantic relatedness algorithm (ESA) [7]whichmapswordsintoWikipedia-concept-basedembeddingsandcomputessemanticrelatednessasthecosinesimilarityoftheembeddings.WeonlykeptwikifiedentitiesthathavehadanESASRgreaterthan0.8(outof1)withthenewscontent.InWikiBrain,anSRscoreof0.8effectivelymeansthatthisscoreisat80thpercentileofallSRscores[39].

4.2 Query Expansion TheentitiesthatareoutputfromtheSRfilterareusedas

“seed queries” for retrieving Wikimedia Commons images.However,queryexpansionisalsonecessaryduetoWikipedia’sorganizationanditsrelationshiptothearticlesonwhichdatavisualizationsaboutspecifictopicsappear.For instance,con-tentaboutaconceptisoftencontainednotonlyinthe“mainarticle”abouttheconcept(e.g.,the“UnitedStates”article)butalso in “sub-articles” (e.g., thearticles “Historyof theUnitedStates”, “Geography of theUnited States”, “American Litera-ture”andsoon)[21].Thisissuehasimportantimplicationsforour problem. Consider again a news article that covers theTrumpadministration’schangestoU.S.travelpolicy.Theterm“immigrationpolicy”inthearticlemightbecorrectlydisambig-uatedtotheWikipediaarticle“ImmigrationPolicy”.However,thislinkagewillmisstheWikipediaarticle“ImmigrationPolicyofDonaldTrump”,whichisasub-articleof“ImmigrationPol-icy” and contains useful statistical charts related to foreign-bornworkersintheUSlaborforce.

Toaddressthisissue,VizByWikiexpandsthequeriestoin-cludearticlesthatarehighlysemanticallyrelatedtowikifiedentitiesusingESA.Similartothecriteriaabove,toensurethattheseexpandedentities are highly relevant,we only includethemiftheyhaveanSR>0.8withthenewsarticle.

Finally, theoutputofStage1 consistsofWikimediaCom-mons images that are extracted from theWikipedia articlesidentifiedfromtheabovetwosteps.

4.3 Evaluation Wevalidatedthefeasibilityofourtopicfilteringapproach

usingthecombined dataset.Table1 reports thatonaverage,our approach could “wikify” 6.3Wikipedia entities for eachnewsarticle(aftertheSRfilter).Table1alsoshowsthatqueryexpansion with semantic relatedness successfully increasedthisnumberto10.3.FromtheseWikipediaarticles,Stage1ex-tractedanaverageof69.6uniqueimage candidates foreachnewsarticle(6473uniqueimagesintotalforthecombinedda-taset).TheresultsinTable1demonstratethatourtopicfilter-ingapproachisaviableapproachtoretrievemanyimagecan-didatesforthelaterstagesofthesystem(althoughitdoesnotprovideaguaranteethatthisisthebestsuchapproach,atopictowhichwereturninDiscussion).

5 STAGE 2: IDENTIFICATION ThegoalStage2istoidentifythevisualization“needles”out

ofthelarge“haystack”ofdiverseimagesfromStage1.

Table 1. Performance of Stage 1: Topic Filtering

Avg. # of Wiki articles per news

Avg. # of images extracted per news article

Wikification 6.3 56.0 After query expansion 10.3 69.6

5.1 Problem Definition Prior literature [11] has identified that common types of

newsdatavisualizationsincludemaps,linegraphs,bargraphs,bubblecharts,scatterplots,treemapsandpiecharts.Lesscom-mon forms suchasareagraphsandVenndiagramsarealsobroadlyviewedasdatavisualizations [27,32]. InVizByWiki,weassumethatthedefinitionofdatavisualizationssubsumesalloftheabovetypes.

Asmentionedabove,informationvisualizationresearchershaveexaminedtheproblemofdistinguishingdifferenttypesofdata visualizations from oneanother.However, the problemwefacehere–separatingdatavisualizationsfromnon-datavis-ualizations–presentstwomajorchallengesthatmakeitdis-tinctfrompriorwork.Firstandforemost,unlikeourproblem,researchonvisualizationtypeclassificationusuallystartswithacorpuscontainingonlydatavisualizations[14,27,32,35].Bycontrast, Wikimedia Commons containsmuch more diversetypesofimages,includingmanynon-datavisualizationimagesthatsharevisualsimilaritieswithdatavisualizations(e.g.,en-gineeringdiagrams,photosofpapermaps,logos).Second,duetothecrowdsourcednatureofWikimediaCommons,evenvis-ualizations of the same type can be visually heterogeneous,whichincreasesthedifficultyofclassification.Datasetsusedinthetypeclassificationproblemdonotcontainthisdegreeofheterogeneity;theyareoftenmanuallycuratedbyresearchers[32],designedinthesamefashionfromthesametool[27],orcarefullygeneratedtospecificstandardsbyprofessionals[35].

Toaddressbothofthesechallenges,VizByWikicontainsatwo-stepdatavisualizationidentifierthatleveragesbothtex-tualandvisualfeatures.Weexplaineachstepindetailbelow.

5.1 Keyword Filtering Inthefirststepofourdatavisualizationidentificationpro-

cess,anaivekeywordfiltereliminatesobviousnon-datavisu-alizations using rule-based heuristics. The filter uses textmetadata from both Wikipedia and Wikimedia Commons.MetadataconsideredincludesimagecaptionsfromWikipedia,filedescriptionsandcategorytagsfromWikimediaCommons,andmachine-generatedEXIFmetadata.Todevelopourheuris-tics,oneresearcherwentthroughaseriesofexampleimagesand identifiedkeywords that indicateobviousnon-visualiza-tions. These included “photograph,” “picture,” “image,” “fea-tured,”“photo,”“portrait,”“roadsign,”and“coatsofarms.”Wealso excluded images that contain camera EXIF information(e.g.,acameramake/model),whichclearlyindicateanimagefromadigitalcameraratherthanadatavisualization.Onthecombineddataset,thekeywordfilteridentified5718imagesasobviousnon-datavisualizationsoutofthe6473imagesfromStage1’soutput.This leftuswith755candidate imagesthatwerepotentiallydatavisualizations.

5.2 Image Classifier In the second stepofourdatavisualization identification

process,we trainedan imageclassifierusingvisual features.

Whilekeywordfilteringeffectivelyscreensoutmanyobviousnon-datavisualizations, theoutputof thekeywordfilter stillcontainsmanyimagesthatarenotdatavisualizations.Thisisprimarilytheresultof three issues:(1)sparseness:manyim-ageson theCommonshave limitedmetadata,(2)errors: themetadatathatisavailablecanbeinaccurate,and(3)coverage:itisintractabletodevelopacompletesetoffilterkeywords.

Assuch,toincreaseprecision,weusedapre-trainedconvo-lutional neural network (CNN) to design an image classifierthatdifferentiatesbetweendatavisualizationsandother im-ages.CNNshavebeenshowntobeeffectiveinthedatavisuali-zationclassificationtasksdescribedinrelatedwork(e.g.,[14,27,35]),buttheyalsorequireenormousnumbersofgroundtruthimagesfortraining.Onewaytoaddressthisistoleveragethepoweroftransferlearningwithapre-trainedCNN,andweadopted this approach in VizByWiki. Specifically, we used apre-trainedCNNasafeatureextractor:weleveragedtheout-putfromthesecond-to-lastCNNlayerasavectorrepresenta-tionofeachimage,andfedtheserepresentationsintoatradi-tionalclassifier.Thisapproachhasbeensuccessfullyemployedonawiderangeofimagerecognitiontasks[29].

Inourimplementation,weusedtheInceptionV3CNNpre-trainedonImageNet[36]andasupportvectormachine(SVM)classifier.Becausethefeaturesfromthesecond-to-lastlayerofthepre-trainedCNNaresparseandhigh-dimensional(1024di-mensions),featureengineeringwasnecessarybeforetrainingonourrelativelysmallgroundtruthdataset(detailedbelow).WeappliedPrincipleComponentAnalysisandusedthetop-20principle components as features (which account for about51%variation).These20featureswerenormalizedtoaid inthetrainingprocess.

Toobtainagroundtruthdatasetthataccuratelyrepresentstheunderlyingdatadistributionofthisproblem,welabeledall755imagesthatwereoutputfromourkeywordfilteringstep.Sincemanuallydifferentiatingdatavisualizations fromotherimages is a relativelyunambiguous task(under the concretedefinitionofnewsdatavisualizationsenumeratedatthebegin-ningofthissection),oneresearchermanuallysortedourim-agesintoadatavisualizationclass(455images)andanon-datavisualization class (300 images).This datasetwas then splitintoadevelopmentset(50%),whichwasusedtotrainandtunethehyperparametersoftheSVM,andanevaluationset(50%),whichwasused toevaluate classifierperformance.Weusedgridsearchtotuneavarietyofhyperparameters.

ThebestperformingSVMclassifierwasfoundtoemployaradialbasisfunction(RBF)kernelwith𝛾 = 0.01and𝐶 = 100.Table2showstheresultsofthisclassifieronourtestdataset.We were able to achieve an average F1 score of 0.91 and

Table 2. Performance of image classifier

Class Precision Recall F1-score

non-dataviz 0.89 0.88 0.89

dataviz 0.93 0.91 0.91

avg 0.91 0.91 0.91

roughlyequallyhighF1scoresforboththenon-datavisualiza-tionclass(0.89)andthedatavisualizationclass(0.91).Theseareimportantresultsfortworeasons.First, theyrepresentamore-than-adequateoverallaccuracyforourVizByWikiproto-type,allowingustocontinuetotheStage3rankingtask.Sec-ond,theyindicatethatthevisualfeatureslearnedfordetectingobjects in natural scene images (one of themain ImageNettasks)areusefulforidentifyingdatavisualizations(whicharemostlycomputer-generatedwithverydifferentvisualcharac-teristics),afindingthatdeservesmoreexploration.

6 STAGE 3: RANKING InStage1,VizByWikiextractsimagesrelevanttoaninput

news article fromWikimedia Commons and in Stage 2, Viz-ByWikifiltersoutimagesthatarenotdatavisualizations.ThegoalofStage3–thefinalstage–istorankthedatavisualiza-tionsoutputfromStage2accordingtotheirusefulnesstothereader(asinFigure1b).Inthissection,wefirstdiscusshowweformulatedtheStage3problemintoa“learningtorank”prob-lem.Wethendocumenthowwecollectedanovelvisualizationusefulness ground truth dataset through crowdsourcing.Lastly,weusethisdatasettoconducttwoimportantevalua-tions:onefortheoverallsystem’sgeneral feasibilityandtheotherspecificallyfortheranker’sperformance.

6.1 Problem Formulation WeformulatedStage3asalearningtorankproblemsimilar

totheonethatistypicalforsearchengines:givenanewsarti-cle,ourgoalwastorankasetofdatavisualizationsbytheirusefulnesstothenewsarticle.Duetothenoveltyofthisprob-lem, we had to construct our own ground truth dataset. Tobuildthisdataset,weusedthedatavisualizationsoutputfromourworkinStage2andpairedthemwiththeircorrespondingnewsarticles.Wemanuallycorrectedallclassificationmistakesinordertoensureadata-visualization-onlydataset,allowingustofocuspurelyontherankingtask.Thisdatasetconsistsof572{newsarticle,candidatedatavisualization}pairs(Notethatonedatavisualizationcanbepairedwithmultiplearticles).

For each pair, we generated both textual and visual fea-tures.Specifically,thetextualfeaturesareasfollows:

• content-caption: The semantic relatedness score (com-putedbyExplicitSemanticAnalysis)betweenthenewsar-ticlecontentandthevisualizationcaptionwrittenbytheeditorsoftheWikipediaarticleinwhichtheimageisused(thearticleidentifiedintheStage1).NotethateventhoughmetadataforimagesissparseandcanbeinaccurateintheCommons,almostallimageshaveacaptionwhentheyareincludedinWikipedia.

• content-WPtitle:ThesemanticrelatednessscorebetweenthenewsarticlecontentandthetitleoftheWikipediaarti-clethatcontainsthecandidateimage

• title-caption:Thesemanticrelatednessscorebetweenthenewsarticletitleandthevisualizationcaption.

Thevisualfeaturesaremuchsimpler:

• CNNembed:Thesearethesamefeaturesthatareusedtotrainthe imageclassifierinStage2.Theyarethetop20principalcomponentsfromthe1024-dimensionfeaturesextractedfromapre-trainedCNN.

Forourrankingalgorithm,weusedthepopularRankSVM[13],whichemploysapairwisemethodthatistrainedtomini-mizethenumberofinversions.WeimplementedtheRankSVMasalinearkernelSVMusingthePythonpackagescikit-learn.

6.2 Collecting Ground Truth Ratings Foreach{newsarticle,candidatedatavisualization}pair,we

usedcrowdsourcingtocollectgroundtruthusefulnessratings.Thisdatasetallowedustoassessthefeasibilityofthenewsdatavisualization retrieval problem (Section 6.3) and to learn torankthedatavisualizationsforeachnewsarticle(Section6.4).

2.2.1 TaskUI. ThecrowdsourcingplatformweusedwasAmazonMechanicalTurk(MTurk).Figure3showstheUIofthetask (re-scaled for clarity). Upon accepting our task, acrowdworker(“Turker”)wasshownatasktutorialandanex-ampleisprovided.TheTurkerwasthenrandomlyassignedtooneofthe100newsarticlesinthecombinedcorpus.

Afterreadingthearticle,theTurkerwaspresentedwithallthe candidate data visualizations that are extracted by Viz-ByWikiforthearticle(outputofStage2)andwasaskedtorateeachvisualizationonascaleof0-3accordingtohowusefulthedatavisualizationis.EachdatavisualizationwasaccompaniedwithitsoriginalWikipediaarticlecaptionandcouldbeclickedtozoominiftheTurkerwantedtoexamineitsdetails.

Usefulnesswasassessedonafour-pointscale:0=“notuse-ful”;1=“somewhatuseful”;2=“useful”;3=“veryuseful”).Ausefulvisualizationwasdefinedasonethat“helpedexplainorprovidecontext”tothearticle.Weconsideredevaluatingtheretrievedvisualizationsonvariouslower-levelcharacteristics

Figure3.MTurktaskUIforusefulnessratingscollec-tion.Newsarticleattenuatedduetolength.

fromthevisualizationdomain,e.g.,expressiveness[22]andin-terestingness[8]–ratherthanusefulness.However,wedeter-minedthatusefulness,asdefinedabove,wouldmoredirectlycaptureoveralluserexperienceatthisstageoftheexplorationofthenewsdatavisualizationproblem.

2.2.1 Improving the quality of crowdsourced data. As iswell-knowninthehumancomputationdomain,crowdsourceddataissubjecttoqualityissuessuchasspam,errors,andbiases[12]. For instance, unless precautions are taken, somecrowdworkerswillfillinrandomanswerswithoutreadingthequestions in order tomakeasmuchmoney as possible in ashortperiodof time.Assuch,weimplementedthefollowingstrategiesinourtasktoimprovecrowdsourcingquality:

1) FollowingChangetal.[3],weensuredthatinthefinalda-taset,nooneworkerdidmorethan5%ofallthetasks.Thissimpletechniqueeffectivelyeliminateslarge-scalespam.

2) Tominimize the effect of personal biases and uninten-tionalerrors,werelyonredundancy[12].Foreachpairof{newsarticle,candidatedatavisualization},wecollected4ratingsandusedthemedianratingasthefinalrating.

3) WeaddedverificationquestionsassuggestedbyKitturetal. [16].Afterworkers read thenewsarticle andbeforetheystartedratingthedatavisualizations,workerswererequired toansweramultiple-choicequestionabout anobviousfactinthearticle.

Intotal,for572pairsof{newsarticle,candidatedatavisual-ization},wecollected2288ratings(4*572).Tofacilitatefurtheradvancementonthenewsdatavisualizationretrievalproblem,wearereleasingourgroundtruthdata(seesystemURLabove).

6.3 Evaluation 1: General Feasibility of the News Data Visualization Retrieval Problem

With theground truthdata, an importantquestiontoad-dressevenbeforetrainingtherankerregardsthefeasibilityofthebroadernewsdatavisualizationretrievalproblem.Inotherwords,canWikimediaCommonsprovideusefulvisualizationsforanon-trivialnumberofpopularnewsarticles?

Toanswerthisquestion,weusedtwometrics:1)foragivennewsarticle,howmany“good”datavisualizationscouldbere-trievedbyVizByWikiand2)howmanynewsarticlescouldbeaugmentedbyatleastone“good”datavisualization.Weusedtwodefinitionsfor“good”datavisualizations:1)datavisuali-zationswithTurkers’medianrankinggreaterorequalto1(i.e.somewhatuseful),2)datavisualizationswithTurkers’medianrankinggreaterorequalto2(i.e.useful).

Table3reportsthefeasibilityevaluationofVizByWiki.Mostimportantly, for our core news dataset uniform, VizByWikicouldretrieveatleastonesomewhatusefuldatavisualizationfor48.3%ofarticlesandcouldretrieveatleastoneusefuldatavisualizationfor21.7%ofarticles.Recallthatouruniformda-tasetcontainspopularonlinenewsarticlesthatarerandomlyanduniformlysampledfromdiversetopics.Hence,ourresultsdemonstratethatourapproachtonewsvisualizationretrieval

andtheuseoftheVizByWikisystemspecificallycouldresultinanywherefromone-fifthtoone-halfofpopularonlinenewsarti-clesbeingenhancedbyatleastonedatavisualization.Wenotethatweobservedroughlythesameresultsfortheadhocda-tasetaswell.

TounderstandthearticlesforwhichVizByWikididnotre-trieveatleastonesomewhatusefuldatavisualization,onere-searchercarefullyreadover20sucharticles.Wefoundthat1)most of thesearticles did not explicitly reference any struc-tureddatainthetextand2)someofthesearticlescoveredveryrecentbreakingnewsinwhichdatamightneedtobegatheredorupdatedquickly(e.g.,anearthquake,election).Wereturntobothofthesepointsbelow.

Lookingatthenumberofdatavisualizationsthatwerere-trievedforeacharticle,VizByWikiwasabletoretrieveanaver-ageof4.6somewhatusefulvisualizationsand3.5usefulvis-ualizationsforourcoredatasetuniform(includingallthezerosfor articles forwhich no visualization could be retrieved). Asimilartrendcanbeobservedfortheadhocdataset.However,theactual number of good data visualizations varies signifi-cantlyacrossarticles.Assuch,itisreasonabletoconcludethatalthoughourapproachlikelyisfeasibleforlarge-scalevisuali-zationenhancement,theperformanceofourapproachisbetterforsomearticlesthanforothers.TheseresultsalsosuggestthatforarticlesforwhichVizByWikicanproduceusefulvisualiza-tions,rankingisimportantasthereisoftenanon-trivialnum-berofvisualizationsperarticle.Weaddressthisrankingprob-leminthenextsub-section.

6.4 Evaluation 2: Performance of the Ranker Forourrankingexperiment,weusedthecombineddataset.

Weused50%ofthedatasetasadevelopmentset(trainingandhyperparametertuning),andheldout50%forevaluation.

Weassessedourdatavisualizationrankerusingtheconven-tional search engine evaluation approach involving thenDCG@kmetric.Toputourranker’sperformanceintocontext,we additionally implemented a ranking algorithm fromAgrawaletal.[1],whichinvolvedrankingWikipediaimagesfortheir relevance toa particular section from a textbook. Thismethodessentiallyreliesontokenoverlapbetween thekey-words of the text and the descriptions of the image. TheAgrawaletal.approachprovidesausefulbaselinetohelpun-derstandourranker’sperformance.

Table4showstheperformanceofourrankertrainedondif-ferent feature sets and compared to thebaselinemethodbyAgrawaletal.WecomputednDCG@k(wherek=3,5and7)for

Table 3. Feasibility evaluation of VizByWiki. Metric Uniform Ad hoc

% of articles with ³ 1 somewhat useful dataviz 48.3% 52.5%

% of articles with ³ 1 useful dataviz 21.7% 27.5%

avg. # somewhat useful dataviz per article 4.6(5.3) 4.2(3.9)

avg. # useful dataviz per article 3.5(4.0) 2.7(1.7)

Note: Standard deviations shown in parentheses where relevant.

each news article and the mean and standard deviation ofnDCG@kiswhatisshowninTable4.Theresultsinrow2showthatourrankertrainedontextualfeaturesaloneoutperformsthebaselinefrompriorworkbyasolidmargin.Forfurthercon-text,weadditionallyseethatthisversionoftheranker’sper-formance(e.g.,nDCG@5=0.82)iscomparabletopastresearchinthewebsearchdomainthatalsoinvolveddefiningnewprob-lems(e.g.,[6,38]).Theresultsinrow3arealsoquiteinforma-tive.Theyshowthataddingvisualfeaturesdecreasesrankingquality.Ourhypothesishereisthatunlikeisthecaseformanyotherimageretrievalproblems,ourimagescanbevisuallysim-ilarwhilebeingsemanticallyquitedifferent.Thatis,datavisu-alizationsof thesametype(e.g.,barcharts) lookroughlythesame,butalmostalwayscoversubstantiallydifferenttopics.

Focusingourattentiononourtextualfeatures,wefurtherinvestigateddifferentpermutationsofthesefeatures.Thecom-bination of content-WPtitle (semantic relatedness betweennewscontentandWikipediaarticle)andcontent-caption(se-manticrelatednessbetweenthenewscontentandimagecap-tion)worksthebest.Arankertrainedwiththesetwofeaturesachieveda0.82nDCG@5,whichissimilartotheperformanceoftherankertrainedonalltextualfeatures.Thetitle-captionfeature(SRbetweentitleandthecaption),however,appearstobelesseffective.Assuch,forreasonsofbothperformanceandparsimony,weusethemodeltrainedwithjustcontent-WPti-tleandcontent-caption inthe finalVizByWikisystem.Withthismodel,forinstance,thesystemwasabletogiveatopranktothevisualizationinFigure3forthearticleinFigure3(asin-dicatedbythescoreinFigure3).

6 DISCUSSION TheaboveevaluationsshowedthatVizByWikiisabletore-

trieveusefulvisualizationsforuptoapproximatelyhalfofpop-ularonlinenewsarticlesofdiversetypesandisabletorankthemwithreasonablequality.However,itisimportanttopointoutthatthesystemhasseveralnotablelimitations.

Firstand foremost,VizByWiki is limitedby thequalityofdatainWikimediaCommons.Despitebeingthelargestreposi-toryofitstype,theCommonssuffersfromthesamemetadatastandardizationissuesthatarecommontoallpeer-productionsystems[9].Fortunately,withtherecentlyannouncedmulti-year project aiming to standardize data inWikimedia Com-mons [40], chances are that metadata quality will improve,makingVizByWikimoreeffectiveinthefuture.

Second,VizByWiki sometimes recommends visualizationswitholderdata.Relatedly, italso sometimesfails toretrievedata visualizations for news that has very recently broken.Therearetwopotentialcauseshere.First,VizByWikiisusingastaticsnapshotoftheCommons’images,andthismayhavere-sultedinusservingolderversionsofvisualizationsthancur-rentlyexistintheCommons.Thisproblemcouldbeaddressedwithalarger-scaledeploymentusingrealtimeCommonsandWikipediadata.Second,itmaybethatthereisalagbetweenaneventoccurringandWikipediaeditorsupdatingtheirvisuali-zationstoincludethenewdata.Futureworkshouldexaminethelagtimeforvisualizationgeneration,ashasbeendonefortext(whichfoundlagtimeberelativelysmall[15]).

Third,weobservedthatVizByWikiworksbetteronsomearticles than on others. Future work should seek to answerquestions suchas:What characteristicsmakea newsarticlesuitablefordatavisualizations?Whatistheavailabilityofdif-ferenttypesanddifferenttopicsdatavisualizationsonWiki-mediaCommonsandontheWebmoregenerally?

Finally,crowdsourcedvisualizationsmightnotconformtospecific aesthetic requirements from publishers. However,thereisapromisingopportunitytochainVizByWikiwithotherpipelines that reverse-engineer data visualizations (e.g., [14,27,35])tosupportacompleteprocessoffindingdatavisuali-zation,extractingdataandredesigningthegraphics.

7 CONCLUSION Toaddressthechallengeofautomaticallygeneratinglarge

numbersofdatavisualizationsfornewsarticles,thispaperde-finedthenewsdatavisualizationretrievalproblem,whichinvolvesminingdatavisualizationsfromthewebtoenhancenewsarticles.Weshowedthatthisproblemwastractablebydesigninganend-to-endsystem,VizByWiki,whichminesapowerful-yet-untapped corpus of data visualizations: Wiki-media Commons. We evaluated VizByWiki using popularonlinenewsarticlesofdifferenttypes.Wefoundthatthesys-temcouldretrieveusefulvisualizationsformanypopulararti-clesandthatitcouldachievesatisfyingrankingquality.Tofa-cilitatefurtherprogressonthenewsdatavisualizationsystem,wearereleasingademoofthesystem,ourgroundtruthdata,andoursourcecode(seeURLonp.2).

ACKNOWLEDGEMENTS ThisworkwasfundedinpartbytheU.S.NationalScience

Foundation(IIS-1702440,IIS-1707319,CAREERIIS-1707296,andIIS-1421438).

Table 4. The performance of the supervised ranker using dif-ferent features and compared to the baseline method in [1].

Features nDCG@3 nDCG@5 nDCG@7

baseline [1] 0.69 (0.30) 0.74 (0.25) 0.78 (0.21)

all textual features 0.77 (0.32) 0.82 (0.23) 0.84 (0.19)

all textual features all visual features 0.69 (0.31) 0.75 (0.25) 0.78 (0.22)

content-caption title-caption 0.69 (0.29) 0.75 (0.24) 0.79 (0.21)

content-WPtitle title-caption 0.73 (0.26) 0.81 (0.19) 0.83 (0.17)

content-WPtitle content-caption 0.79 (0.26) 0.82 (0.17) 0.85 (0.19)

Note: The nDCG is the mean nDCG across all queries (i.e. news arti-cles). Standard deviations are in brackets.

REFERENCES [1] Agrawal, R. et al. 2011. Enriching textbooks with images. Proceedings

of the 20th ACM international conference on Information and knowledge management (2011), 1847–1856.

[2] Baack, S. 2011. A new style of news reporting: Wikileaks and data-driven journalism. Cyborg Subjects. (2011), 10.

[3] Chang, S. et al. 2015. Got Many Labels?: Deriving Topic Labels from Multiple Sources for Social Media Posts Using Crowdsourcing and En-semble Learning. Proceedings of the 24th International Conference on World Wide Web (New York, NY, USA, 2015), 397–406.

[4] Cheng, X. and Roth, D. 2013. Relational inference for wikification. Ur-bana. 51, 61801 (2013), 16–58.

[5] Delgado, D. et al. 2010. Assisted News Reading with Automated Illus-tration. Proceedings of the 18th ACM International Conference on Mul-timedia (New York, NY, USA, 2010), 1647–1650.

[6] Ensan, F. and Bagheri, E. 2017. Document Retrieval Model Through Semantic Linking. Proceedings of the Tenth ACM International Con-ference on Web Search and Data Mining (New York, NY, USA, 2017), 181–190.

[7] Gabrilovich, E. and Markovitch, S. 2007. Computing semantic related-ness using Wikipedia-based explicit semantic analysis. IJCAI (2007), 1606–1611.

[8] Gao, T. et al. 2014. NewsViews: an automated pipeline for creating cus-tom geovisualizations for news. (2014), 3005–3014.

[9] Hall, A. et al. 2017. Freedom Versus Standardization: Structured Data Generation in a Peer Production Community. Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (New York, NY, USA, 2017), 6352–6362.

[10] Howard, A.B. 2014. The Art and Science of Data-Driven Journalism. (2014).

[11] Hullman, J. et al. 2013. Contextifier: Automatic Generation of Anno-tated Stock Visualizations. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (New York, NY, USA, 2013), 2707–2716.

[12] Ipeirotis, P.G. et al. 2010. Quality Management on Amazon Mechanical Turk. Proceedings of the ACM SIGKDD Workshop on Human Compu-tation (New York, NY, USA, 2010), 64–67.

[13] Joachims, T. 2002. Optimizing Search Engines Using Clickthrough Data. Proceedings of the Eighth ACM SIGKDD International Confer-ence on Knowledge Discovery and Data Mining (New York, NY, USA, 2002), 133–142.

[14] Jung, D. et al. 2017. ChartSense: Interactive Data Extraction from Chart Images. Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (New York, NY, USA, 2017), 6706–6717.

[15] Keegan, B. et al. 2013. Hot Off the Wiki: Structures and Dynamics of Wikipedia’s Coverage of Breaking News Events. American Behavioral Scientist. 57, 5 (May 2013), 595–622. DOI:https://doi.org/10.1177/0002764212469367.

[16] Kittur, A. et al. 2008. Crowdsourcing User Studies with Mechanical Turk. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (New York, NY, USA, 2008), 453–456.

[17] Li, W. and Zhuge, H. 2014. Summarising news with texts and pictures. Semantics, Knowledge and Grids (SKG), 2014 10th International Con-ference on (2014), 100–107.

[18] Li, Z. et al. 2016. Multimedia News Summarization in Search. ACM Trans. Intell. Syst. Technol. 7, 3 (Feb. 2016), 33:1–33:20.

DOI:https://doi.org/10.1145/2822907. [19] Li, Z. et al. 2011. News contextualization with geographic and visual

information. Proceedings of the 19th ACM international conference on Multimedia (2011), 133–142.

[20] Li, Z. 2017. Understanding-Oriented Multimedia News Summariza-tion. Understanding-Oriented Multimedia Content Analysis. Springer, Singapore. 131–153.

[21] Lin, Y. et al. 2017. Problematizing and Addressing the Article-as-Con-cept Assumption in Wikipedia. Proceedings of the 2017 ACM Confer-ence on Computer Supported Cooperative Work and Social Computing (New York, NY, USA, 2017), 2052–2067.

[22] Mackinlay, J. 1986. Automating the Design of Graphical Presentations of Relational Information. ACM Trans. Graph. 5, 2 (Apr. 1986), 110–141.

DOI:https://doi.org/10.1145/22949.22950. [23] Marcus, A. et al. 2013. Data In Context: Aiding News Consumers while

Taming Dataspaces. DBCrowd 2013. 47, (2013). [24] Mihalcea, R. and Csomai, A. 2007. Wikify!: Linking Documents to En-

cyclopedic Knowledge. Proceedings of the Sixteenth ACM Conference on Conference on Information and Knowledge Management (New York, NY, USA, 2007), 233–242.

[25] Noraset, T. et al. 2014. WebSAIL Wikifier at ERD 2014. Proceedings of the First International Workshop on Entity Recognition & Disam-biguation (New York, NY, USA, 2014), 119–124.

[26] Parasie, S. and Dagiral, E. 2013. Data-driven journalism and the public good:“Computer-assisted-reporters” and “programmer-journalists” in Chicago. New media & society. 15, 6 (2013), 853–871.

[27] Poco, J. and Heer, J. 2017. Reverse-Engineering Visualizations: Recov-ering Visual Encodings from Chart Images. Computer Graphics Fo-rum. 36, 3 (Jun. 2017), 353–363. DOI:https://doi.org/10.1111/cgf.13193.

[28] Ramisa, A. et al. 2016. Breakingnews: Article annotation by image and text processing. arXiv preprint arXiv:1603.07141. (2016).

[29] Razavian, A.S. et al. 2014. CNN Features off-the-shelf: an Astounding Baseline for Recognition. arXiv:1403.6382 [cs]. (Mar. 2014).

[30] Ren, D. et al. 2014. iVisDesigner: Expressive Interactive Design of In-formation Visualizations. IEEE Transactions on Visualization and Computer Graphics. 20, 12 (Dec. 2014), 2092–2101. DOI:https://doi.org/10.1109/TVCG.2014.2346291.

[31] Satyanarayan, A. and Heer, J. 2014. Lyra: An Interactive Visualization Design Environment. Computer Graphics Forum. 33, 3 (Jun. 2014), 351–360. DOI:https://doi.org/10.1111/cgf.12391.

[32] Savva, M. et al. 2011. Revision: Automated classification, analysis and redesign of chart images. Proceedings of the 24th annual ACM sympo-sium on User interface software and technology (2011), 393–402.

[33] Segel, E. and Heer, J. 2010. Narrative visualization: Telling stories with data. IEEE transactions on visualization and computer graphics. 16, 6 (2010), 1139–1148.

[34] Sen, S. et al. 2014. WikiBrain: Democratizing Computation on Wikipe-dia. Proceedings of The International Symposium on Open Collabora-tion (New York, NY, USA, 2014), 27:1–27:10.

[35] Siegel, N. et al. 2016. FigureSeer: Parsing Result-Figures in Research Papers. Computer Vision – ECCV 2016 (Oct. 2016), 664–680.

[36] Szegedy, C. et al. 2016. Rethinking the Inception Architecture for Com-puter Vision. (2016), 2818–2826.

[37] Tsikrika, T. et al. 2011. Overview of the Wikipedia Image Retrieval Task at ImageCLEF 2011. CLEF (Notebook Papers/Labs/Workshop) (2011).

[38] Wang, P. et al. 2017. Concept Embedded Convolutional Semantic Model for Question Retrieval. Proceedings of the Tenth ACM Interna-tional Conference on Web Search and Data Mining (New York, NY, USA, 2017), 395–403.

[39] WikiBrain: Advanced SR usage.: https://shilad.github.io/wikibrain/tu-torial/advancedsr.html. Accessed: 2017-10-31.

[40] Wikimedia Foundation 2017. Wikimedia Foundation receives $3 mil-lion grant from Alfred P. Sloan Foundation to make freely licensed im-ages accessible and reusable across the web. Retrieved from https://blog.wikimedia.org/2017/01/09/sloan-foundation-structured-data/

[41] Wongsuphasawat, K. et al. 2016. Voyager: Exploratory Analysis via Faceted Browsing of Visualization Recommendations. IEEE Transac-tions on Visualization and Computer Graphics. 22, 1 (Jan. 2016), 649–658. DOI:https://doi.org/10.1109/TVCG.2015.2467191.

[42] 2017. Help:Adding image. Wikipedia. Retrieved from https://en.wik-ipedia.org/w/index.php?title=Help:Adding_image&oldid=764170156