Upload
george-awad
View
79
Download
1
Embed Size (px)
Citation preview
TRECVID2016WorkshopNa6onalIns6tuteofStandardsandTechnology
Mul6mediaEventDetec6onTaskNov.15,2016
DavidJoy,JonathanFiscus,AndrewDelgado
[email protected]/comments
MEDSessionSchedule9:00–11:20 Tuesday,Nov.15
9:00–9:20 MEDTaskOverview
9:20–9:40 VIREO(CityUniversityofHongKong)
9:40–10:00 INF(CarnegieMellonU.;BeijingU.ofPostsandTelecommunica6on;U.AutonomadeMadrid;ShandongU.;XianJiatongU.;SingaporeManagementU.)
10:00–10:20 MediaMill(UniversityofAmsterdam)
10:20–10:40 Break
10:40–11:00 BUPT-MCPRL(BeijingUniversityofPostsandTelecommunica6ons)
11:00–11:20 MEDDiscussion
Quicklyfindinstancesofeventsinalargecollec6onofsearchvideos
Multimedia Event Detection (MED)
Execu6onHardwareRepor6ng• 3ClassesofCompu6ngHardware• Small: 100CPUcores,1,000GPUcores• Medium: 1,000CPUcores,10,000GPUcores• Large: 3,000CPUcores,30,000GPUcores
Evaluation Conditions
EventModel
No6onalSystemDiagram
Evalua6onData
MetadataMED
ModelGenera6on
QueryExecu6on
RankedVideosid036id839id983id312id033id239id783id912…
EventKit
QueryTrainingCondi6ons
NumberofExemplars
Pre-SpecifiedEvents 0 10 100
Ad-HocEvents 10
Interac6veAd-HocEvents
10
SearchCollec6on• MED16Eval-Full->198Kvideos,4,738hours• MED16Eval-Sub->32Kvideosubset,783hours
Mul6mediaEventDetec6onTask
AMEDeventisacomplexac6vityoccurringataspecificplaceand6meinvolvingpeopleinterac6ngwithotherpeopleand/orobjects
MED‘16Overview
• MEDevalua6onsfrom2010-2016– SupportedbytheIARPAAladdinProgramandLDCcollecteddata
– Severalsimplifica6onsin2015,whichwerecon6nuedin2016
• What’snewinMED2016– Introduc6onofnewtestdataset,asubsetofthe*Yahoo!FlickrCrea6veCommons100Million(YFCC100M)videos
– 10newAd-Hocevents*-Disclaimer:Certaincommercialequipment,instruments,ormaterialsareiden6fiedinthispaperinordertospecifytheexperimentalprocedureadequately.Suchiden6fica6onisnotintendedtoimplyrecommenda6onorendorsementbytheNa6onalIns6tuteofStandardsandTechnology,norisitintendedtoimplythatthematerialsorequipmentiden6fiedarenecessarilythebestavailableforthepurpose.
TheTRECVIDMED2016EventsPre-SpecifiedEvents Ad-HocEvents
MED‘14PSEvents MED‘14AHEvents NewEvents
Aqemp6ngabiketrick Beekeeping Camping
Cleaninganappliance Weddingshower CrossingaBarrier
Dogshow Non-motorizedveh.repair OpeningaPackage
Givingdirec6onstoaloca6on Fixingmusicalinstrument MakingaSandSculpture
Marriageproposal Horseridingcompe66on MissingaShotonaNet
Renova6ngahome Fellingatree Opera6ngaRemoteControlledVehicle
Rockclimbing Parkingavehicle PlayingaBoardGame
Townhallmee6ng Playingfetch MakingaSnowSculpture
WinningaracewithoutavehicleTailga6ng MakingaBeverage
Workingonametalcrassproject
Tuningmusicalinstrument Cheerleading
OperaTngaRemoteControlledVehicle
Illustrative Examples • Positive instances of the event • Non-Positive “miss” clips that do not contain the event
EvidenTalDescripTon:• scene:indoorsoroutdoors• objects/people:remotecontrolvehicles(cars,trucks,planes,
helicopters,trains,etc.),remotes,antennas,racetrack,plas6ctakeofframp
• ac6vi6es:direc6ngremotecontrolvehicles,turningonvehicles,crashingvehicles
• audio:enginesrevving,motorwhirring,explana6onoftypeofvehicle,discussionofwherethevehicleisgoingorcouldgo,discussionofwhatisseenonavideofeedfromthevehicle
Definition: An individual operates a vehicle remotely with a controller Explication: Remote controlled vehicles are self-propelled machines that are powered by a motor or engine of some kind and whose movement is controlled from a distance by human inputs to a remote control device …
ExampleEventKit
Miss!
TheTestDataDatacollecTon
Testset #ofvideos
DuraTon(Hrs)
Avg.duraTon(Secs)
HAVICProgress
MED16EvalFull 98,003 3,713 136
MED16EvalSub 16,000 620 139
YFCC100MSubset
MED16EvalFull 100,000 1,025 37
MED16EvalSub 16,000 163 37
Total MED16EvalFull 198,003 4,738 86
MED16EvalSub 32,000 783 88
• HAVICProgress– Engineeredtarget
richness– Controlledsampling
ofInternetvideodomain
• YFCC100MSubset– Randomselec6on*– Shorterdura6on
videos
*-ExcludingYLI-MEDcorpusvideos(~50kvideos);Excludingvideosnotavailablebymmcommons.org’sAWSS3datastore(~5kvideos)
12MED2016FinishersByCondi6onYears
Team
AH PS
Organiza6on10Ex 0Ex 10Ex 100Ex
SML MED SML MED SML MED SML MED
6
INF Sub Sub Sub CarnegieMellonUniversityetal.MediaMill Full Full MediaMill-UniversityofAmsterdamNIIHitachiUIT Full Sub Na6onalIns6tuteofInforma6csTokyoTech Full Full Full TokyoIns6tuteofTechnology
5 VIREO Full Full Full Full CityUniversityofHongKong&TNO
3
ITICERTH Sub Sub Sub Informa6csandTelema6csInst.KU-ISPL Sub Sub KoreaUniversity
nwudan Full Full FullNTTMediaIntelligenceLaboratoriesandFudanUniversity
MCIS Sub Sub BeijingIns6tuteofTechnologyMcislab
2 BUPTMCPRL Sub SubMul6mediaCommunica6onandPaqernRecogni6onLabsBUPT
Eqer Sub Sub EqerSolu6ons1 PKUMI Full PekingUniversity
3 1 4 1 10 1 8 1
0
10
20
2010(Pilot)2011 2012 2013 2014 2015 2016
NumberofMEDFinishers
AH–Ad-Hoceventcondi6onPS–Pre-Specifedeventcondi6on0Ex–0exemplarcondi6on10Ex–10exemplarcondi6on100Ex–100exemplarcondi6onSML–Small-sizedhardwareMED–Medium-sizedhardwareFull–processedMED16EvalFulltestsetSub–processedMED16EvalSubtestset-Redoutlineindicatesarequiredcondi6on
Pre-SpecifiedEventMAPPrimarySystems–HAVICProgressSubset
MAP(EvalSub-ProgressSubset)=1.02*MAP(EvalFull-ProgressSubset)+5.96R^2=0.996
EvalFull EvalSub
Pre-SpecifiedAPbySystemandEventPrimarySystems–10Ex–MED16EvalSub–MixedSystemSize
ProgressSubset
MAPàMeanInferredAveragePrecision(MInfAP)
• FollowsAslametal.procedure,Sta6s6calMethodforSystemEvalua6onUsingIncompleteJudgmentsProceedingsofthe29thACMSIGIRConference,Seaqle,2006.
– Stra6fied,variabledensity,pooledassessmentproceduretoapproximateMAP
• MInfAPinthe2016evalua6on– Progress–MAPandMInfAP200
(simulated)onPSandAH– Progress+YFCC100M–MInfAP200onPS
andAH• ForMED‘15,NISTranexperimentswith
2014datatoop6mizethestratasizesandsamplingrate.ThissamesamplingratewasusedforMED‘16
– Define2strata• 1-60->100%• 61-200->20%
PS-EvalSub-ProgressSubset--MInfAP200(Simulated)=1.14*MAP+0.421R^2=0.99
Pre-SpecifiedEvalSubSimulatedMInfAP200ProgressSubset
Pre-SpecifiedEventMInfAP200Progress+YFCC100M
EvalFull EvalSub
PerformanceonHAVICvs.Yahoo!OurAqempttoScorePrecision@10
• UnabletoscorePrecision@10forbothHAVICandYahoo!– Stra6fiedsamplingdidnotyieldsufficientjudgements
• Forexample:– E022–Cleaninganappliance
– Propor6onofsubsetoftop200clips,binnedbyrank
– Teamsshownarerepresenta6vebasedonMInfAP200scores
PerformanceonHAVICvs.Yahoo!Good,Bad,UglyEvents
Event AP(top5) Descrip6onE022 8.42 CleaninganapplianceE028 53.78 Townhallmee6ngE039 61.4 Tailga6ng
• Stra6fiedrandomsamplenotsufficientforheterogeneousdata
• Wewillcon6nuetoworkonscoringYahoo!separately
Ad-HocEventResults
• 10newevents– 10Exemplartrainingonly
• MED16EvalFulltherequiredcondi6on
• ReferenceGenera6on– Pooledassessmentwithusingall
submissions– Stratadefini6on
• 1:60:100%• 61:200:20%
Ad-HocMInfAP200onEvalFull
Ad-HocInfAPbySystemandEventPrimarySystems–10Ex–MED16EvalFull–MixedSystemSize
Ad-HocPooledAssessmentEventRichnessvs.InfAP
E051 CampingE052 CrossingaBarrierE053 OpeningaPackageE054 MakingaSandSculptureE055 MissingaShotonaNetE056 Opera6ngaRemoteControlledVehicleE057 PlayingaBoardGameE058 MakingaSnowSculptureE059 MakingaBeverageE060 Cheerleading
Ad-Hoc10Ex–MInfAP200BoxPlotsEventRichnessinAnnota6onPools
MED‘16Summary
• Pre-SpecifiedResults– Onlyoneteambuilta“Medium”hardwaresystem– Mostteamsprocessedthesubset(783hr.)testset(MED16EvalSub)
– No6ceableimprovementoverlastyear’sPre-SpecifiedresultsonProgress
– Stra6fiedrandomsamplingonheterogeneousdatanotpowerfulenoughtodeterminedifferencesinperformance
• Ad-HocResults– Only4of12teamspar6cipated– Noteamspar6cipatedinInterac6veEventQuerytest
MED’17Plans
• NISTintendstocon6nueMEDinastreamlinedfashion
• NISTtoreleaseProgressannota6ons• MakeupofdatasetsTBD,HAVIC+YFCC100M• Discon6nuingsupportforInterac6veAd-Hoccondi6on
• Counter-proposals?
Thankyou!
Ques6ons?