Upload
hirokatsu-kataoka
View
359
Download
0
Embed Size (px)
Citation preview
DominantCodewordsSelec2onwithTopicModelforAc2onRecogni2on
HirokatsuKATAOKA,MasakiHayashi†,YoshimitsuAOKI†,KenjiIWATA,YutakaSATOH,SlobodanIlic‡
Na2onalIns2tuteofAdvancedIndustrialScienceandTechnology(AIST)†KeioUniversity
‡TechnischeUniversitätMünchen
hPp://www.hirokatsukataoka.net/
HumanSensing• Severaltasksareincludedinvision-basedhumansensing– Detec2on,tracking,facerecogni2on– Posturees2ma2on,ac2onanalysis(eventrecogni2on)– Ac2onrecogni2onisabletoextendhumansensingapplica2ons
Mentalstate
BodySitua2on
APen2on
Ac2onAnalysis
shakinghands
Lookatpeople
Detec2on GazeEs2ma2on
Ac2onRecogni2on
PostureEs2ma2on
FaceRecogni2on
Trajectoryextrac2on
Tracking
UnderstandingHumanAc2ons• Ac2onisdefinedassomethingthatpeopledoorcausetohappen– e.g.walking,running, si^ng
Thisimagecontainsamanwalking• Ac2onrecogni2on
Classifica2on• Ac2ondetec2on
Classifica2on&localiza2on
Walking
Walking
Ac2onRecogni2on
Ac2onDetec2on
CurrentAc2onRecogni2on• Trajectory-basedrepresenta2on
ImprovedDenseTrajectories(IDT)[Wang+,ICCV13] Trajectory-pooledDeep-convolu2onalDescriptors(TDD)[Wang+,CVPR15]
• “Dense”representa2onisimportant• Trajectory-basedhand-craied
descrip2on(HOG/HOF/MBH/Traj.)
• Intersec2onofhand-craiedanddeep-learnedfeatures
• Intui2vely,IDTfeatureisreplacedbyConvMaps
• CNNisbasedonTwo-streamConvNet[Simonyan+,NIPS14]
TypicalAc2onRecogni2onPipeline
Trajectories/keypointsextrac2on Featuredescrip2on
Codewordpooling Classifica2on
e.g.BoF,VLAD,FisherVectors hPp://www.analy2calway.com/images/SMV/svmFeatureSpace2.gif
e.g.SVM
TypicalAc2onRecogni2onPipeline• Twostrategiesformoresophis2catedfeaturevector
Trajectoryextrac2on Trajectorydescrip2on
1.Trajectoryrefinement!(maincontribu2on)
2.BePerfeature!
Ourproposal• 1.Dominantcodewordsselec2on(DCS)– Significantac2onrepresenta2onbyusingtopicmodel(typicalLDA)– Topic-basedgroupingfromdensetrajectories– Noiseelimina2onateachtopic
• 2.Co-occurrencefeaturerepresenta2onforbePerfeat.– Asanaddi2onalfeatureintotypicalHOG/HOF/MBH/Traj.– Thefeatureisbasedon[Kataoka+,ACCV14]whichisimprovedco-occurrence
featurefocusingonextrac2ngsubtlemo2on
××××
×××
××
×
×
×
×× ×
Therearesomenoiseeveninthesta2cscene!
Flowchartofdominantcodewordsselec2on(DCS)• 1.Featureextrac2on– DenseTrajectories
• 2.Topicmodelingtodivideprimi2vemo2on– Eachtopicisapproxima2ngeachprimi2vemo2on
• 3.Noisecancelingateachtopic• 4.Dominantdensetrajectories(DDT)
Featureextrac2on• Improveddensetrajectories[Wang+,ICCV13]– HOG/HOF/MBHx,/MBHy– CoHOG,ExtendedCoHOG[Kataoka+,ACCV14]– Bag-of-features(BoF)toinputoftopicmodel
ExtendedCoHOG
TopicModel• LatentDirichletalloca2on(LDA)[Blei+,JMLR03]– Weappliedsimplifiedmodel[Griffiths+,04]– TypicalLDA– Parameters• Probabilitydistribu2onoftopic:Θ• Topic:T• Hyperparameter:α,β• DT-BoFfromvideo:v
TopicModelforDCS• Noiserejec2onateachtopic– Eachtopicindicateseach“primi2vemo2on”– In-topicadap2vethresholding
×××××××
××
×
×
“cutoffends” Topic1 Topic2 Topic3 Topic4
MPIIcooking
Eachtopicindicateseach“primi2vemo2on”
× ××× ××× ×
×××××××
××
×
×
“cutoffends”
××× × ×××
××
×
××× Unsophis2catedrejec2onisnotdesirable
× Eliminatedvector
Dominantdensetrajectoies(DDT)• Allrefinedtopicsareintegrated– DDT
– DDTisamixtureofdominantcodewordsusingANDopera2on e.g.• T1={vw1,vw3,vw5}=>T1’={vw1,vw5}• T2={vw1,vw2,vw7}=>T2’={vw1,vw2,vw7}• T3={vw3,vw6,vw7}=>T3’={vw7}
• DDT={vw1,vw2,vw5,vw7}
• DDTissophis2catedrepresenta2onwithdominantcodewords
Experiments• Fourdatasetsforac2onrecogni2on
【INRIA surgery】 View: 4 Activity: 4
view1 view2
view3 view4
【IXMAS】 View: 5 Activity: 11
【MPII cooking activities】 View: 1 Activity: 65
view1 view2
view3 view4
view5
【NTSEL traffic】 View: 1 Activity: 4
Parameters• DT,classifica2on&LDAarebasedonthepreviousworks– # ofcodeword: 4,000– # oftrajectorypooling:15frame– TheparametersarefollowingoriginalDT– αandβofLDAaresetat1.0and0.01– 1,000Gibbssampleritera2ons– Thetopicmodelingparamsarebasedon[GriffithsandSteyvers,04]
• Dominantcodewordsselec2on(DCS)– Topic-basedcodewordsaresetas1%ofthefrequency– #oftopic:#ofviewx#ofac2on
ComparisonofDT• DominantDTvsDT– ByusingHOG,HOF,MBHx,MBHyandcombinedfeatures
– Dominantcodewordsselec2oniseffec2veontheDT
Co-occurrencefeatureinDDT• Onfine-graineddataset– MPIIcooking[Rohrbach+,CVPR12]– Extendedco-occurrencefeatureisimprovedbyDCS– DDTisimprovedwithextendedco-occurrencefeature
– [Ni+,ECCV14]and[Ni+,CVPR15]arestate-of-the-artwithmid-levelfeatures
TopicVisualiza2on
××
×
×
×
×
×
××× ××××× ×
××××××
×××
×
××× ×××
×× ×
×
× ×
×××××
××
×
××××××
×××
×××× ×××××
×××
×××
×
×
××
×××
××
× ×
×
×
××××
×× ××
××××××
×
×
×
××
×
×××
×××××××
××××
Topic1(View2)
“sit”View1
“sit”View2
Topic1(View1)
Topic2(View2)
Topic2(View1)
Topic3(View2)
Topic3(View1)
Topic4(View2)
Topic4(View1)
“getup”View2 Topic1(View2) Topic2(View2) Topic3(View2) Topic4(View2)
“crossing” Topic1 Topic2 Topic3 Topic4
“cutoffends” Topic1 Topic2 Topic3 Topic4
INRIAsurgery
INRIAsurgery
IXMAS
NTSELtraffic
MPIIcooking
Conclusion• Dominantcodewordsselec2on(DCS)forac2onrecogni2on– Weappliedtopicmodelfornoisecancelingintrajectory-basedrepresenta2on– Extrafeature(co-occurrencefeature)
• Futureworks– Exploringparametersandmoreeffec2vevisualiza2on(on-going…)– Seman2cflowwithtopicmodels