Embed Size (px)
Citation preview
Active Learning for Online Recognition of Human Activities fromStreaming Videos.
Rocco De [email protected]
Ilaria [email protected]
Fabio [email protected]
Barbara [email protected]
Nicolo [email protected]
April 12, 2016
AbstractRecognising human activities from streaming videosposes unique challenges to learning algorithms: predic-tive models need to be scalable, incrementally trainable,and must remain bounded in size even when the datastream is arbitrarily long. Furthermore, as parameter tun-ing is problematic in a streaming setting, suitable ap-proaches should be parameterless, and make no assump-tions on what class labels may occur in the stream. Wepresent here an approach to the recognition of human ac-tions from streaming data which meets all these require-ments by: (1) incrementally learning a model which adap-tively covers the feature space with simple local clas-sifiers; (2) employing an active learning strategy to re-duce annotation requests; (3) achieving promising accu-racy within a fixed model size. Extensive experiments onstandard benchmarks show that our approach is compet-itive with state-of-the-art non-incremental methods, andoutperforms the existing active incremental baselines.
1 IntroductionThe pervasive presence of cameras in our everyday liveshas created a strong demand for automatic methods ableto analyse real time video streams. This is especially chal-lenging for videos depicting human activities, as it is thecase in TV footages, surveillance cameras, human com-puter and human robot interactions, and many other ap-plications. The mainstream approaches to action and ac-
tivity recognition are typically based on an offline trainingphase (for a review of previous work in activity recog-nition we refer the reader to Sec. 2). Such a settingleads to several critical issues when dealing with stream-ing videos:
How to continuously learn about activities from the in-coming data? The dynamic nature of the streaming videosetting implies that at each time instant new data is madeavailable to the system, which needs to continuously learnfrom it. This implies refining its model of known humanactivities and adding new previously unseen activities onthe fly.
How to minimise the required annotation effort? Strictlyrelated to the ability to learn on the fly is the issue of howmany video fragments should be annotated. While fornewly observed activities one might assume that all videofragments should be manually annotated, when analysingfootage of known actions only a fraction of the video frag-ments will likely bring new information, worthy of theannotation effort. In this context, the system should au-tomatically select which fragments are the most informa-tive, and asks for the help of a human annotator in thosecases.
How to optimise the algorithm’s heuristics dynamically?System’s components, such as the features chosen and thealgorithm parameters, have a crucial impact on the finalperformance of any framework. When learning contin-uously from incoming data, it is difficult to chose thesetwo components optimally, or even just properly. This
1
arX
iv:1
604.
0285
5v1
[st
at.M
L]
11
Apr
201
6
is because, by the very nature of dynamic data, it is notpossible to anticipate what new instances of the knownclasses the system will face, or what new activities it willbe asked to learn.
The contribution of this paper is an algorithm for hu-man activity recognition from streaming videos which,to the best of our knowledge, is the first to address allthe challenges listed above in a principled manner. Ourstarting point is a very recently proposed local algorithmfor classification of data streams [5] that is incrementallytrainable and nonparametric1, while providing theoreti-cal guarantees on performance. We leverage on this re-sult, and extend it to the active learning setting. Thisleads to a framework that matches all the requirementslisted above: (1) it incrementally learns from the incom-ing data stream, with respect to both known classes andnew classes, while being computationally efficient; (2) itsactive learning component evaluates the informative con-tent of the incoming data with respect to the level of confi-dence of the system and allows to decide when the cost ofmanual annotation is worthwhile; (3) lastly, the nonpara-metric nature of the approach, combined with its intrinsiclocality, allows for fully data-driven learning.
Fig. 1 illustrates the workflow of our approach: (i)each video is associated with an arbitrary number of fea-ture vectors in a feature space; (ii) the incoming trainingvectors are used to sequentially cover the feature spacewith balls; (iii) each ball is associated with an estimateof the conditional class probabilities obtained by collect-ing statistics around its centre, these estimates are used tomake predictions on new unlabeled samples; (iv) the ra-dius of each ball is adjusted accordingly how well eachball predict the new samples around its centre; (v) ballcentres are incrementally adjusted to fit the actual datadistribution; (vi) the set of balls is organized in a treestructure [17], so that predictions can be computed in timelogarithmic in the number of balls. We call our algorithmFast active Incremental Visual covERing (FIVER). Exten-sive experiments on several publicly available databasesshow that our approach outperforms all existing algo-rithms for activity recognition from streaming data. Fur-thermore, we show that by combining FIVER with the ro-
1In nonparametric models the model structure is not specified a pri-ori, but determined from data. The implication is not that such modelscompletely lack parameters, but that the number and value of the param-eters are flexible and not fixed in advance.
bust temporal segmentation algorithm presented in [7] weobtain a system able to deal in a straightforward mannerwith a realistic continuous active recognition scenario.
2 Related WorkWithin the vast literature related to action recognition(see [24] and references therein), research focusing on thestreaming setting has gained momentum only recently [8].Important features required in this context are: (1) Incre-mental Updates: as a large amount of data is generallypresented sequentially in a stream, it is desirable for algo-rithms to update the model adaptively, rather than re-trainfrom scratch. (2) Incremental Learning of New Activi-ties: algorithms should be able to accomodate on the flynew upcoming classes. (3) Bounded Size Model: as thestream could be very large, models should keep a boundedmemory footprint, allowing for real-time prediction whileavoiding storage issues. This implies the ability to discarduseless or old information, which is critical to the track-ing of “drifting concepts” (i.e., when the optimal decisionsurface changes over time and the model needs to be rear-ranged accordingly [28]). (4) Data-Driven: as parametertuning is problematic in streaming settings, systems withfew or no parameters are preferable. (5) Nonparametric:as the true structure of the data is only progressively re-vealed as more examples from the stream are observed,nonparametric algorithms [10], which are not committedto any specific family of decision surfaces, are preferable.(6) Active Learning: in a streaming setting, the systemneeds to learn from each incoming video stream. How-ever, training labels can only be provided by human an-notators, who should only be invoked when the systemhas low confidence in its own prediction of the currentlabel. (7) Bounded Request rate: as querying human an-notators is expensive, any practical active learning systemfor streaming settings should impose a bound on the queryrate.
Table 1 lists the previous efforts in human activityrecognition involving incremental and/or active learningcomponents which, due to their features, are the closestalternatives to our approach.
A feature tree-based incremental recognition approachwas proposed in [25], where the tree is free to growwithout bounds as more examples are fed to the learner.As this requires to store all the presented instances, the
2
Figure 1: Our approach vs Bag-of-Words. In the classical BoW pipeline [27] (top) vocabulary size, clusteringmethod, and classifier parameters are the result of tuning by cross-validation. In contrast, our approach (see Alg. 1) iscompletely data-driven (nonparametric and parameterless). Our local balls cover the feature space as BoW codewordsdo, but their number, size, and location dynamically depend on the distribution of feature vectors and associated labels.Rather than having a global classifier act on the entire feature space, a cover of local classifiers associated with updatedlocal class statistics is incrementally built. An active learning component, based on a suitable confidence measure (seeAlg. 3), makes it possible to perform active temporal segmentation, leading to continuous activity recognition (Fig. 3).
Incr. Upd. New Activ. Bound. Size Data Driv. Nonpar. Act. Learn. Bound. Rate
De Rosa et al. [4]√ √ ⊗ ≈
(one param.)√ ⊗ ⊗
Hasan et al. [11]≈
(minibatch)⊗ √ ⊗ ⊗ √ ⊗
Hasan et al. [12]≈
(minibatch)⊗ ⊗ ⊗ ⊗ √ ⊗
Hasan et al. [13]≈
(minibatch)⊗ ⊗ ⊗ ⊗ √ ⊗
FIVER√ √ √ √ √ √ √
Table 1: Identified open challenges in previous work on human action recognition for streaming data. To the best ofour knowledge, FIVER is the only algorithm equipped with all these features, crucial for dealing with streaming data.LEGEND:
√: exhibits the feature; ≈: partially exhibits the feature;
⊗: does not possess the feature.
method is infeasible for continuous recognition fromstreaming videos where the number of activities can getvery large over time. A human tracking-based incre-mental activity learning framework was brought forwardin [22]; nevertheless, it requires annotation on the location
of the human body in the initial frame, heavily restrictingits applicability. For these reasons [25] and [22] are notlisted in Table 1.
Our work shares similarities with the incremental al-gorithm proposed in [4], on which to some extent we
3
build. Both methods adopt a nonparametric, incrementalball covering of the feature space strategy. FIVER, how-ever, brings to the table crucial new features that makes ituniquely suitable for dealing with streaming data. First,it does not rely on any input parameters, which are in-convenient to tune in streaming settings. Second, it limitsthe model size, thus allowing the tracking of drifting con-cepts: when the number of allocated balls exceeds a givenbudget, FIVER discards them with a probability propor-tional to their error rate. Third, it dynamically adjusts theball centres, thus yielding very compact models while im-proving performance. The resulting covering resemblesa visual dictionary, learned incrementally and directly us-able for predictions (see Fig. 1), where the balls play therole of visual code words. Finally, the active learningmodule defines the interaction between the learning sys-tem and the labeler agent, limiting the number of annota-tions requested.
The use of incremental active learning for activityrecognition tasks has been recently investigated in [11,12]. The approach in [12] uses an ensemble of lin-ear SVM classifiers incrementally created in a sequenceof mini-batch learning phases. A confidence measureover the SVM outputs is defined, where each individualclassifier output is weighted by the training error. Twouser-defined thresholds control the query rate of labeledvideos. Non-confident instances, which are close to aclass boundary, are forwarded to the annotator, while theothers are discarded. Notably, the set of ensemble clas-sifiers can become very large, as an arbitrary number ofSVMs can be added in each batch phase. Furthermore,the method requires model initialisation and has severalparameters to be tuned on a validation set; this require-ment makes the approach unsuitable for a truly streamingcontext. The method in [11] initially learns features in anunsupervised manner using a deep neural network. Then,a multinomial logistic regression classifier is learned in-crementally. The posterior class probability output is used(as in [12]) to select what videos need supervised informa-tion. This method too presents several parameters, can-not deal with new classes, and requires initialisation. Thesame authors have recently presented in [13] an extensionof the methods [12, 11] that attempts to mine informa-tion from the scene’s context. However, the core learningsystem suffers of the same drawbacks discussed above.Moreover, the active modules used in [12, 11, 13] cannot
explicitly limit the query rate, a crucial feature for realworld applications where the cost of human annotationhas to be limited.
3 Fast Active Incremental VisualCovering
This section describes the FIVER algorithm, which is theheart of our system for activity recognition from stream-ing videos. Sec. 3.1 describes our incremental visual cov-ering approach based on [4], Sec. 3.2 shows how to keepthe memory footprint bounded via a technique introducedin [5], but until now never applied to activity recognition.In Sec. 3.3 we introduce a mechanism that performs activelearning on the stream. The FIVER algorithm is finallysummarized in Sec. 3.4.
3.1 (Passive) Incremental Learning
We assume the learner is trained on a stream of(pre-segmented)2, labeled videos (V 1, y1), (V 2, y2), . . . .Each video V i is associated with a set of Ti local descrip-tors {x(i)
t }Tit=1, where each descriptor x(i)
t ∈ Rd belongsto a d-dimensional feature space. Each video label yi de-notes an activity from a set Y = {1, . . . , C} of possibleclasses, which may change over time. The classifier istrained incrementally, via small adjustments to the currentmodel every time a new labeled video is presented to thelearner. In what follows, we drop the superscript i and re-index the local features, thus assuming that the learner isfed a sequence (x1, y1), (x2, y2), . . . of labeled local fea-ture examples, where (xt, yt) =
(x(i)t , yi
)∈ Rd × Y for
some t and 1 ≤ t < Ti. The feature space is adaptivelycovered by a set S of balls depending on the complexityof the classification problem. Unlike [4], where the ballswere always centered on input samples, here we adapt theAUTO-ADJ version of ABACOC described in [5]. Thisuses a K-means-like update step in which the centre ofeach ball is shifted towards the average of the trainingsamples that were correctly predicted by the local clas-sifier; in practise, the balls track the feature clusters, or“visual codewords”. Following [5] we initialize the ra-
2Although videos are assumed pre-segmented here, it is possible tolocalise and segment beforehand activity instances, see for example [11,12].
4
dius of any new ball to the distance from the closest ball.The resulting setting is parameterless3.
Incremental updates. The sequence of observed trainingexamples (xt, yt) is used to build a set S of balls thatcover the region of the feature space they span. For eachball, an empirical distribution of classes is kept. For eachball centre cs ∈ S , we keep updated the number ns(y) ofdata points xt of each class y ∈ Y that at time t belongto the ball. These counts are used to compute the classprobability estimates (activity scores) for each ball centrecs ∈ S as follows:
ps(y) =ns(y)
nsy = 1, . . . , C (1)
where ns = ns(1) + · · ·+ ns(C).More specifically, the training algorithm operates as
follows —see Alg. 1. Initially, the set of balls S is empty.For each training example xt, we efficiently4 compute itsnearest ball centre cπ(t) ∈ S . If xt does not belong toits closest ball, i.e., the distance ρ(cπ(t),xt) between xtand cπ(t) is greater than the ball’s radius επ(t), a new ballwith centre xt and initial radius Rt equal to ρ(cπ(t),xt)is created and added to S. The label yt is used to ini-tialise the empirical class distribution for the new ballvia (1). If xt does belong to the nearest ball, its labelyt is used to update the error count mπ(t) for that ball.The local classifier centred at cπ(t) makes a mistake on(xt, yt) if and only if yt 6= argmaxy∈Y pπ(t)(y). When-ever this happens, the radius is set to its initial valueRπ(t)scaled by a polynomial function in the current error count:επ(t) = Rπ(t)m
−1/4π(t) .5
If the prediction is correct, the ball centre is set to theaverage of the correctly classified instances within theball, allowing it to move towards the majority class cen-troid. Finally, the class probability estimates pπ(t)(y) forthe local classifier centred in cπ(t) are updated via (1).Notably, a-priori knowledge of the full set of classes Y isnot needed, as our incremental learning approach can addnew labels to Y as soon as they first appear in the stream.
3Unlike [4], we set the feature space intrinsic dimension parameter dto 2, as we empirically found that it does not significantly affect perfor-mance.
4For example, [17] embeds S in a tree where nearest neighbourqueries and updates can be performed in time O(ln |S|) —see also [16].
5This comes from the decay function defined in [4] with d = 2.
Algorithm 1 ABACOC ([5], adapted to feature sets)Input: feature space metric ρ
1: Ball centres S = ∅ and set of labels Y = ∅ initialised
2: for i = 1, 2, . . . do3: Get labeled video (V i, yi)4: if yi /∈ Y then5: Set Y = Y ∪ {yi} // add class on the fly6: end if7: for t = 1, . . . , Ti do8: if S ≡ ∅ then9: if t = 1 then
10: save x1;11: else12: S = {x2}, set ε2 = R2 = ρ
(x1,x2)
13: Use yi to initialize estimates p2 via (1)14: end if15: else16: Let cs ∈ S be the nearest neighbour of xt in
S17: if ρ(cs,xt) ≤ εs (xt belongs to ball at cs)
then18: if yi 6= argmax
y∈Yps(y) then
19: // shrink radius on errors20: Set ms = ms + 1, εs = Rsm
−1/4s
21: else22: // update ball centre if correct prediction23: Set ∆ = xt − cs, ns = ns + 124: Set cs = cs + ∆/ns25: end if26: Use yi to update ps via (1)27: else28: S = S ∪ {xt}, set εt = Rt = ρ(cs,xt),
nt = 1, mt = 029: use yi to initialize estimates pt30: end if31: end if32: end for33: end for
5
Prediction. In the prediction phase, we proceed similarly:for each xt associated with an unlabelled video V i, itsnearest neighbour cπ(t) ∈ S is efficiently located. Then,assuming that the local features are i.i.d., the label of thetest video V i is predicted using the following maximumlikelihood estimate:
yi = argmaxy∈Y
Ti∏t=1
pπ(t)(y) . (2)
3.2 Constant Model SizeIn order to curb the system’s memory footprint, we adoptthe simple approach proposed in [5] based on deleting ex-isting balls whenever a given budget parameter on the la-bel query rate is attained. This is crucial for real-timeapplications, as NN search, used in both training and pre-diction, is logarithmic in the number of balls. The prob-ability of deleting any given ball is proportional to thenumber of mistakes made so far by the associated clas-sifier. Namely, if the budget is reached and a new ball hasto be added, an existing ball s is deleted with probability
Pdisc(s) =ms + 1∑
r∈S mr + |S|, where ms is the number of
mistakes made by ball s ∈ S. This helps addressing con-cept drift: ball classifiers that accumulate many mistakesare removed to make room for a more accurate descriptionof the data.
3.3 Streaming Active LearningWe now introduce the active learning system for stream-ing settings, which bounds the rate of queries to hu-man annotators. The technique we propose is inspiredfrom [32]. Whenever a new segmented video is presentedto the model, the system makes a prediction and then in-vokes the active learning module in order to determinewhether the label should be requested. In particular, ifthe confidence of the prediction is below a certain thresh-old, i.e., the prediction is ambiguous, then a query is is-sued to the annotator unless the query rate budget is vio-lated. When the label is not requested, the model is notupdated. Instead of selecting a fixed confidence thresholdon the query instances, we use the so-called Variable Un-certainty Strategy [32] (VarUnStr), which queries theleast certain instances within a time interval.
Measuring prediction confidence. Eq. (2) shows thatclass estimates pπ(t)(y) associated with ball centres nearthe current input instance should be considered more re-liable than those associated with faraway centres, as thecorresponding region of the feature space has alreadybeen explored (see Fig. 3-right). We thus adapt the RBFKernel [3] to scale ball estimates based on their distancefrom the input examples:
wt = exp
(−ρ(xt, cπ(t))
2
2ε2π(t)
), (3)
where the variance is set to the current ball radius επ(t).Given a test video V i = x1, . . . ,xTi
, we thus define aconfidence measure Ci(y) on the estimate of the expectedclass conditional probability for any given class y as:
Ci(y) =1
Ti
Ti∑t=1
wt log pπ(t)(y) ∀y ∈ Y. (4)
Updating the confidence threshold. TheVarUnStr [32] strategy continuously updates theconfidence threshold Θ, which triggers requests for newlabels (see Alg. 2). If the prediction confidence is belowthe current threshold Θ over the duration of the lastobserved video, Θ is decreased by a fraction τ in orderto query the most uncertain instances first. Otherwise,the threshold is increased to avoid interruptions of thelearning process when the algorithm is not asking forlabels. As explained in [32], the parameter τ can be setto a default value of 0.01. In the experimental sectionwe follow this suggestion, thus our algorithm remainsparameterless.
3.4 FIVER AlgorithmThe FIVER algorithm (see Alg. 3), combines all the ele-ments described above. Namely, FIVER trains the modelover the video stream via Alg. 1, while controlling thememory footprint as described in Sec. 3.2. Concurrently,Alg. 2 represents the active learning module, which asksonly for the most informative instances while not exceed-ing the budget rate.
4 ExperimentsTo emphasize the versatility of our approach, we havetested FIVER in both the batch and streaming learning
6
Algorithm 2 Variable Uncertainty StrategyInput: incoming video V i, classifier model, threshold
adjustment step τ ∈ (0, 1] (default is 0.01)Output: labeling ∈ {true, false}
1: Initialize: confidence threshold Θ = 1 and store thelatest value during operation
2: Calculate the confidence associated with the majorityclass Ci(yi) via (4)
3: if Ci(yi) < Θ then4: decrease the confidence threshold Θ = (1− τ)Θ5: return true6: else7: increase the confidence threshold Θ = (1 + τ)Θ8: return false9: end if
settings. In the batch setting, for each dataset we have fol-lowed the standard evaluation protocol (specific train-testsplits orK-fold cross-validation with specific values ofK—see below for details) and compared FIVER’s results tothose of competing incremental and offline methods. Inthe streaming setting, instead, we assessed different vari-ants of FIVER using the “online accuracy” or “sequentialrisk” [9] as evaluation measure. This measure capturesthe average error made by the sequence of incrementallylearned models in a procedure where we first predict thetest item on the current model, and then use the result toadjust the model itself.
Notably, the streaming setting used in our experimentsis very strict: we do not use seed training sets, mini-batchtraining, cross-validation sets, or assume any preliminaryknowledge on the number of classes. As the other incre-mental methods rely on much richer sources of informa-tion than those allowed in our streaming setting, we couldonly evaluate them in the batch setting.
4.1 Datasets and Feature ExtractionWe assessed our method on the following datasets:KTH [26] (all scenarios), UCF11 [24] and VIRAT [23] foraction recognition, SKIG [20] and MSRGesture3D [31]for gesture recognition, JAPVOW [21] and AUSLAN [15]for sign language recognition (UCI Repository [1]).
The first five datasets contain mostly footage material:we decided to extract efficient local features at frame level
Algorithm 3 FIVERInput: annotation budget B, maximum number of balls
M , video stream (V 1, y1), (V 2, y2), . . .1: Initialise online prediction accuracy A0 = 02: for i = 1, 2, . . . do3: Receive video V i
4: Predict yi (Eq 2)5: Update accuracyAi =
(1− 1
i
)Ai−1+ 1
i I{yi = yi}
6: if query rate ≤ budget B then7: if Query Strategy (Alg. 2) returns true then8: Request true label yi and update query rate9: Use (V i, yi) to update model (Alg. 1)
10: if |S| > M (memory exceeded) then11: Discard one ball (see Sec. 3.2)12: end if13: end if14: end if15: end for
in order to focus on truly real-time prediction. In par-ticular, from KTH, UCF11, and VIRAT sequences wecomputed improved dense trajectories [30], due to theiroutstanding performance in action recognition tasks. Foreach video, three types of features were extracted, namelyHistogram of Oriented Gradient (HOG), Histogram ofOptical Flow (HOF) and Motion Boundary Histogram(MBH). We ran the code published on the INRIA Website,keeping all the default parameters except for the trajectorylength, set to 8 frames, and the number of descriptor bins(16 BINs for HOG, MBHx and MBHy, and 18 BINs forHOF). Every 8 frames we obtained a variable number ofactive trajectories. We then accumulated all the trajecto-ries for each descriptor, and concatenated all the descrip-tors, obtaining a collection of vectors of 66 dimensionsfor each video. In this setting, each vector is a summaryof the three local descriptors extracted from each videoframe. For VIRAT, we initialised the improved trajectoryalgorithm using the bounding boxes released along withthe dataset. For the other datasets, we did not rely on anyinitialisation.
From SKIG we extracted the same information as [7],which consists of 3DHOF on the RBG frames and GHOG(Global Histogram of Oriented Gradient) on the depthframes. For MSRGesture3D only depth information
7
is available: we extracted two-level pyramidal HOG(PHOG) features using 32 bins. For all the experiments,the Euclidean distance has been used as metric ρ (seeAlg. 1), as pilot tests using an Lp norm with varying pdid not show any improvement.
DATASET FIVER Batch Increm.KTH 98.50% 95.00% [29] 97.00% [12]
UCF11 79.36% 76.10% [19] 66.00% [12]VIRAT 57.20% 55.40% [14] 54.20% [11]SKIG 98.30% 88.70% [20] 97.50% [4]
MSRG3D 91.25% 86.50% [31] 90.33% [4]JAPVOW 96.75% 95.67% [2] 98.01% [4]AUSLAN 72.60% 83.81% [2] 72.32% [4]
Table 2: Multiclass accuracies of FIVER comparedagainst best batch and incremental methods on sevenbenchmark datasets.
4.2 Comparison with competitors in abatch setting
We ran a first set of experiments in a batch setting, i.e.,running FIVER on a random permutation of the giventraining set and then applying the resulting classifier onthe test set. We compared FIVER against:Incremental algorithms [12, 11, 4], which follow an in-cremental learning approach similar to ours.Batch algorithms [29, 19, 14, 20, 31, 2], which have un-restricted access to training data for learning, as opposedto incremental methods that can access the data only se-quentially. Notably, the performance of incremental al-gorithms is typically poorer than that obtained using thecorresponding batch versions [6].
We used 5-fold cross-validation averaged on ten runsfor KTH, UCF11 and VIRAT, like the incremental com-petitors described in Sec. 2. For SKIG and MSRGes-ture3D we carried out a 3-fold cross-validation as in [31].The available training and test sets were used forJAPVOW and AUSLAN. As shown in Table 2, FIVERis among the best methods on all the datasets, no matterwhether batch or incremental setting is considered. Thisdemonstrates that our algorithm, combined with state-of-
the-art features, provides an accurate and efficient classi-fication system across the board.
4.3 FIVER performance in a streaming set-ting
In a pure streaming setting the data arrive sequentially,and the number of activities depicted in each video is notknown a priori. We conducted extensive experiments ex-ploring the following scenarios, which correspond to dif-ferent variants of FIVER:Full. This is the least realistic case, where we assumethat all the incoming instances are manually annotated, wehave no memory requirements, and we use each incomingtraining sample to incrementally update the model.VarUn. In this case, the active learning component de-scribed in Alg. 3 is used, including the VarUnStr strat-egy described in Alg. 2, to decide what instances requiremanual annotation. The query rate, calculated as the frac-tion of videos for which a label was requested amongthose observed so far [32], is upper bounded by an inputbudget parameter B ∈ (0, 1].Rnd. The Random strategy queries the labels of incominginstances with probability equal to the query rate budgetB.VarUnFix. In this very realistic scenario we also assumethat we have a limited memory space to store labeledtraining instances. We apply the method of Sec. 3.2 tolimit the number of balls stored in the model. In all ourtests we set the model size to 5000 instances. Notably,in BoW methods a large amount of codewords are gener-ally necessary to successfully predict video labels —seefor instance [30], where the authors use four different vi-sual vocabularies of 100,000 words (one for each localdescriptor).
We built ten random permutations of the videos in eachdataset. The algorithm had to predict the label of each newincoming video. After each prediction, if the active learn-ing system requested the true label, the video along withits label were fed to the model as a new training exam-ple. We ran all the competing algorithms with the samerange B ∈ {.05, .1, .15, .2, .25, .3, .35, .4, .45, .5, .75, 1}of budget values, and plotted the resulting online accu-racy, averaged over ten different streams, against the av-erage query rate. Importantly, the budget is only an upper
8
0 0.2 0.4 0.6 0.8 1
0.7
0.8
0.9
10.8%
0.1%
VarUnVarUnFixRndFull
(a) KTH
0 0.2 0.4 0.6 0.8 1
0.4
0.6
4%
0.2%
VarUnVarUnFixRndFull
(b) UCF11
0 0.2 0.4 0.6 0.8 1
0.52
0.54
0.56 1.8%
0.1%
VarUnVarUnFixRndFull
(c) VIRAT
0 0.2 0.4 0.6 0.8 1
0.4
0.6
0.82%1.8%
VarUnVarUnFixRndFull
(d) MSR3D
0 0.2 0.4 0.6 0.8 1
0.6
0.8
1%
0.9%
VarUnVarUnFixRndFull
(e) SKIG
0 0.2 0.4 0.6 0.8 1
0.6
0.8
11.2%1.1%
VarUnVarUnFixRndFull
(f) JAPVOW
0 0.2 0.4 0.6 0.8 1
0.4
0.6
0.8
0.3%0.28%
VarUnVarUnFixRndFull
(g) AUSLAN
0 0.2 0.4 0.6 0.8 10
0.2
0.4
0.6
0.8
1
Online PerfBalls Fraction
(h)
Figure 2: Plots (a) to (g) show the active online performance of the Full, Rnd, VarUn and VarUnFix variants ofFIVER on different benchmarks. The x-axis is the percentage of label requested by the active learning module, whilethe y-axis plots the average online accuracy over ten random permutations of the videos. In the coloured boxes, thepercentage of the input data selected as centres by VarUn(red) and VarUnFix(green) with budget B = 1 are shown.Plot h represents the evolution of accuracy and model size over the sequentially fed videos on the KTH dataset. Theblu (circle) curve shows the fraction of input data selected as centres, and the red (square) curve the online accuracy.Notably, the fraction of centers added diminish over time as the accuracy improves.
9
Figure 3: Left: evolution of 10 action class probabilities over time for a test sequence containing three actions. Middletop: the pink line plots the standard deviation of class probabilities for each frame of the same sequence —the cyancurve is their average standard deviation computed over a short interval of frames. For each segmented activity, aconfidence measure is computed (4) —the predicted activity label is discarded when confidence is below an adaptivethreshold Θ. Right: examples of three time series associated with low, medium and high confidence, respectively.
limit to the actual query rate —algorithms generally askfor a smaller number of annotations.
Note that FIVER does not need any validation set as ithas no parameters to tune. This is very important in thestreaming context, where non-adaptive methods whichtune their parameters in an initial validation stage mayperform suboptimally on future data. Plots a to g in Fig. 2represent the recorded performance on the various bench-marks for all the presented scenarios. The figure showsthat VarUn performs as well as Full on most datasets,even though it queries only around 50% of all the labels.On KTH, for example, VarUn achieves 90% online ac-curacy while accessing only less than 20% of the labels.The Rnd method performs typically worse and needs allthe labels to reach the performance of Full. VarUnFixworks almost as well as VarUn on the simplest datasetsand slightly worse on the complex ones; this is due to thefixed budget control that has to discard information in or-der to keep the model size fixed. For example, both VarUnand Rnd use 4% of the input data around the 50% queryrate for UCF11, whereas VarUnFix use only 0.2% of thedata –this is shown in the red and green boxes in Fig. 2respectively, as final percentage of input examples usedas model centres. Therefore, VarUnFix is extremely goodat compressing the data, and allows for efficient computa-tion at the cost of a little performance degradation.
4.4 Active Continuous Activity Recognition
Although in Sec. 3 we assumed that the incoming videosVi are pre-segmented, whenever feature vectors {x(i)
t }Tit=1
are extracted on a frame-by-frame basis we can exploit theactivity scores (2) computed over a short temporal win-dow to perform automated temporal segmentation. Thissegmentation procedure is based on the evolution of classprobabilities over time [4] (Fig. 3.Left), where transi-tions between action instances can be associated with lo-cal minima of the standard deviation of class scores (pinkcurve) over the temporal window (Fig. 3.Middle-top).
In addition, unlike what has been done in [4], we usethe confidence measure (4) to discard or send to supervi-sion any detected activity with confidence below a cer-tain threshold (Fig. 3,Middle-bottom), as discussed inSec. 3.3. This is crucial in applications such as human-robot interaction, where it is preferable for the robot notto perform any action when prediction confidence is low,as this may lead to safety issues or communication errors.
We tested this active approach to temporal segmenta-tion on the same dataset of ten manipulative actions usedin [4]. Each action was recorded 60 times in two differentillumination settings and backgrounds, and 3DHOF andHOG descriptors were extracted for each frame. We ex-cluded four out of ten gestures from the learning phase,
10
and evaluated our algorithm on sequences representingpick and place activities formed by grasping, moving andreleasing actions. The system was evaluated on its abil-ity to predict the correct class when a known gesture wasperformed, and to request supervision when an unknowngesture was observed.
To compare the estimated class sequence with theground truth we employed the Levenshtein distance [18],originally used in [4]: S+D+I
N . In this case, each action istreated as a symbol in a sequence – S represents the num-ber of substitutions (misclassifications), D the number ofdeletions (false negatives) and I the number of insertions(false positives). Over 20 test sequences, we achieved aLevenshtein distance error of 0.14, compared to the 0.36reported in [4].
5 Conclusion and future work
We presented an incremental active human activity recog-nition framework, well suited for streaming recognitionproblems, especially when the amount of data to processis large. Our approach exhibits a number of desirablefeatures: it deals with sets of local descriptors extractedfrom videos, it learns in an incremental fashion, it embedsan active learning module, it is capable of learning newclasses on the fly, it limits memory usage, and it predictsnew data in real-time. In addition, the method is nonpara-metric and does not require expensive validation sessionsfor training, as it has no parameters to be tuned. Resultsdemonstrate its competitiveness in terms of accuracy withrespect to traditional batch approaches, as well as promis-ing performance in a truly streaming scenario. Future re-search will explore the use of confidence measures to au-tomatically discover new activity classes by associatingthem with low confidence trajectories (see Fig. 3, right).
References
[1] UCI machine learning repository, 2007.
[2] A. Antonucci, R. de Rosa, A. Giusti, and F. Cuz-zolin. Robust classification of multivariate time se-ries by imprecise hidden markov models. In Int. J.Approx. Reasoning IJAR, 2014.
[3] Y.-W. Chang, C.-J. Hsieh, K.-W. Chang, M. Ring-gaard, and C.-J. Lin. Training and testing low-degree polynomial data mappings via linear svm.The Journal of Machine Learning Research, 11,2010.
[4] R. De Rosa, N. Cesa-Bianchi, I. Gori, and F. Cuz-zolin. Online action recognition via nonparametricincremental learning. In BMVC, 2014.
[5] R. De Rosa, F. Orabona, and N. Cesa-Bianchi. TheABACOC algorithm: a novel approach for nonpara-metric classification of data streams. In Data Mining(ICDM), 2015 IEEE International Conference on.IEEE, 2015.
[6] O. Dekel and Y. Singer. Data-driven online to batchconversions. In Advances in Neural InformationProcessing Systems, pages 267–274, 2005.
[7] S. R. Fanello, I. Gori, G. Metta, and F. Odone. Keepit simple and sparse: Real-time action recognition.J. Mach. Learn. Res., 14(1), Jan. 2013.
[8] M. M. Gaber, A. Zaslavsky, and S. Krishnaswamy.A survey of classification methods in data streams.In Data Streams, pages 39–59. Springer, 2007.
[9] J. Gama, R. Sebastiao, and P. P. Rodrigues. On eval-uating stream learning algorithms. Machine Learn-ing, 2013.
[10] L. Gyrfi, M. Kohler, A. Krzyzak, and H. Walk.A distribution-free theory of nonparametric regres-sion. Springer series in statistics. Springer, NewYork, Berlin, Paris, 2002.
[11] M. Hasan and A. K. Roy-Chowdhury. Continuouslearning of human activity models using deep nets.In Computer Vision–ECCV 2014. Springer, 2014.
[12] M. Hasan and A. K. Roy-Chowdhury. Incremen-tal activity modeling and recognition in streamingvideos. In Computer Vision and Pattern Recognition(CVPR), 2014 IEEE Conference on. IEEE, 2014.
[13] M. Hasan and A. K. Roy-Chowdhury. Context awareactive learning of activity recognition models. InComputer Vision (ICCV), 2015 IEEE InternationalConference on. IEEE, 2015.
[14] Y.-G. Jiang, C.-W. Ngo, and J. Yang. Towards op-timal bag-of-features for object categorization andsemantic video retrieval. In Proceedings of the 6th
11
ACM international conference on Image and videoretrieval. ACM, 2007.
[15] J. Kies. Empirical Methods for Evaluating Video-Mediated Collaborative Work. PhD thesis, VirginiaTech, 1997.
[16] S. Kpotufe and F. Orabona. Regression-tree tuningin a streaming setting. In NIPS, 2013.
[17] R. Krauthgamer and J. R. Lee. Navigating nets:Simple algorithms for proximity search. In Proceed-ings of the Fifteenth Annual ACM-SIAM Symposiumon Discrete Algorithms, SODA ’04, Philadelphia,PA, USA, 2004. Society for Industrial and AppliedMathematics.
[18] V. Levenshtein. Binary codes capable of correctingdeletions, insertions, and reversals. Soviet Physics -Doklady, 1966.
[19] J. Liu, Y. Yang, and M. Shah. Learning semantic vi-sual vocabularies using diffusion distance. In Com-puter Vision and Pattern Recognition, 2009. CVPR2009. IEEE Conference on. IEEE, 2009.
[20] L. Liu and L. Shao. Learning discriminative repre-sentations from rgb-d video data. In Proceedings ofthe Twenty-Third International Joint Conference onArtificial Intelligence, IJCAI’13. AAAI Press, 2013.
[21] J. T. M. Kudo and M. Shimbo. Multidimensionalcurve classification using passing-through regions.Pattern Recognition Letters, 20, 1999.
[22] R. Minhas, A. A. Mohammed, and Q. J. Wu. Incre-mental learning in human action recognition basedon snippets. Circuits and Systems for Video Tech-nology, IEEE Transactions on, 22(11), 2012.
[23] S. Oh, A. Hoogs, A. Perera, N. Cuntoor, C.-C.Chen, J. T. Lee, S. Mukherjee, J. Aggarwal, H. Lee,L. Davis, et al. A large-scale benchmark dataset forevent recognition in surveillance video. In ComputerVision and Pattern Recognition (CVPR), 2011 IEEEConference on. IEEE, 2011.
[24] R. Poppe. A survey on vision-based human actionrecognition. Image and vision computing, 28(6),2010.
[25] K. K. Reddy, J. Liu, and M. Shah. Incremental ac-tion recognition using feature-tree. In Computer Vi-sion, 2009 IEEE 12th International Conference on.IEEE, 2009.
[26] C. Schuldt, I. Laptev, and B. Caputo. Recognizinghuman actions: a local SVM approach. In Proc.of International Conference on Pattern Recognition,2004.
[27] J. Sivic and A. Zisserman. Video google: A text re-trieval approach to object matching in videos. InComputer Vision, 2003. Proceedings. Ninth IEEEInternational Conference on. IEEE, 2003.
[28] A. Tsymbal. The problem of concept drift: Defini-tions and related work. Technical Report TCD-CS-2004-15, Trinity College Dublin, 2004.
[29] H. Wang, A. Klaser, C. Schmid, and C.-L. Liu. Ac-tion recognition by dense trajectories. In ComputerVision and Pattern Recognition (CVPR), 2011 IEEEConference on. IEEE, 2011.
[30] H. Wang and C. Schmid. Action recognitionwith improved trajectories. In Computer Vision(ICCV), 2013 IEEE International Conference on.IEEE, 2013.
[31] J. Wang, Z. Liu, J. Chorowski, Z. Chen, and Y. Wu.Robust 3d action recognition with random occu-pancy patterns. In Proceedings of the 12th Euro-pean Conference on Computer Vision - Volume PartII, ECCV’12, Berlin, Heidelberg, 2012. Springer-Verlag.
[32] I. Zliobaite, A. Bifet, B. Pfahringer, and G. Holmes.Active learning with drifting streaming data. IEEEtransactions on neural networks and learning sys-tems, 25(1), 2014.
12
