IEEE TRANSACTIONS PATTERN ANALYSIS AND MACHINE ...xgwang/papers/ouyangZWpami16.pdf · Citation information: DOI 10.1109/TPAMI.2016.2587642, IEEE Transactions on Pattern Analysis and

0162-8828 (c) 2016 IEEE Personal use is permitted but republicationredistribution requires IEEE permission See httpwwwieeeorgpublications_standardspublicationsrightsindexhtml for more information

This article has been accepted for publication in a future issue of this journal but has not been fully edited Content may change prior to final publication Citation information DOI 101109TPAMI20162587642 IEEETransactions on Pattern Analysis and Machine Intelligence

IEEE TRANSACTIONS PATTERN ANALYSIS AND MACHINE INTELLIGENCE 1

DeepID-Net Object Detection with DeformablePart Based Convolutional Neural Networks

Wanli Ouyang Member IEEE Xingyu Zeng Student Member IEEE Xiaogang Wang Member IEEE Shi Qiu Member IEEE Ping Luo Member IEEE

Yonglong Tian Student Member IEEE Hongsheng Li Member IEEE Shuo Yang Student Member IEEE Zhe Wang Student Member IEEE Hongyang Li Kun Wang Junjie Yan

Chen-Change Loy Member IEEE Xiaoou Tang Fellow IEEE

AbstractmdashIn this paper we propose deformable deep convolutional neural networks for generic object detection This new deeplearning object detection framework has innovations in multiple aspects In the proposed new deep architecture a new deformationconstrained pooling (def-pooling) layer models the deformation of object parts with geometric constraint and penalty A new pre-trainingstrategy is proposed to learn feature representations more suitable for the object detection task and with good generalization capabilityBy changing the net structures training strategies adding and removing some key components in the detection pipeline a set ofmodels with large diversity are obtained which significantly improves the effectiveness of model averaging The proposed approachimproves the mean averaged precision obtained by RCNN [16] which was the state-of-the-art from 31 to 503 on the ILSVRC2014detection test set It also outperforms the winner of ILSVRC2014 GoogLeNet by 61 Detailed component-wise analysis is alsoprovided through extensive experimental evaluation which provides a global view for people to understand the deep learning objectdetection pipeline

Index TermsmdashCNN convolutional neural networks object detection deep learning deep model

1 INTRODUCTION

Object detection is one of the fundamental challenges incomputer vision and has attracted a great deal of researchinterest [11] [50] Intra-class variation in appearance anddeformation are among the main challenges of this task

Because of its power in learning features the convolutionalneural network (CNN) is being widely used in recent large-scale detection and recognition systems [60] [54] [21] [27][82] [81] Since training deep models is a non-convex op-timization problem with millions of parameters the choiceof a good initial point is a crucial but unsolved problemespecially when deep CNN goes deeper [60] [54] [27] Itis also easy to overfit to a small training set Researchers findthat supervised pretraining on large-scale image classificationdata and then finetuning for the target object detection taskis a practical solution [10] [45] [78] [16] However weobserve that there is still a gap between the pretraining taskand the finetuning task that makes pretraining less effectiveThe problem of the training scheme is the mismatch betweenpretraining for the image classification task and fine-tuning forthe object detection task For image classification the input isa whole image and the task is to recognize the object within

bull Wanli Ouyang (equal contribution) Xingyu Zeng (equal contribution)Xiaogang Wang Hongsheng Li Zhe Wang and Hongyang Li are withthe Department of Electronic Engineering at the Chinese University ofHong Kong Hong Kong Shi Qiu Ping Luo Yonglong Tian Shuo YangChen-Change Loy and Xiaoou Tang are with the Department of ElectronicEngineering at the Chinese University of Hong Kong Hong Kong WanliOuyang and Xiaogang Wang are corresponding authorsE-mail wlouyang xgwangeecuhkeduhk

this image Therefore learned feature representations haverobustness to scale and location change of objects in imagesTaking Fig 1(a) as an example no matter how large and wherea person is in the image the image should be classified asperson However robustness to object size and location is notrequired for object detection For object detection candidateregions are cropped and warped before they are used as inputof the deep model Therefore the positive candidate regionsfor the object class person should have their locations alignedand their sizes normalized On the contrary the deep modelis expected to be sensitive to the change in position andsize in order to accurately localize objects An example toillustrate the mismatch is shown in Fig 1 (a) Because of suchmismatch the image classification task is not an ideal choiceto pretrain the deep model for object detection Therefore anew pretraining scheme is proposed to train the deep modelfor object detection more effectively

Part deformation handling is a key factor for the recentprogress in object detection [12] [83] [13] [73] [37] Ournew CNN layer is motivated by three observations Firstdeformable visual patterns are shared by objects of differentcategories For example the circular visual pattern is sharedby both banjo and ipod as shown in Fig 1(b) Second theregularity on deformation exists for visual patterns at differentsemantic levels For example human upper bodies humanheads and human mouths are parts at different semantic levelswith different deformation properties Third a deformable partat a higher level is composed of deformable parts at a lowerlevel For example a human upper body is composed of a headand other body parts With these observations we design a newdeformation-constrained pooling (def-pooling) layer to learn




Deep model Deformable pattern Examples

Pattern

Deformationpenalty

Pattern

Deformationpenalty

Image

Visualizedmodel

Pretrained on object-level annoation Pretrained on image-level annotation

(a)

(b)

Higher penalty

Figure 1 The motivation of this paper in new pretrainingscheme (a) and jointly learning feature representation anddeformable object parts shared by multiple object classes atdifferent semantic levels (b) In (a) a model pretrained on image-level annotation is more robust to size and location changewhile a model pretrained on object-level annotation is better inrepresenting objects with tight bounding boxes In (b) when ipodrotates its circular pattern moves horizontally at the bottom ofthe bounding box Therefore the circular patterns have smallerpenalty moving horizontally but higher penalty moving verticallyThe curvature part of the circular pattern are often at the bottomright positions of the circular pattern Magnitudes of deformationpenalty are normalized to make them comparable across thetwo examples in (a) for visualization Best viewed in color

the shared visual patterns and their deformation propertiesfor multiple object classes at different semantic levels andcomposition levels

The performance of deep learning systems depends signifi-cantly on implementation details [4] However an evaluationof the performance of the recent deep architectures on thecommon ground for large-scale object detection is missingAs a respect to the investigation on details in deep learning[4] [16] this paper compares the performance of recentdeep models including AlexNet [25] ZF [75] Overfeat [52]and GoogLeNet [60] under the same setting for differentpretraining-finetuning schemes We also provide experimentalanalysis on the properties that cause the accuracy variation indifferent object classes

In this paper we propose a deformable deep convolutionalneural network for object detection named as DeepID-NetIn DeepID-Net we jointly learn the feature representationand part deformation for a large number of object cate-gories We also investigate many aspects in effectively andefficiently training and aggregating the deep models includingbounding box rejection training schemes context modeling

and model averaging The proposed new framework signif-icantly advances the state-of-the-art for deep learning basedgeneric object detection such as the well known RCNN [16]framework This paper also provides detailed component-wiseexperimental results on how our approach can improve themean Averaged Precision (AP) obtained by RCNN [16] from310 to mean AP 503 step-by-step on the ImageNet LargeScale Visual Recognition Challenge 2014 (ILSVRC2014) ob-ject detection task

The contributions of this paper are as follows1) A new deep learning framework for object detection It

effectively integrates feature representation learning partdeformation learning context modeling model averagingand bounding box location refinement into the detec-tion system Detailed component-wise analysis is providedthrough extensive experimental evaluation This paper isalso the first to investigate the influence of CNN structuresfor the large-scale object detection task under the samesetting By changing the configuration of this frameworkmultiple detectors with large diversity are generated whichleads to more effective model averaging

2) A new scheme for pretraining the deep CNN modelWe propose to pretrain the deep model on the ImageNetimage classification and localization dataset with 1000-class object-level annotations instead of with image-levelannotations which are commonly used in existing deeplearning object detection [16] [60] Then the deep model isfine-tuned on the ImageNetPASCAL-VOC object detectiondataset with 20020 classes which are the target objectclasses in the two datasets

3) A new deformation constrained pooling (def-pooling) layerwhich enriches the deep model by learning the deformationof object parts at any information abstraction levels Thedef-pooling layer can be used for replacing the max-poolinglayer and learning the deformation properties of parts

4) Analysis on the object properties that influence the variationin object detection accuracy for different classesPreliminary version of this paper is published in [38] This

paper include more analysis on the proposed approach and addexperimental investigation on the properties that influence theaccuracy in detecting objects

The models pretrained by both image-level annotation andobject-level annotation for AlexNet [25] ZF [75] overfeat[52] and GoogLeNet [60] and the models after fine-tuningon ILSVRC2014 are provided online 1

2 RELATED WORK

Since many objects have non-rigid deformation the abilityto handle deformation improves detection performance De-formable part-based models were used in [12] [83] [41][70] [65] for handling translational movement of parts Tohandle more complex articulations size change and rotationof parts were modeled in [13] and mixture of part appearanceand articulation types were modeled in [3] [72] A dictionaryof shared deformable patterns was learned in [20] In theseapproaches features were manually designed

1 wwweecuhkeduhksimwlouyangprojectsImageNet




Because of the power on learning feature representationdeep models have been widely used for object recognition de-tection and other vision tasks [25] [52] [75] [21] [53] [85][19] [27] [16] [35] [40] [76] [77] [33] [29] [57] [59][57] [58] [31] [32] [30] [80] [79] [68] [69] [15] [47][26] [46] [34] [39] [42] [43] Krizhevsky et al proposeda neural network with 60 million parameters and 650000neurons [25] This neural network was the first to show thepower of deep CNN in large-scale computer vision task Lateron Razavian et al [45] showed that the OverFeat network[52] trained for object classification on ILSVRC13 was a goodfeature representation for the diverse range of recognition tasksof object image classification scene recognition fine grainedrecognition attribute detection and image retrieval applied toa diverse set of datasets As a further step of using the modeltrained for ImageNet object classification Girshick et al foundfinetuning the CNN pretrained on the ImageNet object classi-fication to be effective on various object detection benchmarkdatasets [16] In existing deep CNN models max pooling andaverage pooling were useful in handling deformation but couldnot learn the deformation penalty and geometric models ofobject parts The deformation layer was first proposed in [36]for pedestrian detection In this paper we extend it to generalobject detection on ImageNet In [36] the deformation layerwas constrained to be placed after the last convolutional layerIn this work the def-pooling layer can be placed after all theconvolutional layers to capture geometric deformation at allthe information abstraction levels In [36] it was assumedthat a pedestrian only had one instance of a body part soeach part filter only had one optimal response in a detectionwindow In this work it is assumed that an object has multipleinstances of a part (eg a car has many wheels) so each partfilter is allowed to have multiple response peaks in a detectionwindow Moreover we allow multiple object categories toshare deformable parts and jointly learn them with a singlenetwork This new model is more suitable for general objectdetection

The use of context has gained attention in recent workson object detection The context information investigated inliterature includes regions surrounding objects [5] [8] [14]object-scene interaction [9] [22] and the presence locationorientation and size relationship among objects [2] [64] [66][7] [44] [14] [56] [9] [74] [8] [71] [40] [6] [51] [61]In this paper we use whole-image classification scores overa large number of classes from a deep model as globalcontextual information to refine detection scores

Besides feature learning deformation modeling and contextmodeling there were also other important components in theobject detection pipeline such as pretraining [16] networkstructures [52] [75] [25] refinement of bounding box loca-tions [16] and model averaging [75] [25] [21] While thesecomponents were studies individually in different works weintegrate them into a complete pipeline and take a globalview of them with component-wise analysis under the sameexperimental setting It is an important step to understand andadvance deep learning based object detection

Image Proposed bounding boxes

Selective search +

Edgeboxes

DeepID-Net(Pretrain

def-pooling)

Box rejection

Context modeling

Remaining bounding boxes

Model averaging

Bounding box regression

person

horse

person

horse

person

horse

person

horse

Final result Single-model result

Figure 2 Overview of our approach Detailed description isgiven in Section 31 Texts in red highlight the steps that arenot present in RCNN [16]

3 METHOD

31 Overview of our approach

An overview of our proposed approach is shown in Fig 2 Wetake the ImageNet object detection task as an example TheImageNet image classification and localization dataset with1000 classes is chosen to pretrain the deep model Its objectdetection dataset has 200 object classes In the experimentalsection the approach is also applied to the PASCAL VOC Thepretraining data keeps the same while the detection datasetonly has 20 object classes The steps of our approach aresummarized as follows1) Selective search proposed in [55] and edgeboxes proposed

in [84] are used to propose candidate bounding boxes2) An existing detector RCNN [16] in our experiment is

used to reject bounding boxes that are most likely to bebackground

3) An image region in a bounding box is cropped and fedinto the DeepID-Net to obtain 200 detection scores Eachdetection score measures the confidence on the croppedimage containing one specific object class Details are givenin Section 32

4) The 1000-class whole-image classification scores of a deepmodel are used as contextual information to refine thedetection scores of each candidate bounding box Detailsare given in Section 36

5) Average of multiple deep model outputs is used to improvethe detection accuracy Details are given in Section 37

6) Bounding box regression proposed in RCNN [16] is usedto reduce localization errors

32 Architecture of DeepID-Net

DeepID-Net in Fig 3 has three parts(a) The baseline deep model The ZF model proposed in [ 75]

is used as the default baseline deep model when it is notspecified

(b) Branches with def-pooling layers The input of these layersis the conv5 the last convolutional layer of the baselinemodel The output of conv5 is convolved with part filtersof variable sizes and the proposed def-pooling layers inSection 34 are used to learn the deformation constraintof these part filters Parts (a)-(b) output 200-class objectdetection scores For the cropped image region that containsa horse as shown in Fig 3(a) its ideal output should havea high score for the object class horse but low scores forother classes




(a) Existing deep model (ZF)conv5 fc6 fc7

128

128

128

128

conv61

conv63

conv71

conv73

def61

def63

Image

Candidate region

(b) Layers with def-pooling layer

Refined 200-class scores

1000-class scores

200-class scores

(c) Deep model (clarifai-fast) for 1000-class image classification

Figure 3 Architecture of DeepID-Net with three parts (a) base-line deep model which is ZF [75] in our single-model detector(b) layers of part filters with variable sizes and def-pooling layers(c) deep model to obtain 1000-class image classification scoresThe 1000-class image classification scores are used to refinethe 200-class bounding box classification scores

(c) The deep model (ZF) to obtain image classification scoresof 1000 classes Its input is the whole image as shownin Fig 3(c) The image classification scores are used ascontextual information to refine the classification scores ofbounding boxes Detail are given in Section 36

33 New pretraining strategy

The widely used training scheme in deep learning based objectdetection [16] [78] [60] including RCNN is denoted byScheme 0 and described as follows1) Pretrain deep models by using the image classification task

ie using image-level annotations from the ImageNet imageclassification and localization training data

2) Fine-tune deep models for the object detection task ieusing object-level annotations from the object detectiontraining data The parameters learned in Step (1) are usedas initialization

The deep model structures at the pretraining and fine-tuningstages are only different in the last fully connected layer forpredicting labels (1 000 classes for the ImageNet classificationtask vs 200 classes for the ImageNet detection task) Exceptfor the last fully connected layers for classification the pa-rameters learned at the pretraining stage are directly used asinitial values for the fine-tuning stage

We propose to pretrain the deep model on a large auxiliaryobject detection training data instead of the image classifica-tion data Since the ImageNet Cls-Loc data provides object-level bounding boxes for 1000 classes more diverse in contentthan the ImageNet Det data with 200 classes we use theimage regions cropped by these bounding boxes to pretrainthe baseline deep model in Fig 3(a) The proposed pretrainingstrategy is denoted as Scheme 1 and bridges the image- vsobject-level annotation gap in RCNN

1) Pretrain the deep model with object-level annotations of1 000 classes from ImageNet Cls-Loc train data

2) Fine-tune the deep model for the 200-class object detectiontask ie using object-level annotations of 200 classes fromImageNet Det train and val1 (validation set 1) data Usethe parameters in Step (1) as initialization

Compared with the training scheme of RCNN experimentalresults show that the proposed scheme improves mean AP by45 on ImageNet Det val2 (validation set 2) If only the 200target classes (instead of 1000 classes) from the ImageNetCls-Loc train data are selected for pretraining in Step (1) themean AP on ImageNet Det val2 drops by 57

Another potential mismatch between pretraining and fine-tuning comes from the fact that the ImageNet classificationand localization (Cls-Loc) data has 1 000 classes while theImageNet detection (Det) data only targets on 200 classesmost of which are within the 1 000 classes In many practicalapplications the number of object classes to be detected issmall People question the usefulness of auxiliary training dataoutside the target object classes Our experimental study showsthat feature representations pretrained with 1 000 classes havebetter generalization capability which leads to better detectionaccuracy than pretraining with a subset of the Cls-Loc dataonly belonging to the 200 target classes in detection

34 Def-pooling layer341 DPM and its relationship with CNNIn the deformable part based model (DPM) [12] for objectdetection the following steps are used at the testing stage1) Extract HOG feature maps from the input image2) Obtain the part detection score maps by filtering the HOG

feature maps with the learned part filtersdetectors The partfilters are learned by latent SVM

3) Obtain deformable part score maps by subtracting defor-mation penalty from part detection score maps

4) Sum up the deformable part score maps from multiple partsto obtain the final object detection score map

Denote the convolutional layer at the lth layer by conv lDenote the output maps of convl by Ml The steps abovefor DPM have the following relationship for CNN1) The HOG feature map in DPM corresponds to the output

of a convolutional layer Consecutive convolutional layersand pooling layers can be considered as extracting featuremaps from input image

2) The part detection maps in DPM correspond to the outputresponse maps of the convolutional layer For example theoutput of convlminus1 is the feature maps Mlminus1 which aretreated as input feature maps of convl Filtering on HOGfeature maps using part filters in DPM is similar to filteringon the feature maps Mlminus1 using the filters of convl inCNN Each output channel of convl corresponds to a partdetection map in DPM The filter of convl for an outputchannel corresponds to a part filter in DPM The responsemap in CNN is called part detection map in the followingof this paper

3) The deformation penalty in DPM for each part correspondsto the deformation penalty in our proposed def-poolinglayer for CNN Details are given in Section 342




Extract HOG feature

Extract feature by convolutional

layers

Filter HOG feature map with part

detector

Subtract deformation penalty from part

detection score maps

Sum up deformable part scoresDPM

CNNFilter CNN feature

map with a convolutional layer

Subtract deformation penalty by def-pooling

layer

Sum up deformable part scores by a special

convolutional layer

Figure 4 The relationship between the operations in the DPMand the CNN

4) Summing up the deformable part score maps for multipleparts in DPM is similar to summing up multiple def-poolinglayer output maps with a special case of convolutional layerDef-pooling layer output maps can be summed up by usinga convolutional layer after the def-pooling layer with filtersize 1 times 1 and filter coefficients being constant 1 for theconvolutional layer

In summary HOG feature extraction part detector filteringdeformation penalty subtraction and part detection scoresaggregation in DPM [12] have their corresponding operationsin the CNN as shown in Fig 4

342 Definition of the def-pooling layerSimilar to max-pooling and average-pooling the input of adef-pooling layer is the output of a convolutional layer Theconvolutional layer produces C maps of size W timesH DenoteMc as the cth map Denote the (i j)th element of Mc bym

(ij)c i = 1 W j = 1 H The def-pooling layer

takes a small block with center (x y) and size (2R + 1) times(2R+1) from the Mc and produce the element of the outputas follows

b(xy)c = maxδxδyisinminusRmiddotmiddotmiddot R

mc(zδxδy δx δy) (1)

where x = 1 W y = 1 H (2)

mc(zδxδy δx δy) = mzδxδyc minus φ(δx δy) (3)

zδxδy = [x y]T + [δx δy]T (4)

φ(δx δy) =

Nsum

n=1

acndδxδycn (5)

bull (x y) denotes the assumed anchor location of object partbull (δx δy) denotes the translationdisplacement of object part

from the anchor positionbull zδxδy as defined in (4) is the deformed location from the

assumed anchor positionbull m

zδxδyc in (3) is the element in Mc at the location zδxδy

It is considered as the score of matching the cth part filterwith the features at the deformed location zδxδy

bull φ(δx δy) in (3) and (5) is the deformation penalty ofplacing the part from the assumed anchor position (x y)

to the deformed location zδxδy acn and dδxδycn in (5)

are parameters of deformation that can be pre-defined orlearned by back-propagation (BP) N denotes the numberof parameters acn and d

δxδycn

bull mc(zδxδy δx δy) in (1) and (3) is the deformable partscore It is obtained by subtracting the deformation penaltyφ(δx δy) from the visual matching score m

zδxδyc

bull b(xy)c is the (x y)th element of the output of the def-

pooling layer For the anchor location (x y) b(xy)c is

obtained by taking the maximum deformable part score

mc(zδxδy δx δy) within the displacement range R ieδx δy isin minusR middot middot middot RThe def-pooling layer can be better understood through the

following examplesExample 1 If N = 1 an = 1 dδxδy1 = 0 for |δx| |δy| le k

and dδxδy1 = infin for |δx| |δy| gt k then this corresponds

to max-pooling with kernel size k It shows that the max-pooling layer is a special case of the def-pooling layerPenalty becomes very large when deformation reaches certainrange Since the use of different kernel sizes in max-poolingcorresponds to different maps of deformation penalty that canbe learned by BP def-pooling provides the ability to learn themap that implicitly decides the kernel size for max-pooling

Example 2 If N = 1 and an = 1 then dδxδy1 is the

deformation parameterpenalty of moving a part from theanchor location (x y) by (δx δy) If the part is not allowed tomove we have d001 = 0 and d

δxδy1 = infin for (δx δy) = (0 0)

If the part has penalty 1 when it is not at the assumedlocation (x y) then we have d00

1 = 0 and dδxδy1 = 1 for

(δx δy) = (0 0) It allows to assign different penalty todisplacement in different directions If the part has penalty2 moving leftward and penalty 1 moving rightward then wehave d

δxδy1 = 1 for δx lt 0 and d

δxδy1 = 2 for δx gt 0 Fig

6 shows some learned deformation parameters dδxδy1 Fig 7

shows some visualized partsExample 3 The deformation layer in [36] is a special

case of the def-pooling layer by enforcing that z δxδy in (1)covers all the locations in convlminus1i and only one output witha pre-defined location is allowed for the def-pooling layer(ie R = infin sx = W and sy = H) The proof can befound in Appendix A To implement quadratic deformationpenalty used in [12] we can predefine dδxδycn n=1234 =δx δy (δx)2 (δy)2 and learn parameters an As shown inAppendix A the def-pooling layer under this setting canrepresent deformation constraint in the deformable part basedmodel (DPM) [12] and the DP-DPM [18]

Take Example 2 as an example for BP learning acn is theparameter in this layer and dlowast is pre-defined constant Thenwe have

partb(xy)c

partacn= minusd

(ΔxΔy)cn

(ΔxΔy) = argmaxδxδyisinminusRmiddotmiddotmiddot Rmzδxδyc minus φ(δx δy)

(6)

where (ΔxΔy) is the position with the maximum value in(1) The gradients for the parameters in the layers before thedef-pooling layer are back-propagated like max-pooling layer

Similar to max-pooling and average pooling subsamplingcan be done as follows

b(xy)c = b(sxmiddotxsymiddoty)c (7)

For Mc of size W times H the subsampled output has sizeWsx

times Hsy

Therefore multiple instances of an object part atmultiple anchor locations are allowed for each part filer

In our implementation N = 1 R = 2 and Example 2is used for def-pooling there are no fully connected layersafter conv71 2 3 in Fig 3 We did not find improvement onImageNet by further increasing N and R Further study on Nand R could be done on other datasets and particular categoriesin the future work




filter

Input Convolution result M

Deformationpenalty

Output B

Maxpooling

Viewedas

blocks

Block-wiseaddition

Figure 5 Def-pooling layer The part detection map and thedeformation penalty are summed up Block-wise max pooling isthen performed on the summed map to obtain the output B ofsize H

sytimes W

sx(3times 1 in this example)

Figure 6 The learned deformation penalty for different visualpatterns The penalties in map 1 are low at diagonal positionsThe penalties in map 2 and 3 are low at vertical and horizontallocations separately The penalties in map 4 are high at thebottom right corner and low at the upper left corner

343 Analysis

A visual pattern has different spatial distributions in differentobject classes For example traffic lights and ipods havegeometric constraints on the circular visual pattern in Fig8 The weights connecting the convolutional layers conv7 1 -conv73 in Fig 3 and classification scores are determined bythe spatial distributions of visual patterns for different classesFor example the car class will have large positive weights inthe bottom region but negative weights in the upper region forthe circular pattern On the other hand the traffic light classwill have positive weights in the upper region for the circularpattern

A single output of the convolutional layer conv7 1 in Fig 3is from multiple part scores in def61 The relationship betweenparts of the same layer def61 is modeled by conv71

Circular Bird body Shape of bird or Starfish

Bottle neck HandleFoot or stand

Repeated circular shape

Pedicle Wheel Wheel

Patterns of animal head

Figure 7 The learned part filters visualized using deepdraw [ 1]

Figure 8 Repeated visual patterns in multiple object classes

The def-pooling layer has the following advantages1) It can replace any pooling layer and learn deformation

of parts with different sizes and semantic meanings Forexample at a higher level visual patterns can be large partseg human upper bodies and the def-pooling layer cancapture the deformation constraint of human upper partsAt a middle level the visual patterns can be smaller partseg heads At the lowest level the visual patterns can bevery small eg mouths A human upper part is composedof a deformable head and other parts The human head iscomposed of a deformable mouth and other parts Objectparts at different semantic abstraction levels with differentdeformation constraints are captured by def-pooling layersat different levels The composition of object parts isnaturally implemented by CNN with hierarchical layers

2) The def-pooling layer allows for multiple deformable partswith the same visual cue ie multiple response peaks areallowed for one filter This design is from our observationthat an object may have multiple object parts with the samevisual pattern For example three light bulbs co-exist in atraffic light in Fig 5

3) As shown in Fig 3 the def-pooling layer is a sharedrepresentation for multiple classes and therefore the learnedvisual patterns in the def-pooling layer can be shared amongthese classes As examples in Fig 8 the learned circularvisual patterns are shared as different object parts in trafficlights cars and ipods

The layers proposed in [36] [18] does not have these ad-vantages because they can only be placed after the finalconvolutional layer assume one instance per object part anddoes not share visual patterns among classes

35 Fine-tuning the deep model with hinge-loss

In RCNN feature representation is first learned with thesoftmax loss in the deep model after fine-tuning Then in aseparate step the learned feature representation is input to alinear binary SVM classifier for detection of each class In ourapproach the softmax loss is replaced by the 200 binary hingelosses when fine-tuning the deep model Thus the deep modelfine-tuning and SVM learning steps in RCNN are mergedinto one step The extra training time required for extractingfeatures (sim 24 days with one Titan GPU) is saved

36 Contextual modeling

The deep model learned for the image classification task (Fig3 (c)) takes scene information into consideration while thedeep model for object detection (Fig 3 (a) and (b)) focuseson local bounding boxes The 1000-class image classificationscores are used as contextual features and concatenated withthe 200-class object detection scores to form a 1200 dimen-sional feature vector based on which a linear SVM is learnedto refine the 200-class detection scores For a specific objectclass not all object classes from the image classification modelare useful We learn a two-stage SVM to remove most of theclasses In the first stage all scores from the 1000 classesare used for learning a linear SVM At the second stagethe 10 classes with the largest magnitude in the linear SVM




Volleyball

Bathing cap

Golf ball

Seat belt

Figure 9 The SVM weights on image classification scores (a)for the object detection class volleyball (b)

weights learned in the first stage are selected as features andthen a linear SVM is learned for a given object class to bedetected Therefore only the classification scores of 10 classesfrom the image classification deep model are used for eachclass to be detected The SVM is explicitly trained but notwithin the network framework If 5 20 50 100 or 1000classes are used the mAP drops by 0 02 08 09 and45 respectively when compared with the result of using 10classes This result shows that only a few number of classesare helpful for detection The heuristic selection of 10 classeshelps to remove the effect from uncorrelated classes

Take object detection for class volleyball as an examplein Figure 9 If only considering local regions cropped frombounding boxes volleyballs are easy to be confused withbathing caps and golf balls In this case the contextual infor-mation from the whole-image classification scores is helpfulsince bathing caps appear in scenes of beach and swimmingpools golf balls appear in grass fields and volleyballs appearin stadiums The whole images of the three classes can bebetter distinguished because of the global scenery informationFig 9 plots the learned linear SVM weights on the 1000-classimage classification scores It is observed that image classesbathing cap and golf ball suppress the existence of volleyballin the refinement of detection scores with negative weightswhile the image class volleyball enhances the detection scoreof volleyball

37 Combining models with high diversity

Model averaging has been widely used in object detection Inexisting works [75] [25] [21] the same deep architecture wasused Models were different in cropping images at differentlocations or using different learned parameters In our modelaveraging scheme we learn models under multiple settingsThe settings of the models used for model averaging are shownin Table 4 They are different in net structures pretrainingschemes loss functions for the deep model training addingdef-pooling layer or not The motivation is that models gen-erated in this way have higher diversity and are complementaryto each other in improving the detection results after modelaveraging For example although model no 4 has low mAPit is found by greedy search because its pretraining schemeis different from other models and provides complementaryscores for the averaged scores

The 6 models as shown in Table 4 are automatically selectedby greedy search on ImageNet Det val2 from 10 models andthe mAP of model averaging is 503 on the test data ofILSVRC2014 while the mAP of the best single model is479

Person Animal Vehicle Furniture

Ours 586

5

3049

41

19

5224

716

36

2221

21

LocSimOthBG

G-Net 63 35

29

59 28

111

66135

1641

1516

28

Figure 10 Fraction of high-scored false positives on VOC-2007 that are due to poor localization (Loc) confusion withsimilar objects (Sim) confusion with other VOC objects (Oth)or confusion with background or unlabeled objects (BG)

4 EXPERIMENTAL RESULTS

Our experimental results are implemented based on the Caffe[24] Only selective search is used for proposing regions if notspecified

Overall result on PASCAL VOC For the VOC-2007 detec-tion dataset [11] we follow the approach in [16] for splittingthe training and testing data Table 1 shows the experimentalresults on VOC-2007 testing data which include approachesusing hand-crafted features [17] [48] [63] [62] [12] deepCNN features [16] [21] [15] [47] [46] [68] and CNNfeatures with deformation learning [18] Since all the state-of-the-art works reported single-model results on this datasetwe also report the single-model result only Our model waspretrained on bounding box annotation with deformationwithout context and with GoogLeNet as the baseline netOurs outperforms RCNN [16] and SPP [21] by about 5 inmAP RCNN SPN and our model are all pre-trained on theImageNet Cls-Loc training data and fine-tuned on the VOC-2007 training data Table 2 shows the per-class mAPs for ourapproach with G-Ntt and RCNN with VGG and GoogleNet[16] Fig 10 shows the analysis on false positives using theapproach in [23]

Overall result on MS-COCO Without using context oursingle model has mAP 256 on the MS-COCO Test-devdataset [28]

Experimental Setup on ImageNet The ImageNet LargeScale Visual Recognition Challenge (ILSVRC) 2014 [49]contains two different datasets 1) the classification and lo-calization (Cls-Loc) dataset and 2) the detection (Det) datasetThe training data of Cls-Loc contains 12 million images withlabels of 1 000 categories It is used to pretrain deep modelsThe same split of train and validation data from the Cls-Locis used for image-level annotation and object-level annotationpretraining The Det contains 200 object categories and issplit into three subsets train validation (val) and test dataWe follow RCNN [16] in splitting the val data into val1 andval2 Val1 is used to train models val2 is used to evaluateseparate components and test is used to evaluating the overallperformance The val1val2 split is the same as that in [16]

Overall result on ImageNet Det RCNN [16] is used asthe state-of-the-art for comparison The source code pro-vided by the authors was used and we were able to re-peat their results Table 3 summarizes the results fromILSVRC2014 object detection challenge It includes the best




Table 1Detection mAP () on VOC-2007 test All approaches are trained on VOC-2007 data Bounding box regression is used in DPM

SPP RCNN RCNN-V5 fRCN and our approach Only a single model is used for all approachesapproach DPM HSC-DPM Regionlet Flair DP-DPM SPP RCNN RCNN-v5 fRCN RPN YOLO Superpixel Label ours

[17] [48] [63] [62] [18] [21] [16] [16] [15] [47] [46] [68]337 343 417 333 452 585 631 660 669 676 591 614 690

Table 2VOC-2007 test detection average precision () for RCNN using VGG and our approach

aero bike bird boat bottle bus car cat chair cow table dog horse mbike person plant sheep sofa train tv mAPRCNN+VGG 734 770 634 454 446 751 781 798 405 737 622 794 781 731 642 356 668 672 704 711 660RCNN+G-Net 737 724 654 472 446 712 774 742 426 711 575 722 727 749 625 378 679 664 653 709 644Ours+G-Net 771 768 756 545 519 761 795 777 480 782 611 821 781 761 659 354 753 672 717 719 690

Table 3Detection mAP () on ILSVRC2014 for top ranked approaches with single model (sgl) and average model (avg)

approach Flair RCNN Berkeley Vision UvA-Euvision DeepInsight GoogLeNet Superpixel Label ours[62] [16] [16] [55] [67] [60] [68]

ImageNet val2 (avg) na na na na 42 445 454 507ImageNet val2 (sgl) na 310 334 na 401 388 428 482ImageNet test (avg) 226 na na na 405 439 450 503ImageNet test (sgl) na 314 345 354 402 380 425 479

Table 4Models used for model averaging The result of mAP is on val 2without bounding box regression and context For net design

D-Def(O) denotes our DeepID-Net that uses def-pooling layersusing Overfeat as baseline structure D-Def(G) denotes

DeepID-Net that uses def-pooling layers using GoogLeNet asbaseline structure G-net denotes GoogLeNet For pretraining0 denotes the pretraining scheme of RCNN [16] 1 denotes theScheme 1 in Section 33 For loss of net h denotes hinge losss denotes softmax loss Bounding box rejection is used for all

models Selective search and edgeboxes are used forproposing regions

model number 1 2 3 4 5 6net design D-Def(O) D-Def(G) G-net G-net D-Def(G) D-Def(G)Pretrain 1 1 1 0 1 1

loss of net h h s s h hmAP () 433 473 455 421 473 449

results on the test data submitted to ILSVRC2014 fromGoogLeNet [60] DeepInsight[67] UvA-Euvision[55] Berke-ley Vision[16] which ranked top among all the teams par-ticipating in the challenge In terms of single-model andmodel averaging performance we achieve the highest mAPIt outperforms the winner of ILSVRC2014 GoogleNet by61 on mAP Table 4 shows the 6 models we used in modelaveraging

41 Ablation study

The ImageNet Det is used for ablation study Bounding boxregression is not used if not specified

411 Baseline deep model and bounding box rejection

As shown in Fig 3 a baseline deep model is used in ourDeepID-Net Table 5 shows the results for different baselinedeep models and bounding box rejection choices AlexNet in[25] is denoted as A-net ZF in [75] is denoted as Z-net andOverfeat in [52] is denoted as O-net GoogLeNet in [60] is

denoted as G-net Except for the two components investigatedin Table 5 other components are the same as RCNN while thenew training schemes and the new components introduced inSection 32 are not included The configuration in the secondcolumn of Table 5 is the same as RCNN (mean mAP 299)Based on RCNN applying bounding box rejection improvesmAP by 1 Therefore bounding box rejection not only savesthe time for training and validating new models which iscritical for future research but also improves detection accu-racy Bounding box rejection is not constrained to particulardetectors such as RCNN or fast RCNN The time requiredto process one image is around 35 seconds per image usingRCNN and around 02 seconds using fast RCNN Both withbounding box rejection ZF [75] performs better than AlexNet[25] with 09 mAP improvement Overfeat [52] performsbetter than ZF with 48 mAP improvement GoogLeNet [60]performs better than Overfeat with 12 mAP improvement

Experimental results in Table 5 show the further investi-gation on the influence of bounding box rejection schemein training and testing stage Experimental results on twodifferent CNN architectures ie A-net and Z-net show thatthe mAP is similar whether the rejection scheme in the testingstage is used or not And the rejection scheme in the trainingstage is the main factor in improving the mAP If there isconcern that the rejection scheme results in lower recall of thecandidate windows at the testing stage the rejection schemeat the testing stage can be skipped If not specified boundingbox rejection is used in both training and testing stages

412 Investigation on the number of object classes atthe pretraining stage

In order to investigate the influence from the number of objectclasses at the pretraining stage we use the AlextNet andtrain on the ImageNet classification data without using thebounding box labels Table 7 shows the experimental resultsAs pointed out in [50] the 200 classes in ImageNet detec-tion corresponds to 494 classes in ImageNet classification




Table 5Study of bounding box (bbox) rejection and baseline deep

model on ILSVRC2014 val2 Pretrained without bounding boxlabels Def-pooling context and bounding box regression are

not usedbbox rejection n y y y y

deep model A-net A-net Z-net O-net G-netmAP () 299 309 318 366 378

meadian AP () 289 294 305 367 37

Table 6Study of bounding box (bbox) rejection at the training and

testing stage without context or def-pooling Pretrained withoutbounding box labels Def-pooling context and bounding box

regression are not usedbbox rejection train n y y y ybbox rejection test n y n y n

deep model A-net A-net A-net Z-net Z-netmAP () 299 309 308 318 315

meadian AP () 289 294 293 305 304

Therefore we investigate three pretraining settings 1) use thecorresponding 494-class samples in ImageNet classificationtraining data but train the deep model as a 200-class classifi-cation problem 2) use the corresponding 494-class samples inImageNet classification training data and train the deep modelas a 494-class classification problem 3) use the 1000-classsamples in ImageNet classification training data and train thedeep model as a 1000-class classification problem The samefine-tuning configuration is used for these three pretrainingsettings Experimental results show that 494-class pretrainingperforms better than 200-class pretraining by 3 mAP 1000-class pretraining performs better than 494-class pretraining by43 mAP 3000-class pretraining further improves the mAPby 24 compared with 1000-class pretraining For 3000-classpretraining each sample carries much more information foran apple image the 3000-class pretraining provides furtherinformation that it is not the other 2999 classes And the useof more classes makes the training task challenging and noteasy to overfit

413 Investigation on def-pooling layerDifferent deep model structures are investigated and results areshown in Table 8 using the new pretraining scheme in Section33 Our DeepID-Net that uses def-pooling layers as shownin Fig 3 is denoted as D-Def Using the Z-net as baselinedeep model the DeepID-Net that uses def-pooling layer inFig 3 improves mAP by 25 Def-pooling layer improvesmAP by 23 for both O-net and G-net This experimentshows the effectiveness of the def-pooling layer for genericobject detection In our implementation of def-pooling for G-

Table 7Study of number of classes used for pretraining AlexNet isused Pretrained without bounding box labels Def-pooling

context and bounding box regression are not usednumber of classes 200 494 1000 3000

mAP () 226 256 299 323meadian AP () 198 230 289 317

Table 8Investigation on def-pooling for different baseline net structures

on ILSVRC2014 val2 Use pretraining scheme 1 but nobounding box regression or context

net structure Z-net D-Def(Z) O-net D-Def(O) G-net D-Def(G)mAP () 360 385 391 414 404 427

meadian () 349 374 379 419 393 423

net we only replace max-pooling by def-pooling but did notadd an additional feature maps like that in Fig 3(b) 23mAP improvement is still observed on G-net by replacing themax-pooling with def-pooling

414 Investigation on different pretraining schemes andbaseline net structures

There are two different annotation levels image and objectTable 9 shows the results for investigation on annotation levelsand net structures When producing these results other newcomponents introduced in Section 34-36 are not includedFor pretraining we drop the learning rate by 10 when theclassification accuracy of validation data reaches plateau untilno improvement is found on the validation data For fine-tuning we use the same initial learning rate (0001) and thesame number of iterations (20000) for dropping the learningrate by 10 for all net structures which is the same setting inRCNN [16]

Pretraining on object-level-annotation performs better thanpretraining on image-level annotation by 44 mAP for A-netand 42 for Z-net This experiment shows that object-levelannotation is better than image-level annotation in pretrainingdeep model

415 Investigation on the overall pipeline

Table 10 summarizes how performance gets improved byadding each component step-by-step into our pipeline RCNNhas mAP 299 With bounding box rejection mAP is im-proved by about 1 denoted by +1 in Table 10 Basedon that changing A-net to Z-net improves mAP by 09Changing Z-net to O-net improves mAP by 48 O-net to G-net improves mAP by 12 Replacing image-level annotationby object-level annotation in pretraining mAP is increasedby 26 By combining candidates from selective search andedgeboxes [84] mAP is increased by 23 The def-poolinglayer further improves mAP by 22 Pretraining the object-level annotation with multiple scales [4] improves mAP by22 After adding the contextual information from imageclassification scores mAP is increased by 05 Bounding boxregression improves mAP by 04 With model averaging thefinal result is 507

42 Per-Class Accuracy as a Function of ObjectProperties on ILSVRC14 Object Detection Data

Inspired by the analysis in [50] we perform analysis on theobject properties that influence the variation in object detectionaccuracy for different classes in this section Our result with507 mAP on the val2 data is used for analysis




XS S M L XL

04

05

06

07

08

Rigid Deformable02

04

06

08

1

None low MedianHigh02

04

06

08

1

Figure 11 The mean average precision of our best-performingmodel in the y axis as a function of real-word size (left) deforma-bility (middle) and texture (right) in the x axis

009 016 024 035 07802

04

06

08

1

013 029 039 04602

04

06

08

1

039 052 063 07802

04

06

08

1

Figure 12 The mean average precision of our best-performingmodel in the y axis as a function of the variance in aspectratio (left) part existence (middle) and in-plane out-plane ro-tation(right) The x axis denotes the average variance of eachgroup

For real-world size deformability and amount of texturethe following conclusion on the detection accuracy as afunction of these object properties can be drawn from theexperimental results in Fig 11

Real-world size XS for extra small (eg nail) small (egfox) medium (eg bookshelf) large (eg car) or XL for extralarge (eg airplane) The object detection model performssimilar on extra small small or medium ones which isdifferent from the optimistic model in [50] It performs betteron extra large and large objects with extra large objects havingthe highest mean AP (close to 70)

Deformability within instance Rigid (eg mug) or de-formable (eg snake) Similar to [50] we also find thatdeformable objects have higher accuracy than rigid objects

Amount of texture none (eg punching bag) low (eghorse) medium (eg sheep) or high (eg honeycomb) Themodel is better on objects with at least median level of texturecompared to untextured or low textured objects

The three properties above are investigated in [50] usingoptimistic model ie directly compare all the entries in thepast 3 years to obtain the most optimistic measurement ofstate-of-the-art accuracy on each category Using our bestperforming model similar conclusion can be drawn

In the following we investigate new properties that wefound influential to object detection accuracy Object classesare sorted in ascending order using these properties and thenuniformly grouped ie all groups have the number of classes

Variance in aspect ratio Aspect ratio is measured by thewidth of the object bounding box divided by the height of thebounding box Objects with large variance in aspect ratio egband aid and nail are more likely to be slim and have largevariation in rotation which result in the drastic appearancechange of the visual information within the object boundingbox Therefore as shown in Fig 12 objects with lowervariance in aspect ratio performs better

Variance in part existence Many objects have some oftheir parts not existing in the bounding box because ofocclusion or tight-shot The variation in part existence causes

the appearance variation for object of the same class Forexample a backpack with only its belt in the bounding boxis very different in appearance from a backpack with its bagin the bounding box We labeled the object parts and theirexistences for all the 200 classes on the val1 data and usethem for obtaining the variance in part existence As shown inFig 12 objects with lower variance in part existence performsbetter

Variance in rotation In-plane and out-of-plane rotation arefactors that influence the within-class appearance variation Anax with frontal view is very different in appearance from an axwith side view An upright ax is very different in appearancefrom a horizontal ax We labeled the rotation of objects for allthe 200 classes on the val1 data and use them for obtaining thevariance in rotation As shown in Fig 12 objects with lowervariance in rotation performs better

Number of objects per image The number of object perimage for the cth object class denoted by Nc is obtained asfollows

Nc =1

Pc

Pcsum

pc=1

(npc ) (8)

where npc is the number of objects within the image of thepcth sample for class c pc = 1 Pc Nc is obtained fromthe val1 data When there are large number objects within animage they may occlude each other and appear as backgroundfor the ground truth bounding box of each other resulting inthe added complexity of object appearance and backgroundclutter As shown in Fig 13 some small objects bee andbutterfly have less than 2 objects per image on average Andthey have very high AP 906 for butterfly and 769 forbee We find that the images in val1 with these samples aremostly captured by tight shot and they have relatively simplebackground As shown in Fig 13 the model performs betterwhen the number of objects per image is smaller

0 50 100 150 2000

5

10

15

208 413 552 709 97702

04

06

08

1

Figure 13 Number of objects per image for different classes(left) and the detection accuracy as a function of the averagenumber of objects per image

Mean area per bounding box We measure the size of thebounding box by the area (width multiplied by height) of thisbounding box We did not use the bounding box size overimage size in [50] because the image size may have influenceon image classification and object localization but should havesmall influence on object detection in which bounding boxis independently evaluated As shown in Fig 14 the averagearea for different objects varies a lot Sofa has the largest meanarea Although butterfly and bee are extra small in real-worldsize they are large in average areas 368k for bee and 577kfor butterfly As shown in Fig 13 the average AP is higherwhen the mean area per bounding box is larger

Recall from region proposal In our model selective searchand edgeboxes are used for proposing the regions After




0 50 100 150 2000

5

10

15x 10

4

104 205 299 409 62702

04

06

08

1

Figure 14 The average size of the bounding box for differentclasses (left) and the detection accuracy as a function of theaverage number of objects per image

bounding box rejection 302 boxes per image are obtainedon val1 The average recall is 8919 for overlap greater than05 and 7898 for overlap greater than 07

Fig 15 shows the 5 classes with lowest and highest averageprecision and their corresponding factors The 5 object classeswith the lowest accuracy are mostly none-deformable havinglow texture small bounding box size large number of objectsper image large variation in aspect ratio part existence androtation

We also tried other properties like variation in boundingbox size average aspect ratio number of positive samples butdid not find them to have strong correlation to the detectionaccuracy

Fig 16 shows the object classes with large mAP improve-ment and mAP drop when the def-pooling is used 140 ofthe 200 classes have their mAP improved Def-pooling bringslarge mAP gains for mammals like squirrel with deformationand instruments like banjo with rotation However man-madeobjects such as waffle iron digital clock cocktail shakerand vacuum have inconsistent existence of object parts largevariation in rotation and part appearance Therefore the mAPgains are negative for these man-made objects

Fig 17 shows the detection accuracy for object classesgrouped at different WordNet hierarchical levels It can be seenthat vertebrates that are neither mammal nor fish ie bird froglizard snake and turtle have the largest mAP Mammals alsohave large mAP because mammals share similar appearancehave rich texture and have many object classes that helpeach other in learning their feature representations Generallyartifacts have lower mAP because they have low texture andlarge variation in shape Texts in dashed boxes of Fig17 show the absolute mAP gain obtained by bounding boxrejection and def-pooling for each group It can be seen thatdef-pooling has 95 mAP gain for detecting person Def-pooling has higher mAP gain (46) for mammals with regulardeformation and part appearance than substances (04) withirregular deformation and part appearance

5 APPENDIX A RELATIONSHIP BETWEEN THEDEFORMATION LAYER AND THE DPMThe quadratic deformation constraint in [12] can be repre-sented as follows

m(ij)=m(ij) minus a1(iminusb1+a3

2a1)2minusa2(jminusb2+

a4

2a2)2 (9)

where m(ij) is the (i j)th element of the part detectionmap M (b1 b2) is the predefined anchor location of the pthpart They are adjusted by a32a1 and a42a2 which are

Deformable Real size Texture Box size Obj Num A ratio varPart ext var Rot var0

1

2

3

4

5

Fac

tor

grou

p

rub eraserskihorizontal barbackpackladle


1

2

3

4

5

Fac

tor

grou

p

foxarmadillodogbutterflybanjo

Figure 15 The factors for the 5 object classes with lowest AP(top) and highest AP (bottom) The y axis denotes the group in-dex g for the factors in Fig 11-14 eg deformation (g = 0 1)real-world size (g = 1 5) texture (g = 1 4) boxsize (g = 1 5) Larger g denotes higher deformationreal size texture etc The x axis corresponds to different objectclasses eg ski ladle with different factors eg deformationreal size texture Legends denote different object classes

Deformation Real size Texture Box size Obj Num A R var Part ext var Rot var0

1

2

3

4

5

Fact

or G

roup


1

2

3

4

5

Fact

or G

roup

vacuum minus4waffle iron minus6cocktail shaker minus6digital clock minus6

squirrel +21cattle +17banjo +16skunk +14

Figure 16 The factors for the object classes with mAPimprovement (top) and mAP drop (bottom) introduced by thedef-pooling The meaning of x and y axes are the same as 15Legends denote different object classes and their mAP changecaused by def-pooling

automatically learned a1 and a2 (9) decide the deformationcost There is no deformation cost if a1 = a2 = 0 Parts arenot allowed to move if a1 = a2 = infin (b1 b2) and ( a3

2a1 a4

2a2)

jointly decide the center of the part The quadratic constraintin Eq (9) can be represented using Eq (1) as follows

m(ij)=m(ij) minus a1d(ij)1 minus a2d

(ij)2 minus a3d

(ij)3 minus a4d

(ij)4 minusa5

d(ij)1 =(iminus b1)

2 d(ij)2 =(j minus b2)

2 d(ij)3 = iminus b1

d(ij)4 =j minus b2 a5 = a3

2(4a1) + a42(4a2) (10)

In this case a1 a2 a3 and a4 are parameters to be learnedand d

(ij)n for n = 1 2 3 4 are predefined a5 is the same in

all locations and need not be learned The final output is

b = max(ij)

m(ij) (11)

where m(ij) is the (i j)th element of the matrix M in (9)

6 CONCLUSION

This paper proposes a deep learning based object detectionpipeline which integrates the key components of bound-ing box reject pretraining deformation handling contextmodeling bounding box regression and model averaging Itsignificantly advances the state-of-the-art from mAP 310




H18 physical19910

22

H13 matter

H18

H1522

entity507

1304

H11

H10 solid510

497

13instrument

artifact

45 8

454107

128

588causalagent

1

2608

0617

0495

07155

H09 substance solid food

478

510

510

9 13device

465

45840

iperson

588

588 g

1 1538

0495

07155

1304

abstraction(traffic light)

532

1

animals

48

65 6

12114

1822

otherartifacts439

212

vertebrate

aquaticvertebrate

56 4

674

33

othervertebrates74 4

5

otheranimals

617

15656

06238

1319

communication(traffic light)

532

1

othernstruments453

67mammal fish

661 564

56426 2

74412114

1319

0746

Figure 17 The detection accuracy for object classes groupedat different WordNet hierarchical levels Tilted text at the upperright of the circle denotes the number of classes within the 200object classes of the ILSVRC14 detection task for this WordNetsynonym set (synset) Texts at the upper left of the circle denotethe absolute mAP gain obtained by bounding box rejection anddef-pooling Un-tilted text below the circle denote the mAP inpercentage for this WordNet synset For example the WordNetsynset lsquomatterrsquo has height 13 22 object classes and mAP497 bounding box rejection has mAP gain of 134 andthe def-pooling has mAP gain of 04 The lsquoother vertebratesrsquodenotes the object classes that are vertebrates but not mammalor aquatic vertebrate similarly for lsquoother artifactsrsquo and lsquootherinstrumentsrsquo

(obtained by RCNN) to 503 on the ImageNet object taskIts single model and model averaging performances are thebest in ILSVC2014 A global view and detailed component-wise experimental analysis under the same setting are providedto help researchers understand the pipeline of deep learningbased object detection

We enrich the deep model by introducing the def-poolinglayer which has great flexibility to incorporate various defor-mation handling approaches and deep architectures Motivatedby our insights on how to learn feature representations moresuitable for the object detection task and with good generaliza-tion capability a pretraining scheme is proposed By changingthe configurations of the proposed detection pipeline multipledetectors with large diversity are obtained which leads to moreeffective model averaging This work shows the importantmodules in an object detection pipeline although each hasits own parameter setting set in an ad hoc way In the futurewe will design an end-to-end system that jointly learns thesemodules

Acknowledgment This work is supported by the GeneralResearch Fund sponsored by the Research Grants Council ofHong Kong (Project Nos CUHK14206114 CUHK14205615and CUHK14207814 CUHK417011CUHK14203015)

REFERENCES

[1] Deepdraw DeepDraw on githubcomaudunodeepdraw 6[2] O Barinova V Lempitsky and P Kohli On detection of multiple object

instances using hough transforms In CVPR 2010 3[3] L Bourdev and J Malik Poselets body part detectors trained using 3D

human pose annotations In ICCV 2009 2

[4] K Chatfield K Simonyan A Vedaldi and A Zisserman Return ofthe devil in the details Delving deep into convolutional nets In BMVC2014 2 9

[5] N Dalal and B Triggs Histograms of oriented gradients for humandetection In CVPR 2005 3

[6] C Desai and D Ramanan Detecting actions poses and objects withrelational phraselets In ECCV 2012 3

[7] C Desai D Ramanan and C Fowlkes Discriminative models formulti-class object layout In ICCV 2009 3

[8] Y Ding and J Xiao Contextual boost for pedestrian detection InCVPR 2012 3

[9] S K Divvala D Hoiem J H Hays A A Efros and M Hebert Anempirical study of context in object detection In CVPR 2009 3

[10] J Donahue Y Jia O Vinyals J Hoffman N Zhang E Tzeng andT Darrell Decaf A deep convolutional activation feature for genericvisual recognition In ICML 2014 1

[11] M Everingham L V Gool C K IWilliams JWinn and A ZissermanThe pascal visual object classes (voc) challenge IJCV 88(2)303ndash3382010 1 7

[12] P Felzenszwalb R B Grishick DMcAllister and D Ramanan Objectdetection with discriminatively trained part based models IEEE TransPAMI 321627ndash1645 2010 1 2 4 5 7 11

[13] P F Felzenszwalb and D P Huttenlocher Pictorial structures for objectrecognition IJCV 6155ndash79 2005 1 2

[14] C Galleguillos B McFee S Belongie and G Lanckriet Multi-classobject localization by combining local contextual interactions In CVPR2010 3

[15] R Girshick Fast r-cnn ICCV 2015 3 7 8[16] R Girshick J Donahue T Darrell and J Malik Rich feature

hierarchies for accurate object detection and semantic segmentation InCVPR 2014 1 2 3 4 7 8 9

[17] R Girshick P Felzenszwalb and D McAllester Dis-criminatively trained deformable part models release 5httpwwwcsberkeleyedurbglatent-v5 7 8

[18] R Girshick F Iandola T Darrell and J Malik Deformable part modelsare convolutional neural networks arXiv preprint arXiv140954032014 5 6 7 8

[19] Y Gong L Wang R Guo and S Lazebnik Multi-scale orderlesspooling of deep convolutional activation features arXiv preprintarXiv14031840 2014 3

[20] B Hariharan C L Zitnick and P Dollar Detecting objects usingdeformation dictionaries In CVPR 2014 2

[21] K He X Zhang S Ren and J Sun Spatial pyramid pooling in deepconvolutional networks for visual recognition In ECCV 2014 1 3 78

[22] G Heitz and D Koller Learning spatial context Using stuff to findthings In ECCV 2008 3

[23] D Hoiem Y Chodpathumwan and Q Dai Diagnosing error in objectdetectors In Computer VisionndashECCV 2012 pages 340ndash353 Springer2012 7

[24] Y Jia E Shelhamer J Donahue S Karayev J Long R GirshickS Guadarrama and T Darrell Caffe Convolutional architecture forfast feature embedding arXiv preprint arXiv14085093 2014 7

[25] A Krizhevsky I Sutskever and G Hinton Imagenet classification withdeep convolutional neural networks In NIPS 2012 2 3 7 8

[26] K Lenc and A Vedaldi R-cnn minus r arXiv preprintarXiv150606981 2015 3

[27] M Lin Q Chen and S Yan Network in network arXiv preprintarXiv13124400 2013 1 3

[28] T-Y Lin M Maire S Belongie J Hays P Perona D RamananP Dollar and C L Zitnick Microsoft coco Common objects in contextIn ECCV pages 740ndash755 2014 7

[29] P Luo Y Tian X Wang and X Tang Switchable deep network forpedestrian detection In CVPR 2014 3

[30] P Luo X Wang and X Tang Hierarchical face parsing via deeplearning In CVPR 2012 3

[31] P Luo X Wang and X Tang A deep sum-product architecture forrobust facial attributes analysis In ICCV 2013 3

[32] P Luo X Wang and X Tang Pedestrian parsing via deep decompo-sitional neural network In ICCV 2013 3

[33] W Ouyang X Chu and X Wang Multi-source deep learning forhuman pose estimation In CVPR 2014 3

[34] W Ouyang H Li X Zeng and X Wang Learning deep representationwith large-scale attributes In ICCV 2015 3

[35] W Ouyang and X Wang A discriminative deep model for pedestriandetection with occlusion handling In CVPR 2012 3

[36] W Ouyang and X Wang Joint deep learning for pedestrian detectionIn ICCV 2013 3 5 6

[37] W Ouyang and X Wang Single-pedestrian detection aided by multi-pedestrian detection In CVPR 2013 1




Table 9Ablation study of the two pretraining schemes in Section 33 for different net structures on ILSVRC2014 val 2 Scheme 0 only usesimage-level annotation for pretraining Scheme 1 uses object-level annotation for pretraining Def-pooling bounding box regression

and context are not usednet structure A-net A-net A-net Z-net Z-net Z-net Z-net O-net O-net G-net G-netclass number 1000 1000 1000 1000 200 1000 1000 1000 1000 1000 1000bbox rejection n n y y n n y y y y y

pretrain scheme 0 1 1 0 1 1 1 0 1 0 1mAP () 299 343 349 318 299 356 360 366 391 378 404

meadian AP () 289 334 344 305 297 340 349 367 379 370 393

Table 10Ablation study of the overall pipeline for single model on ILSVRC2014 val2 It shows the mean AP after adding each key

component step-by-stepdetection pipeline RCNN +bbox A-net Z-net O-net image to bbox +edgbox +Def +multi-scale +context +bbox

rejection to Z-net to O-net to G-net pretrain candidate pooling pretrain regressionmAP () 299 309 318 366 378 404 427 449 473 478 482

meadian AP () 289 294 305 367 370 393 423 452 478 481 498mAP improvement () +1 +09 +48 +12 +26 +23 +22 +24 +05 +04

[38] W Ouyang X Wang X Zeng S Qiu P Luo Y Tian H Li S YangZ Wang C-C Loy et al Deepid-net Deformable deep convolutionalneural networks for object detection In CVPR 2015 2

[39] W Ouyang X Wang C Zhang and X Yang Factors in finetuningdeep model for object detection In CVPR 2016 3

[40] W Ouyang X Zeng and X Wang Modeling mutual visibilityrelationship in pedestrian detection In CVPR 2013 3

[41] W Ouyang X Zeng and X Wang Single-pedestrian detection aidedby 2-pedestrian detection IEEE Trans PAMI 2015 2

[42] W Ouyang X Zeng and X Wang Learning mutual visibilityrelationship for pedestrian detection with a deep model IJCV pages1ndash14 2016 3

[43] W Ouyang X Zeng and X Wang Partial occlusion handling inpedestrian detection with a deep model IEEE Trans Circuits SystVideo Technol 2016 3

[44] D Park D Ramanan and C Fowlkes Multiresolution models for objectdetection In ECCV 2010 3

[45] A S Razavian H Azizpour J Sullivan and S Carlsson Cnn featuresoff-the-shelf an astounding baseline for recognition arXiv preprintarXiv14036382 2014 1 3

[46] J Redmon S Divvala R Girshick and A Farhadi You only look onceUnified real-time object detection arXiv preprint arXiv1506026402015 3 7 8

[47] S Ren K He R Girshick and J Sun Faster r-cnn Towards real-timeobject detection with region proposal networks NIPS 2015 3 7 8

[48] X Ren and D Ramanan Histograms of sparse codes for objectdetection In CVPR 2013 7 8

[49] O Russakovsky J Deng H Su J Krause S Satheesh S MaZ Huang A Karpathy A Khosla M Bernstein A C Berg and L Fei-Fei Imagenet large scale visual recognition challenge 2014 7

[50] O Russakovsky J Deng H Su J Krause S Satheesh S MaZ Huang A Karpathy A Khosla M Bernstein A C Berg and L Fei-Fei Imagenet large scale visual recognition challenge IJCV 2015 18 9 10

[51] M A Sadeghi and A Farhadi Recognition using visual phrases InCVPR 2011 3

[52] P Sermanet D Eigen X Zhang M Mathieu R Fergus and Y Le-Cun Overfeat Integrated recognition localization and detection usingconvolutional networks arXiv preprint arXiv13126229 2013 2 3 8

[53] K Simonyan A Vedaldi and A Zisserman Deep inside convolutionalnetworks Visualising image classification models and saliency maps InICLR 2014 3

[54] K Simonyan and A Zisserman Very deep convolutional networks forlarge-scale image recognition arXiv preprint arXiv14091556 2014 1

[55] A Smeulders T Gevers N Sebe and C Snoek Segmentation asselective search for object recognition In ICCV 2011 3 8

[56] Z Song Q Chen Z Huang Y Hua and S Yan Contextualizing objectdetection and classification In CVPR 2011 3

[57] Y Sun X Wang and X Tang Deep convolutional network cascadefor facial point detection In CVPR 2013 3

[58] Y Sun X Wang and X Tang Hybrid deep learning for computingface similarities In ICCV 2013 3

[59] Y Sun X Wang and X Tang Deep learning face representation from

predicting 10000 classes In CVPR 2014 3[60] C Szegedy W Liu Y Jia P Sermanet S Reed D Anguelov D Erhan

V Vanhoucke and A Rabinovich Going deeper with convolutionsarXiv preprint arXiv14094842 2014 1 2 4 8

[61] S Tang M Andriluka A Milan K Schindler S Roth and B SchieleLearning people detectors for tracking in crowded scenes ICCV 20133

[62] K E A van de Sande C G M Snoek and A W M SmeuldersFisher and vlad with flair In CVPR 2014 7 8

[63] X Wang M Yang S Zhu and Y Lin Regionlets for generic objectdetection In ICCV 2013 7 8

[64] B Wu and R Nevatia Detection and tracking of multiple partially oc-cluded humans by bayesian combination of edgelet based part detectorsIJCV 75(2)247ndash266 2007 3

[65] J Yan Z Lei L Wen and S Z Li The fastest deformable part modelfor object detection In CVPR 2014 2

[66] J Yan Z Lei D Yi and S Z Li Multi-pedestrian detection in crowdedscenes A global view In CVPR 2012 3

[67] J Yan N Wang Y Yu S Li and D-Y Yeung Deeper vision anddeep insight solutions In ECCV workshop on ILSVRC2014 2014 8

[68] J Yan Y Yu X Zhu Z Lei and S Z Li Object detection by labelingsuperpixels In CVPR 2015 3 7 8

[69] B Yang J Yan Z Lei and S Z Li Convolutional channel featuresIn ICCV pages 82ndash90 2015 3

[70] W Yang W Ouyang H Li and X Wang End-to-end learning ofdeformable mixture of parts and deep convolutional neural networks forhuman pose estimation CVPR 2016 2

[71] Y Yang S Baker A Kannan and D Ramanan Recognizing proxemicsin personal photos In CVPR 2012 3

[72] Y Yang and D Ramanan Articulated pose estimation with flexiblemixtures-of-parts In CVPR 2011 2

[73] Y Yang and D Ramanan Articulated human detection with flexiblemixtures of parts IEEE Trans PAMI 35(12)2878ndash2890 2013 1

[74] B Yao and L Fei-Fei Modeling mutual context of object and humanpose in human-object interaction activities In CVPR 2010 3

[75] M D Zeiler and R Fergus Visualizing and understanding convolutionalneural networks arXiv preprint arXiv13112901 2013 2 3 4 7 8

[76] X Zeng W Ouyang M Wang and X Wang Deep learning of scene-specific classifier for pedestrian detection In ECCV 2014 3

[77] X Zeng W Ouyang and X Wang Multi-stage contextual deep learningfor pedestrian detection In ICCV 2013 3

[78] N Zhang J Donahue R Girshick and T Darrell Part-based r-cnnsfor fine-grained category detection In ECCV 2014 1 4

[79] Y Zhang K Sohn R Villegas G Pan and H Lee Improving objectdetection with deep convolutional networks via bayesian optimizationand structured prediction In CVPR 2015 3

[80] R Zhao W Ouyang H Li and X Wang Saliency detection by multi-context deep learning In CVPR 2015 3

[81] B Zhou A Khosla A Lapedriza A Oliva and A Torralba Objectdetectors emerge in deep scene cnns arXiv14126856 abs141268562014 1

[82] B Zhou A Lapedriza J Xiao A Torralba and A Oliva Learningdeep features for scene recognition using places database In NIPS




2014 1[83] L Zhu Y Chen A Yuille and W Freeman Latent hierarchical

structural learning for object detection In CVPR 2010 1 2[84] C L Zitnick and P Dollar Edge boxes Locating object proposals from

edges In ECCV 2014 3 9[85] W Y Zou X Wang M Sun and Y Lin Generic object detection with

dense neural patterns and regionlets BMVC 2014 3

Wanli Ouyang received the PhD degree in theDepartment of Electronic Engineering The Chi-nese University of Hong Kong where he is nowa Research Assistant Professor His researchinterests include image processing computervision and pattern recognition

Xingyu Zeng received the BS degree in Elec-tronic Engineering and Information Science fromUniversity of Science and Technology of Chinain 2011 He is currently a PHD student in theDepartment of Electronic Engineering at the Chi-nese University of Hong Kong His research in-terests include computer vision and deep learn-ing

Xiaogang Wang received the PhD degree fromthe Computer Science and Artificial IntelligenceLaboratory at the Massachusetts Institute ofTechnology in 2009 He is currently an Assis-tant Professor in the Department of ElectronicEngineering at The Chinese University of HongKong His research interests include computervision and machine learning

Shi Qiu received the BS and PhD degree fromTsinghua University (2009) and the Chinese Uni-versity of Hong Kong (2014) respectively Heis currently a research scientist at SenseTimeGroup Limited His research interests includeobject recognition object detection and imagesearch He worked as a postdoctoral fellow atthe Chinese University of Hong Kong from 2014to 2015

Ping Luo is a postdoctoral researcher at the De-partment of Information Engineering The Chi-nese University of Hong Kong where he re-ceived his PhD in 2014 His research inter-ests include deep learning computer visionand computer graphics focusing on face anal-ysis pedestrian analysis and large-scale objectrecognition and detection

Yonglong Tian is currently an MPhil studentat Department of Information Eingeering in TheChinese University of Hong Kong His researchinterests include computer vision and machinelearning and object detection

Hongsheng Li received the masters and doc-torate degrees in computer science from LehighUniversity Pennsylvania in 2010 and 2012 re-spectively He is an associate professor in theSchool of Electronic Engineering at Universityof Electronic Science and Technology of ChinaHis research interests include computer visionmedical image analysis and machine learning

Shuo Yang received the BE degree in SoftwareEngineering from Wuhan University China in2013 He is currently working toward the PhDdegree in the Department of Information Engi-neering at the Chinese University of Hong KongHis research interests include computer visionand machine learning in particular face detec-tion and object recognition

Zhe Wang received the BEng in Optical En-gineering from Zhejiang University China in2012 He is currently working toward a PhDdegree in the Department of Electronic Engi-neering the Chinese University of Hong KongHis research interest is focused on compressedsensing magnetic resonance imaging imagesegmentation and object detection

Hongyang Li received his BE degree in ma-jor Electronic and Information Engineering atDalian University of Technology (DUT) 2014From 2013 to 2014 he was a research assistantin the computer vision group at DUT He iscurrently a PhD candidate under supervision ofProf Xiaogang Wang at The Chinese Universityof Hong Kong His research interests focus ondeep learning and object recognition

Junjie Yan received his PhD degree in 2015from National Laboratory of Pattern RecognitionChinese Academy of Sciences Since 2016 heis the principle scientist in SenseTime GroupLimited and leads the research in object de-tection and tracking and applications in videosurveillance

Kun Wang received the BS degree in SoftwareEngineering from Beijing Jiaotong University in2014 He is currently a research assistant inthe Department of Electronic Engineering at theChinese University of Hong Kong His researchinterests include crowd analysis and object de-tection with deep learning

Chen Change Loy received the PhD degree inComputer Science from the Queen Mary Uni-versity of London in 2010 He is currently aResearch Assistant Professor in the Departmentof Information Engineering Chinese Universityof Hong Kong Previously he was a postdoc-toral researcher at Vision Semantics Ltd Hisresearch interests include computer vision andpattern recognition with focus on face analysisdeep learning and visual surveillance

Xiaoou Tang received the BS degree from theUniversity of Science and Technology of ChinaHefei in 1990 and the MS degree from theUniversity of Rochester Rochester NY in 1991He received the PhD degree from the Mas-sachusetts Institute of Technology Cambridgein 1996 He is a Professor in the Departmentof Information Engineering and Associate Dean(Research) of the Faculty of Engineering of theChinese University of Hong Kong He workedas the group manager of the Visual Comput-

ing Group at the Microsoft Research Asia from 2005 to 2008 Hisresearch interests include computer vision pattern recognition andvideo processing Dr Tang received the Best Paper Award at the IEEEConference on Computer Vision and Pattern Recognition (CVPR) 2009He is a program chair of the IEEE International Conference on ComputerVision (ICCV) 2009 and has served as an Associate Editor of IEEETransactions on Pattern Analysis and Machine Intelligence (PAMI) andInternational Journal of Computer Vision (IJCV) He is a Fellow of IEEE




Deep model Deformable pattern Examples

Pattern

Deformationpenalty

Pattern

Deformationpenalty

Image

Visualizedmodel

Pretrained on object-level annoation Pretrained on image-level annotation

(a)

(b)

Higher penalty

Figure 1 The motivation of this paper in new pretrainingscheme (a) and jointly learning feature representation anddeformable object parts shared by multiple object classes atdifferent semantic levels (b) In (a) a model pretrained on image-level annotation is more robust to size and location changewhile a model pretrained on object-level annotation is better inrepresenting objects with tight bounding boxes In (b) when ipodrotates its circular pattern moves horizontally at the bottom ofthe bounding box Therefore the circular patterns have smallerpenalty moving horizontally but higher penalty moving verticallyThe curvature part of the circular pattern are often at the bottomright positions of the circular pattern Magnitudes of deformationpenalty are normalized to make them comparable across thetwo examples in (a) for visualization Best viewed in color

the shared visual patterns and their deformation propertiesfor multiple object classes at different semantic levels andcomposition levels

The performance of deep learning systems depends signifi-cantly on implementation details [4] However an evaluationof the performance of the recent deep architectures on thecommon ground for large-scale object detection is missingAs a respect to the investigation on details in deep learning[4] [16] this paper compares the performance of recentdeep models including AlexNet [25] ZF [75] Overfeat [52]and GoogLeNet [60] under the same setting for differentpretraining-finetuning schemes We also provide experimentalanalysis on the properties that cause the accuracy variation indifferent object classes

In this paper we propose a deformable deep convolutionalneural network for object detection named as DeepID-NetIn DeepID-Net we jointly learn the feature representationand part deformation for a large number of object cate-gories We also investigate many aspects in effectively andefficiently training and aggregating the deep models includingbounding box rejection training schemes context modeling

and model averaging The proposed new framework signif-icantly advances the state-of-the-art for deep learning basedgeneric object detection such as the well known RCNN [16]framework This paper also provides detailed component-wiseexperimental results on how our approach can improve themean Averaged Precision (AP) obtained by RCNN [16] from310 to mean AP 503 step-by-step on the ImageNet LargeScale Visual Recognition Challenge 2014 (ILSVRC2014) ob-ject detection task

The contributions of this paper are as follows1) A new deep learning framework for object detection It

effectively integrates feature representation learning partdeformation learning context modeling model averagingand bounding box location refinement into the detec-tion system Detailed component-wise analysis is providedthrough extensive experimental evaluation This paper isalso the first to investigate the influence of CNN structuresfor the large-scale object detection task under the samesetting By changing the configuration of this frameworkmultiple detectors with large diversity are generated whichleads to more effective model averaging

2) A new scheme for pretraining the deep CNN modelWe propose to pretrain the deep model on the ImageNetimage classification and localization dataset with 1000-class object-level annotations instead of with image-levelannotations which are commonly used in existing deeplearning object detection [16] [60] Then the deep model isfine-tuned on the ImageNetPASCAL-VOC object detectiondataset with 20020 classes which are the target objectclasses in the two datasets

3) A new deformation constrained pooling (def-pooling) layerwhich enriches the deep model by learning the deformationof object parts at any information abstraction levels Thedef-pooling layer can be used for replacing the max-poolinglayer and learning the deformation properties of parts

4) Analysis on the object properties that influence the variationin object detection accuracy for different classesPreliminary version of this paper is published in [38] This

paper include more analysis on the proposed approach and addexperimental investigation on the properties that influence theaccuracy in detecting objects

The models pretrained by both image-level annotation andobject-level annotation for AlexNet [25] ZF [75] overfeat[52] and GoogLeNet [60] and the models after fine-tuningon ILSVRC2014 are provided online 1

2 RELATED WORK

Since many objects have non-rigid deformation the abilityto handle deformation improves detection performance De-formable part-based models were used in [12] [83] [41][70] [65] for handling translational movement of parts Tohandle more complex articulations size change and rotationof parts were modeled in [13] and mixture of part appearanceand articulation types were modeled in [3] [72] A dictionaryof shared deformable patterns was learned in [20] In theseapproaches features were manually designed

1 wwweecuhkeduhksimwlouyangprojectsImageNet








Selective search +

Edgeboxes

DeepID-Net(Pretrain

def-pooling)

Box rejection

Context modeling


Model averaging


person

horse

person

horse

person

horse

person

horse



3 METHOD

















128

128

128

128

conv61

conv63

conv71

conv73

def61

def63

Image

Candidate region



1000-class scores

200-class scores

























Extract HOG feature


layers


detector







layer


convolutional layer









where x = 1 W y = 1 H (2)



φ(δx δy) =

Nsum

n=1

acndδxδycn (5)









δxδycn


zδxδyc




















partb(xy)c

partacn= minusd

(ΔxΔy)cn


(6)





times Hsy






filter


Deformationpenalty

Output B

Maxpooling

Viewedas

blocks

Block-wiseaddition


sytimes W



343 Analysis






Pedicle Wheel Wheel
















Volleyball

Bathing cap

Golf ball

Seat belt








Ours 586

5

3049

41

19

5224

716

36

2221

21

LocSimOthBG

G-Net 63 35

29

59 28

111

66135

1641

1516

28













[17] [48] [63] [62] [18] [21] [16] [16] [15] [47] [46] [68]337 343 417 333 452 585 631 660 669 676 591 614 690













41 Ablation study















meadian AP () 289 294 305 367 37





meadian AP () 289 294 293 305 304





mAP () 226 256 299 323meadian AP () 198 230 289 317




meadian () 349 374 379 419 393 423












XS S M L XL

04

05

06

07

08

Rigid Deformable02

04

06

08

1


04

06

08

1


009 016 024 035 07802

04

06

08

1

013 029 039 04602

04

06

08

1

039 052 063 07802

04

06

08

1













Nc =1

Pc

Pcsum

pc=1

(npc ) (8)


0 50 100 150 2000

5

10

15

208 413 552 709 97702

04

06

08

1







0 50 100 150 2000

5

10

15x 10

4

104 205 299 409 62702

04

06

08

1










a4

2a2)2 (9)



1

2

3

4

5

Fac

tor

grou

p



1

2

3

4

5

Fac

tor

grou

p




1

2

3

4

5

Fact

or G

roup


1

2

3

4

5

Fact

or G

roup





2a1 a4

2a2)



(ij)2 minus a3d

(ij)3 minus a4d

(ij)4 minusa5

d(ij)1 =(iminus b1)




2(4a1) + a42(4a2) (10)




b = max(ij)

m(ij) (11)


6 CONCLUSION





H18 physical19910

22

H13 matter

H18

H1522

entity507

1304

H11

H10 solid510

497

13instrument

artifact

45 8

454107

128

588causalagent

1

2608

0617

0495

07155


478

510

510

9 13device

465

45840

iperson

588

588 g

1 1538

0495

07155

1304


532

1

animals

48

65 6

12114

1822

otherartifacts439

212

vertebrate

aquaticvertebrate

56 4

674

33


5

otheranimals

617

15656

06238

1319


532

1

othernstruments453

67mammal fish

661 564

56426 2

74412114

1319

0746





REFERENCES












































meadian AP () 289 334 344 305 297 340 349 367 379 370 393
















































































Selective search +

Edgeboxes

DeepID-Net(Pretrain

def-pooling)

Box rejection

Context modeling


Model averaging


person

horse

person

horse

person

horse

person

horse



3 METHOD

















128

128

128

128

conv61

conv63

conv71

conv73

def61

def63

Image

Candidate region



1000-class scores

200-class scores

























Extract HOG feature


layers


detector







layer


convolutional layer









where x = 1 W y = 1 H (2)



φ(δx δy) =

Nsum

n=1

acndδxδycn (5)









δxδycn


zδxδyc




















partb(xy)c

partacn= minusd

(ΔxΔy)cn


(6)





times Hsy






filter


Deformationpenalty

Output B

Maxpooling

Viewedas

blocks

Block-wiseaddition


sytimes W



343 Analysis






Pedicle Wheel Wheel
















Volleyball

Bathing cap

Golf ball

Seat belt








Ours 586

5

3049

41

19

5224

716

36

2221

21

LocSimOthBG

G-Net 63 35

29

59 28

111

66135

1641

1516

28













[17] [48] [63] [62] [18] [21] [16] [16] [15] [47] [46] [68]337 343 417 333 452 585 631 660 669 676 591 614 690













41 Ablation study















meadian AP () 289 294 305 367 37





meadian AP () 289 294 293 305 304





mAP () 226 256 299 323meadian AP () 198 230 289 317




meadian () 349 374 379 419 393 423












XS S M L XL

04

05

06

07

08

Rigid Deformable02

04

06

08

1


04

06

08

1


009 016 024 035 07802

04

06

08

1

013 029 039 04602

04

06

08

1

039 052 063 07802

04

06

08

1













Nc =1

Pc

Pcsum

pc=1

(npc ) (8)


0 50 100 150 2000

5

10

15

208 413 552 709 97702

04

06

08

1







0 50 100 150 2000

5

10

15x 10

4

104 205 299 409 62702

04

06

08

1










a4

2a2)2 (9)



1

2

3

4

5

Fac

tor

grou

p



1

2

3

4

5

Fac

tor

grou

p




1

2

3

4

5

Fact

or G

roup


1

2

3

4

5

Fact

or G

roup





2a1 a4

2a2)



(ij)2 minus a3d

(ij)3 minus a4d

(ij)4 minusa5

d(ij)1 =(iminus b1)




2(4a1) + a42(4a2) (10)




b = max(ij)

m(ij) (11)


6 CONCLUSION





H18 physical19910

22

H13 matter

H18

H1522

entity507

1304

H11

H10 solid510

497

13instrument

artifact

45 8

454107

128

588causalagent

1

2608

0617

0495

07155


478

510

510

9 13device

465

45840

iperson

588

588 g

1 1538

0495

07155

1304


532

1

animals

48

65 6

12114

1822

otherartifacts439

212

vertebrate

aquaticvertebrate

56 4

674

33


5

otheranimals

617

15656

06238

1319


532

1

othernstruments453

67mammal fish

661 564

56426 2

74412114

1319

0746





REFERENCES












































meadian AP () 289 334 344 305 297 340 349 367 379 370 393













































































128

128

128

128

conv61

conv63

conv71

conv73

def61

def63

Image

Candidate region



1000-class scores

200-class scores

























Extract HOG feature


layers


detector







layer


convolutional layer









where x = 1 W y = 1 H (2)



φ(δx δy) =

Nsum

n=1

acndδxδycn (5)









δxδycn


zδxδyc




















partb(xy)c

partacn= minusd

(ΔxΔy)cn


(6)





times Hsy






filter


Deformationpenalty

Output B

Maxpooling

Viewedas

blocks

Block-wiseaddition


sytimes W



343 Analysis






Pedicle Wheel Wheel
















Volleyball

Bathing cap

Golf ball

Seat belt








Ours 586

5

3049

41

19

5224

716

36

2221

21

LocSimOthBG

G-Net 63 35

29

59 28

111

66135

1641

1516

28













[17] [48] [63] [62] [18] [21] [16] [16] [15] [47] [46] [68]337 343 417 333 452 585 631 660 669 676 591 614 690













41 Ablation study















meadian AP () 289 294 305 367 37





meadian AP () 289 294 293 305 304





mAP () 226 256 299 323meadian AP () 198 230 289 317




meadian () 349 374 379 419 393 423












XS S M L XL

04

05

06

07

08

Rigid Deformable02

04

06

08

1


04

06

08

1


009 016 024 035 07802

04

06

08

1

013 029 039 04602

04

06

08

1

039 052 063 07802

04

06

08

1













Nc =1

Pc

Pcsum

pc=1

(npc ) (8)


0 50 100 150 2000

5

10

15

208 413 552 709 97702

04

06

08

1







0 50 100 150 2000

5

10

15x 10

4

104 205 299 409 62702

04

06

08

1










a4

2a2)2 (9)



1

2

3

4

5

Fac

tor

grou

p



1

2

3

4

5

Fac

tor

grou

p




1

2

3

4

5

Fact

or G

roup


1

2

3

4

5

Fact

or G

roup





2a1 a4

2a2)



(ij)2 minus a3d

(ij)3 minus a4d

(ij)4 minusa5

d(ij)1 =(iminus b1)




2(4a1) + a42(4a2) (10)




b = max(ij)

m(ij) (11)


6 CONCLUSION





H18 physical19910

22

H13 matter

H18

H1522

entity507

1304

H11

H10 solid510

497

13instrument

artifact

45 8

454107

128

588causalagent

1

2608

0617

0495

07155


478

510

510

9 13device

465

45840

iperson

588

588 g

1 1538

0495

07155

1304


532

1

animals

48

65 6

12114

1822

otherartifacts439

212

vertebrate

aquaticvertebrate

56 4

674

33


5

otheranimals

617

15656

06238

1319


532

1

othernstruments453

67mammal fish

661 564

56426 2

74412114

1319

0746





REFERENCES












































meadian AP () 289 334 344 305 297 340 349 367 379 370 393












































































Extract HOG feature


layers


detector







layer


convolutional layer









where x = 1 W y = 1 H (2)



φ(δx δy) =

Nsum

n=1

acndδxδycn (5)









δxδycn


zδxδyc




















partb(xy)c

partacn= minusd

(ΔxΔy)cn


(6)





times Hsy






filter


Deformationpenalty

Output B

Maxpooling

Viewedas

blocks

Block-wiseaddition


sytimes W



343 Analysis






Pedicle Wheel Wheel
















Volleyball

Bathing cap

Golf ball

Seat belt








Ours 586

5

3049

41

19

5224

716

36

2221

21

LocSimOthBG

G-Net 63 35

29

59 28

111

66135

1641

1516

28













[17] [48] [63] [62] [18] [21] [16] [16] [15] [47] [46] [68]337 343 417 333 452 585 631 660 669 676 591 614 690













41 Ablation study















meadian AP () 289 294 305 367 37





meadian AP () 289 294 293 305 304





mAP () 226 256 299 323meadian AP () 198 230 289 317




meadian () 349 374 379 419 393 423












XS S M L XL

04

05

06

07

08

Rigid Deformable02

04

06

08

1


04

06

08

1


009 016 024 035 07802

04

06

08

1

013 029 039 04602

04

06

08

1

039 052 063 07802

04

06

08

1













Nc =1

Pc

Pcsum

pc=1

(npc ) (8)


0 50 100 150 2000

5

10

15

208 413 552 709 97702

04

06

08

1







0 50 100 150 2000

5

10

15x 10

4

104 205 299 409 62702

04

06

08

1










a4

2a2)2 (9)



1

2

3

4

5

Fac

tor

grou

p



1

2

3

4

5

Fac

tor

grou

p




1

2

3

4

5

Fact

or G

roup


1

2

3

4

5

Fact

or G

roup





2a1 a4

2a2)



(ij)2 minus a3d

(ij)3 minus a4d

(ij)4 minusa5

d(ij)1 =(iminus b1)




2(4a1) + a42(4a2) (10)




b = max(ij)

m(ij) (11)


6 CONCLUSION





H18 physical19910

22

H13 matter

H18

H1522

entity507

1304

H11

H10 solid510

497

13instrument

artifact

45 8

454107

128

588causalagent

1

2608

0617

0495

07155


478

510

510

9 13device

465

45840

iperson

588

588 g

1 1538

0495

07155

1304


532

1

animals

48

65 6

12114

1822

otherartifacts439

212

vertebrate

aquaticvertebrate

56 4

674

33


5

otheranimals

617

15656

06238

1319


532

1

othernstruments453

67mammal fish

661 564

56426 2

74412114

1319

0746





REFERENCES












































meadian AP () 289 334 344 305 297 340 349 367 379 370 393












































































filter


Deformationpenalty

Output B

Maxpooling

Viewedas

blocks

Block-wiseaddition


sytimes W



343 Analysis






Pedicle Wheel Wheel
















Volleyball

Bathing cap

Golf ball

Seat belt








Ours 586

5

3049

41

19

5224

716

36

2221

21

LocSimOthBG

G-Net 63 35

29

59 28

111

66135

1641

1516

28













[17] [48] [63] [62] [18] [21] [16] [16] [15] [47] [46] [68]337 343 417 333 452 585 631 660 669 676 591 614 690













41 Ablation study















meadian AP () 289 294 305 367 37





meadian AP () 289 294 293 305 304





mAP () 226 256 299 323meadian AP () 198 230 289 317




meadian () 349 374 379 419 393 423












XS S M L XL

04

05

06

07

08

Rigid Deformable02

04

06

08

1


04

06

08

1


009 016 024 035 07802

04

06

08

1

013 029 039 04602

04

06

08

1

039 052 063 07802

04

06

08

1













Nc =1

Pc

Pcsum

pc=1

(npc ) (8)


0 50 100 150 2000

5

10

15

208 413 552 709 97702

04

06

08

1







0 50 100 150 2000

5

10

15x 10

4

104 205 299 409 62702

04

06

08

1










a4

2a2)2 (9)



1

2

3

4

5

Fac

tor

grou

p



1

2

3

4

5

Fac

tor

grou

p




1

2

3

4

5

Fact

or G

roup


1

2

3

4

5

Fact

or G

roup





2a1 a4

2a2)



(ij)2 minus a3d

(ij)3 minus a4d

(ij)4 minusa5

d(ij)1 =(iminus b1)




2(4a1) + a42(4a2) (10)




b = max(ij)

m(ij) (11)


6 CONCLUSION





H18 physical19910

22

H13 matter

H18

H1522

entity507

1304

H11

H10 solid510

497

13instrument

artifact

45 8

454107

128

588causalagent

1

2608

0617

0495

07155


478

510

510

9 13device

465

45840

iperson

588

588 g

1 1538

0495

07155

1304


532

1

animals

48

65 6

12114

1822

otherartifacts439

212

vertebrate

aquaticvertebrate

56 4

674

33


5

otheranimals

617

15656

06238

1319


532

1

othernstruments453

67mammal fish

661 564

56426 2

74412114

1319

0746





REFERENCES












































meadian AP () 289 334 344 305 297 340 349 367 379 370 393












































































Volleyball

Bathing cap

Golf ball

Seat belt








Ours 586

5

3049

41

19

5224

716

36

2221

21

LocSimOthBG

G-Net 63 35

29

59 28

111

66135

1641

1516

28













[17] [48] [63] [62] [18] [21] [16] [16] [15] [47] [46] [68]337 343 417 333 452 585 631 660 669 676 591 614 690













41 Ablation study















meadian AP () 289 294 305 367 37





meadian AP () 289 294 293 305 304





mAP () 226 256 299 323meadian AP () 198 230 289 317




meadian () 349 374 379 419 393 423












XS S M L XL

04

05

06

07

08

Rigid Deformable02

04

06

08

1


04

06

08

1


009 016 024 035 07802

04

06

08

1

013 029 039 04602

04

06

08

1

039 052 063 07802

04

06

08

1













Nc =1

Pc

Pcsum

pc=1

(npc ) (8)


0 50 100 150 2000

5

10

15

208 413 552 709 97702

04

06

08

1







0 50 100 150 2000

5

10

15x 10

4

104 205 299 409 62702

04

06

08

1










a4

2a2)2 (9)



1

2

3

4

5

Fac

tor

grou

p



1

2

3

4

5

Fac

tor

grou

p




1

2

3

4

5

Fact

or G

roup


1

2

3

4

5

Fact

or G

roup





2a1 a4

2a2)



(ij)2 minus a3d

(ij)3 minus a4d

(ij)4 minusa5

d(ij)1 =(iminus b1)




2(4a1) + a42(4a2) (10)




b = max(ij)

m(ij) (11)


6 CONCLUSION





H18 physical19910

22

H13 matter

H18

H1522

entity507

1304

H11

H10 solid510

497

13instrument

artifact

45 8

454107

128

588causalagent

1

2608

0617

0495

07155


478

510

510

9 13device

465

45840

iperson

588

588 g

1 1538

0495

07155

1304


532

1

animals

48

65 6

12114

1822

otherartifacts439

212

vertebrate

aquaticvertebrate

56 4

674

33


5

otheranimals

617

15656

06238

1319


532

1

othernstruments453

67mammal fish

661 564

56426 2

74412114

1319

0746





REFERENCES












































meadian AP () 289 334 344 305 297 340 349 367 379 370 393














































































[17] [48] [63] [62] [18] [21] [16] [16] [15] [47] [46] [68]337 343 417 333 452 585 631 660 669 676 591 614 690













41 Ablation study















meadian AP () 289 294 305 367 37





meadian AP () 289 294 293 305 304





mAP () 226 256 299 323meadian AP () 198 230 289 317




meadian () 349 374 379 419 393 423












XS S M L XL

04

05

06

07

08

Rigid Deformable02

04

06

08

1


04

06

08

1


009 016 024 035 07802

04

06

08

1

013 029 039 04602

04

06

08

1

039 052 063 07802

04

06

08

1













Nc =1

Pc

Pcsum

pc=1

(npc ) (8)


0 50 100 150 2000

5

10

15

208 413 552 709 97702

04

06

08

1







0 50 100 150 2000

5

10

15x 10

4

104 205 299 409 62702

04

06

08

1










a4

2a2)2 (9)



1

2

3

4

5

Fac

tor

grou

p



1

2

3

4

5

Fac

tor

grou

p




1

2

3

4

5

Fact

or G

roup


1

2

3

4

5

Fact

or G

roup





2a1 a4

2a2)



(ij)2 minus a3d

(ij)3 minus a4d

(ij)4 minusa5

d(ij)1 =(iminus b1)




2(4a1) + a42(4a2) (10)




b = max(ij)

m(ij) (11)


6 CONCLUSION





H18 physical19910

22

H13 matter

H18

H1522

entity507

1304

H11

H10 solid510

497

13instrument

artifact

45 8

454107

128

588causalagent

1

2608

0617

0495

07155


478

510

510

9 13device

465

45840

iperson

588

588 g

1 1538

0495

07155

1304


532

1

animals

48

65 6

12114

1822

otherartifacts439

212

vertebrate

aquaticvertebrate

56 4

674

33


5

otheranimals

617

15656

06238

1319


532

1

othernstruments453

67mammal fish

661 564

56426 2

74412114

1319

0746





REFERENCES












































meadian AP () 289 334 344 305 297 340 349 367 379 370 393
















































































meadian AP () 289 294 305 367 37





meadian AP () 289 294 293 305 304





mAP () 226 256 299 323meadian AP () 198 230 289 317




meadian () 349 374 379 419 393 423












XS S M L XL

04

05

06

07

08

Rigid Deformable02

04

06

08

1


04

06

08

1


009 016 024 035 07802

04

06

08

1

013 029 039 04602

04

06

08

1

039 052 063 07802

04

06

08

1













Nc =1

Pc

Pcsum

pc=1

(npc ) (8)


0 50 100 150 2000

5

10

15

208 413 552 709 97702

04

06

08

1







0 50 100 150 2000

5

10

15x 10

4

104 205 299 409 62702

04

06

08

1










a4

2a2)2 (9)



1

2

3

4

5

Fac

tor

grou

p



1

2

3

4

5

Fac

tor

grou

p




1

2

3

4

5

Fact

or G

roup


1

2

3

4

5

Fact

or G

roup





2a1 a4

2a2)



(ij)2 minus a3d

(ij)3 minus a4d

(ij)4 minusa5

d(ij)1 =(iminus b1)




2(4a1) + a42(4a2) (10)




b = max(ij)

m(ij) (11)


6 CONCLUSION





H18 physical19910

22

H13 matter

H18

H1522

entity507

1304

H11

H10 solid510

497

13instrument

artifact

45 8

454107

128

588causalagent

1

2608

0617

0495

07155


478

510

510

9 13device

465

45840

iperson

588

588 g

1 1538

0495

07155

1304


532

1

animals

48

65 6

12114

1822

otherartifacts439

212

vertebrate

aquaticvertebrate

56 4

674

33


5

otheranimals

617

15656

06238

1319


532

1

othernstruments453

67mammal fish

661 564

56426 2

74412114

1319

0746





REFERENCES












































meadian AP () 289 334 344 305 297 340 349 367 379 370 393












































































XS S M L XL

04

05

06

07

08

Rigid Deformable02

04

06

08

1


04

06

08

1


009 016 024 035 07802

04

06

08

1

013 029 039 04602

04

06

08

1

039 052 063 07802

04

06

08

1













Nc =1

Pc

Pcsum

pc=1

(npc ) (8)


0 50 100 150 2000

5

10

15

208 413 552 709 97702

04

06

08

1







0 50 100 150 2000

5

10

15x 10

4

104 205 299 409 62702

04

06

08

1










a4

2a2)2 (9)



1

2

3

4

5

Fac

tor

grou

p



1

2

3

4

5

Fac

tor

grou

p




1

2

3

4

5

Fact

or G

roup


1

2

3

4

5

Fact

or G

roup





2a1 a4

2a2)



(ij)2 minus a3d

(ij)3 minus a4d

(ij)4 minusa5

d(ij)1 =(iminus b1)




2(4a1) + a42(4a2) (10)




b = max(ij)

m(ij) (11)


6 CONCLUSION





H18 physical19910

22

H13 matter

H18

H1522

entity507

1304

H11

H10 solid510

497

13instrument

artifact

45 8

454107

128

588causalagent

1

2608

0617

0495

07155


478

510

510

9 13device

465

45840

iperson

588

588 g

1 1538

0495

07155

1304


532

1

animals

48

65 6

12114

1822

otherartifacts439

212

vertebrate

aquaticvertebrate

56 4

674

33


5

otheranimals

617

15656

06238

1319


532

1

othernstruments453

67mammal fish

661 564

56426 2

74412114

1319

0746





REFERENCES












































meadian AP () 289 334 344 305 297 340 349 367 379 370 393












































































0 50 100 150 2000

5

10

15x 10

4

104 205 299 409 62702

04

06

08

1










a4

2a2)2 (9)



1

2

3

4

5

Fac

tor

grou

p



1

2

3

4

5

Fac

tor

grou

p




1

2

3

4

5

Fact

or G

roup


1

2

3

4

5

Fact

or G

roup





2a1 a4

2a2)



(ij)2 minus a3d

(ij)3 minus a4d

(ij)4 minusa5

d(ij)1 =(iminus b1)




2(4a1) + a42(4a2) (10)




b = max(ij)

m(ij) (11)


6 CONCLUSION





H18 physical19910

22

H13 matter

H18

H1522

entity507

1304

H11

H10 solid510

497

13instrument

artifact

45 8

454107

128

588causalagent

1

2608

0617

0495

07155


478

510

510

9 13device

465

45840

iperson

588

588 g

1 1538

0495

07155

1304


532

1

animals

48

65 6

12114

1822

otherartifacts439

212

vertebrate

aquaticvertebrate

56 4

674

33


5

otheranimals

617

15656

06238

1319


532

1

othernstruments453

67mammal fish

661 564

56426 2

74412114

1319

0746





REFERENCES












































meadian AP () 289 334 344 305 297 340 349 367 379 370 393












































































H18 physical19910

22

H13 matter

H18

H1522

entity507

1304

H11

H10 solid510

497

13instrument

artifact

45 8

454107

128

588causalagent

1

2608

0617

0495

07155


478

510

510

9 13device

465

45840

iperson

588

588 g

1 1538

0495

07155

1304


532

1

animals

48

65 6

12114

1822

otherartifacts439

212

vertebrate

aquaticvertebrate

56 4

674

33


5

otheranimals

617

15656

06238

1319


532

1

othernstruments453

67mammal fish

661 564

56426 2

74412114

1319

0746





REFERENCES












































meadian AP () 289 334 344 305 297 340 349 367 379 370 393















































































meadian AP () 289 334 344 305 297 340 349 367 379 370 393































































































Documents

IEEE TRANSACTIONS PATTERN ANALYSIS AND MACHINE ...xgwang/papers/ouyangZWpami16.pdf · Citation information: DOI 10.1109/TPAMI.2016.2587642, IEEE Transactions on Pattern Analysis and