(* equal contribution)

[1] R. Girshick. "Fast R-CNN." arXiv preprint arXiv:1504.08083

[2] S. Ren, K. He, R. Girshick and J. Sun. "Faster r-cnn: Towards
real-time object detection with region proposal networks." arXiv
preprint arXiv:1506.01497

[3] C. L. Zitnick and P. Dollár. "Edge boxes: Locating object proposals from edges." ECCV 2014

[4] G. Nebehay and R. Pflugfelder. "Clustering of Static-Adaptive Correspondences for Deformable Object Tracking." CVPR 2015

CLS-LOC:Jing Yang(2), Shaoli Huang(1), Zhengbo Yu(2), Qiang Ma(2), Jiankang Deng(1,2)

VID: Jiankang Deng(1,2), Jing Yang(2), Shaoli Huang(1), Hui Shuai(2),Yi Wu(2), Qingshan Liu(2), Dacheng Tao(1)

(1)University of Technology, Sydney
(2)Nanjing University of Information Science & Technology

1.DET:Spatial Cacade region regression

We first set up Faster RCNN[3] as our baseline. (mAP 45.6% for VGG-16;mAP 47.2% for Google-net).

Object detection is to answer "Where" and "What".

We utilize cascade regression regression model to gradually to refine the location of object, which is helpful to answer "what".

Solid tricks including:

Negative example (discriminative feature is inhomogeneous on the
objects, response maps is helpful for choosing reasonable positive and
negative examples, instead of only using IoU).

Multi-scale(image,joint feature map,inception layer; Answer "where"
from former CNN, and "what" from later with high capacity models).

Learn to combine(NMS inter-class, NMS intra-class; exclusive in space).

Learn to rank(hypothesis：the data number distribution between
validation set and test set is similar. We can choose class-specific
parameters).

Add training samples for classes with little training data.

Design class-specific model for hard classes.

Rank low resolution and dense prediction later.

Model ensemble with multi-view learning.

2.VID: Tempor Cascade region regression

Objectiveness based tracker is designed to track the objects on videos.

Firstly, We train Faster-RCNN[3](VGG-16) with the provided training data (sampling from frames).

The network provides features for tracking.

Secondly,the tracker uses the roi_pooling features from the last
conv layer and tempor information, which can be seen as the tempor Fast
RCNN[2].

(Take the location-indexed features from current frame to predict the bounding box of object on next frame.)

Tempor information and scence cluster(different video from one
scence) are greatly helpful to decide the classes on the videos with
high confidence.

[1]Girshick R, Donahue J, Darrell T, et al. Rich feature hierarchies
for accurate object detection and semantic segmentation[C]//Computer
Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on. IEEE,
2014: 580-587.

[2]Girshick R. Fast R-CNN[J]. arXiv preprint arXiv:1504.08083, 2015.

[3]Ren S, He K, Girshick R, et al. Faster r-cnn: Towards real-time
object detection with region proposal networks[J]. arXiv preprint
arXiv:1506.01497, 2015.

References:

[1] "MatConvNet - Convolutional Neural Networks for MATLAB", A. Vedaldi and K. Lenc, arXiv:1412.4564, 2014

[2] "Very Deep Convolutional Networks for Large-Scale Image
Recognition", Karen Simonyan and Andrew Zisserman, arXiv technical
report, 2014

[3] "Return of the Devil in the Details: Delving Deep into
Convolutional Networks", Ken Chatfield, Karen Simonyan, Andrea Vedaldi,
and Andrew Zisserman, BMVC 2014

Like rectified linear units (ReLUs) [2, 3], leaky ReLUs (LReLUs) and
parametrized ReLUs (PReLUs), ELUs also avoid a vanishing gradient via
the identiy for positive values. However ELUs have improved learning
characteristics compared to the other activation functions. In contrast
to ReLUs, ELUs have negative values which allows them to push mean unit
activations closer to zero. Zero means speed up learning because they
bring the gradient closer to the unit natural gradient.

The unit natural gradient differs from the normal gradient by a bias
shift term, which is proportional to the mean activation of incoming
units. Like batch normalization, ELUs push the mean towards zero, but
with a significantly smaller computational footprint. While other
activation functions like LReLUs and PReLUs also have negative values,
they do not ensure a noise-robust deactivation state. ELUs saturate to a
negative value with smaller inputs and thereby decrease the propagated
variation and information. Therefore ELUs code the degree of presence of
particular phenomena in the input, while they do not quantitatively
model the degree of their absence. Consequently dependencies between
ELU units are much easier to model and distinct concepts are less
likely to interfere.

In this challenge ELU networks considerably speed up learning
compared to a ReLU network with similar classification performance.

[1] Clevert, Djork-Arné et al, Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs), arxiv 2015

[2] Clevert, Djork-Arné et al, Rectified Factor Networks, NIPS 2015

[3] Mayr, Andreas et al, DeepTox: Toxicity Prediction using Deep Learning, Frontiers in Environmental Science 2015

The network for classfication pre-trained on the ILSVRC 2014
classification dataset is modified for bounding box regression. Then the
regressor to predict a bouding box is fine-tuned using the features of
middle and last convolutional layer [2]. At inference time, the modified
greedy-merge techinque [3] for multi-scale prediction is applied on
each scale, and the optimal scale is chosen to determin the final
predicted box.

[1] Szegedy et al., Going Deeper with Convolutions, CVPR, 2015.

[2] Long & Shelhamer et al., Fully Convolutional Networks for Semantic Segmentation, CVPR, 2015.

[3] Sermanet et al., OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks, ICLR, 2014.

[4] Lin et al., Network In Network, ICLR, 2014.

1. The Chinese University of *

2. SenseTime Group Limited

Compared with DeepID-Net in ILSVRC 2014, the new components are as follows.

(1) GoogleNet with batch normalization and VGG are pre-trained on
1000-class ImageNet classification/location data (for the entries of
using official data only) and 3000-class ImageNet classification data
(for the entries of using extra data).

(2) A new cascade method is introduced to generate region proposals. It has higher recall rate with fewer region proposals.

(3) The models are fine-tuned on 200 detection classes with multi-context and multi-crop.

(4) The 200 classes are clustered in a hierarchical way based on
their visual similarity. Instead of finetuning with the 200 classes
once, the models are finetuned for multiple times when the 200 classes
are divided into smaller clusters iteratively. Different clusters share
different feature representations. Feature representations gradually
adapt to individual classes in this way.

Compared with Faster RCNN, the new components are

(1) We cascade the RPN, where the proposals generated by RPN are fed
into a object\background Fast-RCNN. It leads to 93% recall rate with
about 126 proposals per image on val2.

(2) We cascade the Fast-RCNN, where a Fast-RCNN with category-wise
softmax loss is used in the cascade step for hard negative mining. It
leads to about 2% improvement in AP.

Deep-ID models and Faster RCNN models are combined with model averaging.

For the localization task, class labels are predicted with VGG. For
each image and each predicted class, candidate regions are proposed by
employing a learned saliency map and edge boxes [c] with high recall
rate. Candidate regions are assigned with scores by using VGG or
GoogLeNet with BN finetuned on candidate regions. The 1000 classes are
grouped in multiple clustered in a hierarchical way. VGG and GoogleNet
are finetuned for multiple times to adapt to different clusters.

The fastest publicly available multi-GPU caffe code (requires only 6
seconds for 20 iterations when mini-batch size is 256 using GoogleNet
using 4 TitanX) is our strong support [d].

[a] W. Ouyang, X. Wang, X. Zeng, S. Qiu, P. Luo, Y. Tian, H. Li, S.
Yang, Z. Wang, C. Loy, X. Tang, “DeepID-Net: Deformable Deep
Convolutional Neural Networks for Object Detection,” CVPR 2015.

[b] Shaoqing Ren, Kaiming He, Ross Girshick, Jian Sun, “Faster
R-CNN: Towards Real-Time Object Detection with Region Proposal
Networks,” arXiv:1506.01497.

[c] C. Lawrence Zitnick, Piotr Dollár, “Edge Boxes: Locating Object Proposals from Edges”, ECCV2014

[d] https://github.com/yjxiong/caffe

1. The Chinese University of *

2. SenseTime Group Limited

[1] Spyros Gidaris, Nikos Komodakis, "Object detection via a
multi-region & semantic segmentation-aware CNN model",
arXiv:1505.01749

[2] Shaoqing Ren, Kaiming He, Ross Girshick, Jian Sun," Faster
R-CNN: Towards Real-Time Object Detection with Region Proposal
Networks",NIPS, 2015

Our modifications (for Inception-7a-like model) include:

1) Using MSRA weight initialization scheme [3].

2) Using Randomized ReLU units [4].

3) More agressive data augmentations: random rotations, random
skewing, random stretching, brightness, contrast and color
augmentations.

4) Test-time data augmentation: We applied 30 different random data
augmentations to 30 copies of each test image, passed them through net
and averaged predictions.

Also we trained one of our Inception-6-like models on object-level
images (we cut objects from full images by their localization boxes and
used them to train the net).

This is still work-in-progress, for now networks haven't finished
training yet. We also didn't predict any localization boxes yet.

[1] Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift,

Sergey Ioffe, Christian Szegedy http://arxiv.org/abs/1502.03167

[2] Scene Classification with Inception-7, Christian Szegedy with
Julian Ibarz and Vincent Vanhoucke
http://lsun.cs.princeton.edu/slides/Christian.pdf

[3] Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun http://arxiv.org/abs/1502.01852

[4] Empirical Evaluation of Rectified Activations in Convolutional
Network, Bing Xu, Naiyan Wang, Tianqi Chen, Mu Li,
http://arxiv.org/abs/1505.00853

[1] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556, 2014.

[2] He K, Zhang X, Ren S, et al. Delving deep into rectifiers:
Surpassing human-level performance on imagenet classification.
arXiv:1502.01852, 2015.

[3] B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva.
Learning Deep Features for Scene Recognition using Places Database.
Advances in Neural Information Processing Systems 27 (NIPS), 2014.

[4] B. Zhou, A. Khosla, A. Lapedriza, A. Torralba and A. Oliva.
Places2: A Large-Scale Database for Scene Understanding. Arxiv, 2015.

DEEPimagine Co. Ltd. of South Korea

2. Enhancements

- Detection framework

: Tuning the location of ROI(Region Of Interest) projection, Adding the fixed region proposals.

- Models

: More depth models, More inception models.

We focused on a efficiency of the pooling layers and full connected inner product layers.

So we converted pooling layers and full connected inner product layers to our own custom layers.

- Fusion

: Category adaptive multi-model fusion.

3. References

[1] Ross Girshick. "Fast R-CNN: Fast Region-based Convolutional Networks for object detection", CVPR 2015

[2] Shaoqing Ren, Kaiming He, Ross Girshick, Jian Sun. "Faster
R-CNN: Towards Real-Time Object Detection with Region Proposal
Networks", CVPR 2015

[1]R. Girshick, J. Donahue, T. Darrell, J. Malik, "Rich feature
hierarchies for accurate object detection and semantic segmentation",
CVPR 2014.

[2]R. Girshick, "Fast R-CNN", ICCV 2015.

[3]S. Ren, K. He, R. Girshick, J. Sun, "Faster R-CNN: Towards
Real-Time Object Detection with Region Proposal Networks", NIPS 2015.

[4]W. Ouyang etal, "DeepID-Net: Deformable Deep Convolutional Neural Networks for Object detection", CVPR 2015.

[5]S. Gidaris, N, Komodakis, "Object detection via a multi-region & semantic segmentation-aware CNN model", ICCV 2015.

Task 1 Object Classification/Localization

===============

We utilize the GoogLeNet with batch normalization and prelu for
object classification. Three models are trained. The first one uses the
original GoogLeNet architecture but all relu layers are replaced with
prelu layers. The second model is the one mentioned in Ref. [2]. The
third model is fine-tuned from the second model with multi-scale
training. Multi-scale testing and models ensemble is utilized to
generate the final classification result. 144 crops of an image [1] are
used to evaluate one network. Actually, we also tried the method of 150
crops which is described in Ref. [3]. The performance is almost the
same. And merging the results of 144 crops and 150 crops does not bring
to much increased performance. The top-5 error of the three models on
validation is 8.89%, 8.00% and 7.79%, respectively. And using an
ensemble of the three models, we decreased the top-5 error rate to
7.03%. As we all know, ensemble with more models can improve the
performance. But we do not have enough time and GPUs to do that.

To generate a bounding box for each label of an image, we firstly
fine-tune the second classification model with object-level annotations
of 1,000 classes from ImageNet CLS-LOC train data. Moreover, a
background class is added into the network. Then test images are
segmented into ~300 regions by selective search and these regions are
classified by the fine-tuned model into one of 1,001 classes. We select
the top-3 regions with the highest possibility classes generated by the
classification model. A new bounding box is generated by finding a
minimal bounding rectangle of three regions. The localization error is
about 32.9% on validation. We also try the third classification model.
And the localization error on validation is 32.76%. After merging the
two aforementioned results, the localization error decrease to 31.53%.

Task 2 Object Detection

===============

We employ the well-known Fast-RCNN framework [4]. Firstly, we tried
the AlexNet model which is pre-trained on the CLS-LOC dataset with image
–level annotation. When training on the object detection dataset, we
run SDG for 12 epochs, and then lower the learning rate from 0.001 to
0.0001 and train for another 4 epochs. The other setting is the same
with the original Fast-RCNN method. This approach achieves 34.9% MAP on
validation. Then we apply GoogLeNet with Fast-RCNN framework. The pool
layer after inception 4 layers is replaced by a ROI pooling layer. This
trial achieves about 34% map on the validation. In another trial, we
move the ROI pooling layer from the pool4 to pool5 and enlarge the input
size from 600(max 1000) to 786(max 1280). The pooled width and height
is set to 4x4 instead of 1x1. The MAP is about 37.8% on validation. It
is worth noting that the last model needs about 6g GPU memory to train
and 1.5g GPU memory to test. And it has near the same test speed with
AlexNet but gains better performance. We employ a simple strategy to
merge the three results and gain 38.7% MAP.

Task 3 Scene Classification

===============

The Places2 training set has about 8 million images. We reduce the
input size of images from 256x256 to 128x128 and try small network
architectures to accelerate the training procedure. However, due to the
large amount images, we still cannot finish the training before the
submission deadline. We trained two models which architectures are
modified from the original GOOGLENET. The first one only removes the
inception 5 layers. We only trained the model for about 10 epochs. The
top-5 error on validation is about 37.19%. The second model enlarges the
stride of the conv1 layer from 2 to 4 and reduces the kernel number of
the conv2 layer from 192 to 96. For the remaining inception layers,
every kernel number is set as the half of its original number. This
model is trained about 12 epochs and achieves 38.99% top-5 error on the
validation. Unfortunately, about one week ago, we found that the final
output vector was set to 400 instead of 401 due to an oversight. We
correct the error and fine-tune the first model for about 1 epoch. The
top-5 error on validation of this model is about 31.42%.

Task 4 Object Detection from Video

===============

A simple method for this task is to perform object detection in all
frames. But it does not utilize the spatial temporal constraint or
context information between continual frames in a video. Thus, we employ
object detection and object tracking for this track. First, key frames
from a video are selected to detect objects in them using Fast RCNN.
There are about 1.3 million frames in training set. Due to temporal
continuity, we select one frame every 25 frames to train an object
detection model. 52,922 frames are utilized to train the model. Similar
to the approach in the Object Detection track, we run SGD for 12 epochs
and then lower the learning rate from 0.001 to 0.0001 and train for
another 8 epochs. The training procedure takes 1 day and 3 days on a
single K40 for AlexNet and GoogLeNet, respectively. More details about
the object detection can be found in the instruction of the task 2.
During test, if a video has less than 50 frames, we choose two frames in
the middle of the video. If a video has more than 50 frames, we choose a
frame every 25 frames. This results in 6,861 frames of 176,126 on
validation and 12,329 frames of 315,176 on test. We do object detection
on these frames and filter out the objects which confident scores are
larger than 0.2(AlexNet) and 0.4(GoogLeNet). Then, detected objects are
tracked to generate the results of the other frames. The tracking method
we used is TLD [5]. After tracking, we generate the final results for
evaluation. It is worth noting that we set most of parameters
empirically because we have no time to validate them. Three entries are
provided. The first one utilizes AlexNet and achieves 19.1% map on
validation. The second one uses GoogLeNet and achieves 24.6% map on
validation. A simple strategy to merge the two results is employed and
results in the third entry. The map of it is 25.3% on validation.

[1] Szegedy C, Liu W, Jia Y, et al. Going Deeper With
Convolutions[C]//Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition. 2015: 1-9.

[2] Ioffe S, Szegedy C. Batch normalization: Accelerating deep
network training by reducing internal covariate shift[J]. arXiv preprint
arXiv:1502.03167, 2015.

[3] Simonyan K, Zisserman A. Very deep convolutional networks for
large-scale image recognition[J]. arXiv preprint arXiv:1409.1556, 2014.

[4] Girshick R. Fast R-CNN[J]. arXiv preprint arXiv:1504.08083, 2015.

[5] Kalal, Z.; Mikolajczyk, K.; Matas, J.,
"Tracking-Learning-Detection," in Pattern Analysis and Machine
Intelligence, IEEE Transactions on , vol.34, no.7, pp.1409-1422, July
2012

We used pre-trained Imagenet 2014 classification models (VGG16, VGG19) to train detection models.

For ILSVRC 2015 detection datasets, we trained Fast R-CNN and Faster
R-CNN multiple times and the trained models are used for ensemble.

Three algorithms are used for the region proposal: Selective
search[4], Region proposal network step_1[3] and Region proposal network
step_3[3]. The three kinds of proposals are fed to detection models
(VGG16, VGG19) which then are used for ensemble.

Detection results are 41.7% mAP for a single model and 44.1% mAP for the ensemble of multiple models.

References:

[1] Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation

R. Girshick, J. Donahue, T. Darrell, J. Malik

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014

[2] Fast R-CNN

Ross Girshick

IEEE International Conference on Computer Vision (ICCV), 2015

[3] Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Shaoqing Ren, Kaiming He, Ross Girshick, Jian Sun

Neural Information Processing Systems (NIPS), 2015

[4] Selective Search for Object Recognition

Jasper R. R. Uijlings, Koen E. A. van de Sande, Theo Gevers, Arnold W. M. Smeulders

International Journal of Computer Vision, Volume 104 (2), page 154-171, 2013

[1] Chen Z, Lam O, Jacobson A, et al. Convolutional neural
network-based place recognition[J]. arXiv preprint arXiv:1411.1509,
2014.

[2] Zhou B, Khosla A, Lapedriza A, et al. Object detectors emerge in deep scene cnns[J]. arXiv preprint arXiv:1412.6856, 2014.

[3] Simonyan K, Zisserman A. Very deep convolutional networks for
large-scale image recognition[J]. arXiv preprint arXiv:1409.1556, 2014.

The most recent (as of Nov 14, 2015) top-5 and top-1 accuracies of
Henry machine for the Scene401 validation dataset are 68.53% and 36.15%,
respectively.

Here are some characteristics of Henry machine.

- The features used are our own modified version of a selection of
features in the literature. These are engineered features, not learned
features. The feature extraction was done for all the 8.1M training,
380K test, and 20K validation images on their original, high-resolution
version with the original aspect ratio left unchanged. I.e., no image
resizing or rescaling were applied. The entire feature extraction step
using our own implementation took 7 days to complete on a cluster of
home-brewed CPU cluster (listed below), which consists of 12 low-end to
mid-end computers borrowed from family and friends. Five of the twelve
computers are laptops.

- We did not have time to study and implement many strong features
in the literature. The accuracy of Henry machine is expected to increase
once we include more such features. We believe that the craftsmanship
of feature engineering encodes the expertise and ingenuity of the human
designer behind it, and cannot be dispensed away (at least not yet) in
spite of the current popularity of deep learning and representation
learning-based approaches. Our humble goal with Henry machine is to
encourage continued research interest in the craftsmanship of feature
engineering.

- The training of Henry machine for Scene401 was also done using the
home-brewed CPU cluster, and took 21 days to complete (not counting
algorithm design/development/debugging time).

- While Henry machine was trained using traditional classification
methodology, the classifier itself was devised by us from scratch
instead of using conventionally available methods such as SVM. We did
not use SVM because it suffers from several drawbacks. For example, SVM
is fundamentally a binary classifier, and applying its maximum-margin
formulation in a multiclass setting, especially with such a large number
of classes (401 and 1000), could be ad hoc. Also, the output of SVM is
inherently non-probabilistic, which makes it less convenient to use in a
probabilistic setting. The training algorithm behind Henry machine is
our attempt to address these and several other issues (not mentioned
here), while at the same time to make it efficient to train, with small
memory footprint, on mom-and-pop computers at home, using only CPU's.
However, Henry machine is still very far from perfect, and our training
algorithm still needs a lot of improvement. More details of the training
and prediction algorithm will be available in our publication.

- As Nov 13 was fast approaching, we were pressed by time. The delay
was mainly due to hardware heat stress from many
triple-digit-temperature days in Sep and Oct here in California. In the
end of Oct, we bought two GTX 970 graphics card and implemented
high-performance CUDA code (driver version 7.5) to help us finish the
final prediction phase in time.

- We will also release the performance report of Henry machine on the ImageNet1000 CLS-LOC validation dataset.

- The source code for building Henry machine, including the feature
extraction part, was primarily written in Octave (version 4.0.0) and C++
(gcc 4.8.4) on linux machines (Ubuntu x86_64 14.04).

Here is the list of the CPU's of the home-brewed cluster:

* Pentium D 2.8GHz, 3G DDR (2005 Desktop)

* Pentium D 2.8GHz, 3.26G DDR (2005 Desktop)

* Pentium 4 3.0GHz, 3G DDR (2005 Desktop)

* Core 2 Duo T5500 1.66GHz, 3G DDR2 (2006 Laptop)

* Celeron E3200 2.4GHz, 3G DDR2 (2009 Desktop)

* Core i7-720QM 1.6GHz, 20G DDR3 (2009 Laptop)

* Xeon E5620 2.4GHz (x2), 16G DDR3 (2010 Server)

* Core i5-2300 2.8GHz, 6G DDR3 (2011 Desktop)

* Core i7-3610QM 2.3GHz, 16G DDR3 (2012 Laptop)

* Pending info (2012 Laptop)

* Core i7-4770K 3.5GHz, 32G DDR3 (2013 Desktop)

* Core i7-4500U 1.8GHz, 8G DDR3 (2013 Laptop)

[CLS-LOC] We train different models for classification and
localization separately, i.e. GoogLeNet [4,5] for classification and
VGG16 for bounding box regression. The final models achieve 28.9% top-5
cls-loc error and 6.62% cls error on the validation set.

[Scene] Due to the limit of time and GPUs, we have just trained one
CNN model for the scene classification task, namely VGG19, based on the
resized 256x256 image datasets. The top-5 accuracy on the validation set
with single center crop is 79.9%. In test phase, a multi-scale dense
evaluation is adopted for the prediction, whose accuracy on the
validation set is 80.7%.

[VID] First we apply Fast R-CNN with RPN proposals [2] to detect
objects frame by frame. Then a Multi-Object Tracking (MOT) [6] method is
utilized to associate the detections for each snippet. Validation mAP
is 43.1%.

References:

[1] R. Girshick. Fast R-CNN. arXiv 1504.08083, 2015.

[2] Ren S, He K, Girshick R, et al. Faster r-cnn: Towards real-time
object detection with region proposal networks. arXiv 1506.01497, 2015.

[3] K. Simonyan, A. Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv 1409.1556, 2014.

[4] C. Szegedy, W. Liu, Y. Jia, et al. Going Deeper with Convolutions. arXiv 1409.4842, 2014.

[5] S. Ioffe, C. Szegedy. Batch Normalization: Accelerating Deep
Network Training by Reducing Internal Covariate Shift. arXiv 1502.03167,
2015.

[6] Bae S H, Yoon K J. Robust online multi-object tracking based on
tracklet confidence and online discriminative appearance learning. 2014
IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE,
2014: 1218-1225.

[1] B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, And A. Oliva.
“Learning deep features for scene recognition using places database”. In
NIPS 2014.

[2] D. Yoo, S. Park, J. Lee and I. Kweon. “Multi-scale pyramid
pooling for deep convolutional representation”. In CVPR Workshop 2015.

Inha University, South Korea

Our method is mainly based on Fast RCNN framework [1]. The region
proposal algorithm EdgeBoxes [2] is employed to generate region of
interests from an image, and features are generated using 16 layer CNN
network [3] which pre-trained on the ILSVRC 2013 CLS dataset and
fine-tuned on the detection dataset. We perform the final decision of
object localization using the hierarchical ridge regression and the
extended bounding box similar to [5], and the weighted non-maximum
suppression similarly to [1, 4].

[1] R. Girshick. Fast R-CNN. In CoRR, abs/1504.08083, 2015.

[2] C. L. Zitnick and P. Dollár. Edge boxes: Locating object proposals from edges. In ECCV, pages 391–405, 2014.

[3] K. Simonyan and A. Zisserman. Very deep convolutional networks
for large-scale image recognition. In CoRR, abs/1409.1556, 2014.

[4] S. Gidaris and N. Komodakis. Object detection via a multiregion
& semantic segmentation-aware cnn model. In arXiv:1505.01749, 2015.

[5] Y. Zhu, R. Urtasun, R. Salakhutdinov, and S. Fidler. segDeepM:
Exploiting segmentation and context in deep neural networks for object
detection. In CVPR, 2015.

[6] D. M. Blei, A. Y. Ng, M. I. Jordan. Latent dirichlet allocation. In JMLR, pages 993-1022, 2003.

[1] R. Girshick. Fast R-CNN. In CoRR, abs/1504.08083, 2015.

[2] C. L. Zitnick, and P. Dollár. Edge boxes: Locating object proposals from edges. In ECCV, pages 391-405, 2014.

[3] K. Simonyan, and A. Zisserman. Very deep convolutional networks
for large-scale image recognition. In CoRR, abs/1409.1556, 2014.

[4] Q. Wang, F. Chen, W. Xu, and M.-H. Yang. Object tracking via
partial least squares analysis. IEEE Transactions on Image Processing,
pages 4454–4465, 2012.

[5] S. Khim, S. Hong, Y. Kim and P. Rhee. Adaptive visual tracking
using the prioritized Q-learning algorithm: MDP-based parameter learning
approach. Image and Vision Computing, pages 1090-1101, 2014.

[1] R. Girshick, Fast R-CNN, in Proceedings of the International Conference on Computer Vision (ICCV), 2015.

[2] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015

[3] J. Uijlings, K. van de Sande, T. Gevers, and A. Smeulders. Selective search for object recognition. IJCV, 2013.

[4] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov,
D. Erhan, V. Vanhoucke, and A. Rabinovich, Going Deeper with
Convolutions, Cvpr, 2015.

[5] S. Ioffe and C. Szegedy, Batch Normalization: Accelerating Deep
Network Training by Reducing Internal Covariate Shift, Arxiv, 2015.

[1] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott
Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew
Rabinovich, “Going deeper with convolutions,” in Proc. CVPR, 2015.

[2] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in Proc. ICLR, 2015.

[3] Sergey Ioffe and Christian Szegedy, “Batch normalization:
Accelerating deep network training by reducing internal covariate
shift,” in Proc. ICML, 2015.

[4] M.-M. Cheng, Z. Zhang, W.-Y. Lin, and P. H. S. Torr, “BING:
Binarized normed gradients for objectness estimation at 300fps,” in
Proc. CVPR, 2014.

Abstract:

In ILSVRC 2015 challenge, we (team MCG-ICT-CAS) participate in two
tasks: object classification/localization (CLS-LOC) and object detection
(DET) without using any outside images or annotations.

CLS-LOC: we do object classification and localization sequentially. So we will describe them respectively as follows.

For object classification, we use fusion of VGG [1] and GoogleNet
[2] models as our baseline. Although the top-5 accuracy of the GoogleNet
used by us which was released by Caffe platform [3] is 90.73 in the
validation dataset, much lower than that of the initial unreleased
GoogleNet by Google, we can still get the top-5 accuracy of 93.28 after
fusion with VGG (92.55). After further fusion with our own 2 models, we
can improve the accuracy to final 93.72. Our unique contributions are
three-fold:

(1) Sparse CNN model (SPCNN): In the early April of this year, we
propose to learn a compact CNN model named SPCNN for reducing
computation and memory cost. Since the 4K*1K (4M) number of connections
between the 7th and 8th layer is one of the densest connections between
the CNN layers, we focus on how to remove most of small connections
between them because small connections are often unstable and may cause
noise. Actually, for a given category (a neuron in the 8th layer), it is
naturally only related with a very few number of category bases
(neurons of the 7th layer), i.e., the former can be modelled as sparse
linear combinations of a very few number of the latter. Hence, we
propose to select only a very few number of connections with large
weights between a given category and category bases for retraining, and
set other small unstable weights as constant zero without retraining.
Experiments on the validation dataset show that only keeping and
retraining a very small percentage (about 9.12%) of all the initial 4M
connections can still gain 0.32% improvement of top-5 accuracy (92.87%)
compared with that (92.55%) of the initial VGG model after removing the
90.88% small unstable connections.

(2) Sparse Ensemble Learning (SEL): Large scale training dataset
imposes a great challenge for efficiency of both training and testing.
To overcome the low efficiency and unscalability of the classical
methods based on global classification such as SVM, we proposed SEL for
visual concept detection in [4]. It leverages sparse codes of image
features for both partition of large scale training set into small
localities for efficient training and coordination of the individual
classifiers in each locality for final classification. The sparsity
ensures that only a small number of complementary classifiers in the
ensemble will fire on a testing sample, which not only gives better AP
than other fusion methods like average fusion and Adaboost, but also
allows the high efficiency of online detection. In this year’s object
classification task, we use the 4K-dim CNN features as the image
features instead of traditional bag of words (BoW). After expanding the
training dataset from 1.2 million samples to about 6 million samples
with through data argumentation, we use 8K-dim sparse codes of CNN
features to partition the training dataset into 8K number of small
localities for efficient training and use the sparse codes of test
sample for fusion. Experiments on the validation dataset show that
compared with CaffeNet, SEL with CNN features can improve 3.6% and 1.0%
in top-1 and top-5 accuracies respectively.

(3) Additionally, to compare with the widely used average fusion
(AVG) method, we try ordered weighted averaging (OWA) [5] in terms of
top-5 accuracy for fusion of all the models including GoogleNet, VGG,
SPCNN and SEL. Experiments on the validation dataset show that OWA is
better than average fusion.

For object localization, we mainly focus on the following three aspects:

(1) We apply the framework of Fast R-CNN [6] into object
localization task. Instead of using Non-Maximum-Suppression (NMS)
filtering of overlapped detection proposals of standard Fast R-CNN, we
propose to fuse the most probable detection proposals with top-N (N=10)
scores to get the final localization result.

(2) In order to get more confident proposals, we combine three
different region proposals including Selective Search (SS) [7], EdgeBox
(EB) [8], Region Proposal Network (RPN) [9] for both training and
localization of objects.

(3) We try clustering-based object localization to get more positive
training samples in each individual clustering subset besides the
complexity reduction of both training and localization.

After the above measures, we can improve our localization accuracy
from 83.2% (baseline) to around 84.9% in the validation dataset.

DET: An overwhelming majority of existing object detection methods
have focused on how to reduce the number of region proposals while
keeping high object recall without consideration of category
information, which may lead to a lot of false positives due to the
interferences between categories especially when the number of
categories is very large. To eliminate such interferences, in this
year’s DET task, we propose a novel category aggregation among region
proposals based upon our careful observation that more frequently
detected categories around an object have the higher probabilities to be
present in the image. After further exploiting the co-occurrences
relationship between categories, we can determine the most possible
categories for an image in advance. Thus, many false positives of region
proposals can be greatly filtered out before subsequent classification
process. Our experiments on the validation dataset verified the
effectiveness of our proposed both category aggregation and
co-occurrence refinement approaches.

References:

[1] Karen Simonyan, Andrew Zisserman,“Very Deep Convolutional
Networks for Large-Scale Image Recognition”, CoRR abs/1409.1556 (2014)

[2] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott
Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew
Rabinovich, “Going deeper with convolutions”, CVPR 2015.

[3] http://caffe.berkeleyvision.org/model_zoo.html

[4] Sheng Tang, Yan-Tao Zheng, Yu Wang, Tat-Seng Chua, “Sparse
Ensemble Learning for Concept Detection”, IEEE Transactions on
Multimedia, 14 (1): 43-54, February 2012.

[5] Ronald R. Yager, “On ordered weighted averaging aggregation
operators in multicriteria decisionmaking”, IEEE Transactions on
Systems, Man, and Cybernetics 18(1):183-190 (1988).

[6] R. Girshick, “Fast R-CNN”, In Proceedings of the International Conference on Computer Vision (ICCV), 2015.

[7] J. R. R. Uijlings, K. E. A. van de Sande, T. Gevers, and A. W.
M. Smeulders, “Selective search for object recognition”, International
Journal of Computer Vision, 104(2):154-171, 2013.

[8] C. L. Zitnick and P. Dollar, “Edge boxes: Locating object
proposals from edges”, In Computer Vision–ECCV 2014, pages 391-405.
Springer, 2014.

[9] S. Ren, K. He, R. Girshick, and J. Sun. , “Faster R-CNN Towards
real-time object detection with region proposal networks”, In Neural
Information Processing Systems (NIPS), 2015.

Before we train models using the Fast-RCNN framework, we retrain
VGG-16[2] with object-level annotations from CLS/LOC and DET data as
with [3]. We initialize two models with original VGG-16 and other two
models with retrained one.

For all models, we concatenate the whole image features extracted by
CNN with the fc7-layer output and use it as the input to the inner
product layer before Softmax, while we use original fc7-layer output as
the input to the bounding box regressors. We multiply the whole image
features by a constant to make the norm of the whole image features
smaller compared to that of the fc7-layer output.

We replace pool5 layer of one of the models initialized by original
VGG-16 and pool5 layer of one of the models retrained on annotated
objects with RoI Pooling layers, and replace pool4 layers of the other
models with RoI Pooling layers and then train them on training dataset
and the val1 dataset (see [4]) using the Fast-RCNN framework.

During testing, we use object region proposals obtained from
Selective Search[5] and Multibox[6], and we test not only original
images but also horizontally-flipped ones and combine them.

In our experiments, replacing pool4 layer rather than pool5 layer
with RoI Pooling Layer improved mAP on the val2 dataset from 42.9% to
44.2%, and retraining the model with object-level annotation improved
mAP from 44.2% to 45.6%.

We submitted two results. One is obtained by model fusion using the
same weights for all models and the other is obtained by model fusion
using the weights learned separately for each class by Bayesian
optimization on the val2 dataset.

[1] Girshick, Ross. "Fast R-CNN." arXiv preprint arXiv:1504.08083 (2015).

[2] Simonyan, Karen, and Andrew Zisserman. "Very deep convolutional
networks for large-scale image recognition." arXiv preprint
arXiv:1409.1556 (2014).

[3] Ouyang, Wanli, et al. "Deepid-net: Deformable deep convolutional
neural networks for object detection." Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition. 2015.

[4] Girshick, Ross, et al. "Rich feature hierarchies for accurate
object detection and semantic segmentation." Computer Vision and Pattern
Recognition (CVPR), 2014 IEEE Conference on. IEEE, 2014.

[5] Uijlings, Jasper RR, et al. "Selective search for object
recognition." International journal of computer vision 104.2 (2013):
154-171.

[6] Szegedy, Christian, et al. "Scalable, high-quality object detection." arXiv preprint arXiv:1412.1441 (2014).

2- Berkin Malkoç / Miletos Inc - Istanbul Technical University - Department of Physics

3- Mustafa Can Uslu / Miletos Inc - Istanbul Technical University - Department of Physics

4- Onur Kaplan / Miletos Inc - Istanbul Technical University - Department of Electrical and Electronics Engineering

* We used GoogleNet architecture for Classification and
Localization. We also used Simple Linear Iterative Clustering for
Localization.

* For each ground truth label, we selected 3 different nodes based
upon WordNet tree structure, hierarchically. Also, we trained the model
from scratch with these selected three nodes, one for each output level
of the model.

* We tried two different schemes to train the GoogleNet
architecture. In first scheme, we separated the architecture into three
individual parts such that each part has one of the output layers of
GoogleNet architecture and we trained these parts separately. Later, we
reconstructed the original GoogleNet architecture from these parts and
fine-tuned it by using all of the three output layers.

In second scheme, we trained GoogleNet architecture with three output layers without separating them.

* For each image, we selected multiple crops that are more likely to
be possible objects with Simple Linear Iterative Clustering. Then, we
selected five of the crops (one for each Top-5 prediction) with
GoogleNet Classification model.

References :

1- Going Deeper With Convolutions - arXiv:1409.4842

Machine Intelligence and Pattern Analysis Lab,

Seoul National University, Korea

[1] C Szegedy et al, "Going Deeper with Convolutions"

[2] https://github.com/BVLC/caffe/tree/master/models/bvlc_googlenet

Machine Perception Group

Graduate School of Information Science and Technology

The University of Tokyo

Our localization and detection systems are based on deep residual
nets and the "Faster R-CNN" system in our NIPS paper [b]. The extremely
deep representations generalize well, and greatly improve the results of
the Faster R-CNN system. Furthermore, we show that the region proposal
network (RPN) in [b] is a generic framework and performs excellent for
localization.

We only use the ImageNet main competition data. We do not use the Scene/VID data.

The details will be disclosed in a later technical report of [a].

[a] "Deep Residual Learning for Image Recognition", Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun. Tech Report 2015.

[b] "Faster R-CNN: Towards Real-Time Object Detection with Region
Proposal Networks", Shaoqing Ren, Kaiming He, Ross Girshick, Jian Sun.
NIPS 2015.

[1] Ren, He, Girshick, Sun. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. ArXiv 2015.

IMG Lab, National Electronics and Computer Technology Center, Thailand

@article{jia2014caffe,

Author = {Jia, Yangqing and Shelhamer, Evan and Donahue, Jeff and
Karayev, Sergey and Long, Jonathan and Girshick, Ross and Guadarrama,
Sergio and Darrell, Trevor},

Journal = {arXiv preprint arXiv:1408.5093},

Title = {Caffe: Convolutional Architecture for Fast Feature Embedding},

Year = {2014}

}

@incollection{NIPS2012_4824,

title = {ImageNet Classification with Deep Convolutional Neural Networks},

author = {Alex Krizhevsky and Sutskever, Ilya and Geoffrey E. Hinton},

booktitle = {Advances in Neural Information Processing Systems 25},

editor = {F. Pereira and C.J.C. Burges and L. Bottou and K.Q. Weinberger},

pages = {1097--1105},

year = {2012},

publisher = {Curran Associates, Inc.},

url = {http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf}

}

[1]Sergey Ioffe, Christian Szegedy. Batch Normalization:
Accelerating Deep Network Training by Reducing Internal Covariate Shift.
arXiv preprint arXiv:1502.03167

[2]Ren Wu 1 , Shengen Yan, Yi Shan, Qingqing Dang, Gang Sun.Deep
Image: Scaling up Image Recognition. arXiv preprint arXiv:1501.02876

[3]Andrew G. Howard. Some Improvements on Deep Convolutional Neural
Network Based Image Classification. arXiv preprint arXiv:1312.5402

[4]Karen Simonyan, Andrew Zisserman. Very Deep Convolutional
Networks for Large-Scale Image Recognition. arXiv preprint
arXiv:1409.1556

[5]Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao. Towards Good
Practices for Very Deep Two-Stream ConvNets. arXiv preprint
arXiv:1507.02159

[6]Zhenhua Wang, Xingxing Wang, Gang Wang. Learning Fine-grained
Features via a CNN Tree for Large-scale Classification. arXiv preprint

The network has been trained on Intel Parallel Computing Lab’s deep learning library

(PCL-DNN) and all the experiments were performed on 32-node Xeon E5 clusters. A network

of this size typically takes about 30 hrs for training on our deep learning framework.

Multiple experiments for fine-tuning were performed in parallel on NERSC’s Edison and

Cori clusters, as well as Intel’s Endeavor cluster.

[1] Very Deep Convolutional networks for Large-Scale Image Recognition. ICLR 2015.

Karen Simonyan and Andrew Zisserman.

[1] S. Ioffe & C. Szegedy. Batch Normalization: Accelerating
Deep Network Training by Reducing Internal Covariate Shift. In ICML
2015.

[2] K.E.A. van de Sande et al. Segmentation as Selective Search for Object Recognition. In ICCV 2011

[3] R. Girshick. Fast R-CNN. In ICCV 2015.

[1] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature
hierarchies for accurate object detection and semantic segmentation. In
CVPR, 2014.

[2] A. Papazoglou and V. Ferrari, Fast object segmentation in unconstrained video. In ICCV, 2013.

[1]Sergey Ioffe, Christian Szegedy, Batch Normalization:
Accelerating Deep Network Training by Reducing Internal Covariate Shift.
ICML 2015.

[2]Xin Li,Yuhong Guo, Latent Semantic Representation Learning for Scene Classification.ICML 2014.

[3]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,Delving
Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet
Classification. ICCV 2015

[4] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov,

Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich, Going deeper with convolutions. CVPR 2015.

Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences

(i) Multi-scale CNNs: we utilize Inception2 architecture as our main
exploration network structure due to its performance and efficiency. We
propose a multi-scale CNN framework, where we train CNNs from image
patches of two resolutions (224 cropped from 256, and 336 cropped from
384). For CNN at low resolution, we use the same network as Inception2.
For CNN at high resolution, we design a deeper network based on
Inception2.

(ii) Handing Label Ambiguity: As the scene labels are not mutually
exclusive with each other and some categories are easily confused, we
propose two methods to handle this problem. First, according to the
confusion matrix on the validation dataset, we merge some scene
categories into one single super-category. Second, we utilize the
Places205 scene model to test images and use the soft output as another
target to guide the training of CNN.

(iii) Better Optimize CNN: We use a large batch size to train CNN
(1024). Meanwhile, we try to set the decrease of learning rate in the
exponential form. Moreover, we design a locally-supervised learning
method to learn the weight of CNNs.

(iv) Combing CNNs of different architectures: considering the
complementarity of networks with different architectures, we also fuse
the prediction results of networks: VGGNet13, VGGNet16, VGGNet19 and
MSRANet-B.

[1] A. Krizhevsky, I. Sutskever, and G. E. Hinton. ImageNet
classification with deep convolutional neural networks. In NIPS, 2012.

[2] S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep
network training by reducing internal covariate shift. CoRR,
abs/1502.03167, 2015.

[3] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1556, 2014.

[4] K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into
rectifiers: Surpassing human-level performance on imagenet
classification. CoRR, abs/1502.01852, 2015.

[5] Hinton, Geoffrey, Vinyals, Oriol, and Dean, Jeff. Distilling the
knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.

[6] L. Wang, Y. Xiong, Z. Wang, and Y. Qiao. Towards good practices
for very deep two-stream convnets. CoRR, abs/1507.02159, 2015.

[1] Ross Girshick, Fast R-CNN, In ICCV 2015

[1]Xudong Cao. A practical theory for designing very deep convolutional neural networks, 2014. (unpublished)

[2]Ben Graham. Sparse 3D convolutional neural networks, BMVC 2015

[3]Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott
Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew
Rabinovich. Going Deeper with Convolutions, CVPR 2015

[4] D.Mishkin, J. Matas. All you need is a good init, arXiv:1511.06422.

Given a lot of box proposals(eg. from selective search[5], about
1500 boxes per image), the localization task become (a) how to find real
boxes that contain ground truth, (b) what is the category lies in the
box. Enlightened by fast R-CNN[4], We propose a classification
&& localization framework, which combines the global context
information and local box information by selecting proper (class, box)
pairs: first we pre train a googlenet for classification; and then two
kinds of transformations have been made to the network, obtaining
objectness and local category within the box; and finally, a SVM
classifier is applied to select proper (class, box) pairs.

1. Fine-tuning for Objectness

Similar to fast R-CNN, we train a network to indicate the
"objectness" of box. The probability of objectness is computed by a
two-class softmax layer over fully-connected layer. A box proposal is
positive if it has intersection over union (IoU) at least 0.7, negative
if IoU<0.3. No 1000 categories information is used at this stage.
Another Box offsets regression layer is also jointly trained with
objectness, and ignored when the box is negative, as is standard
practice in[4].

At test time, the objectness of all box proposals are verified by
the network. Then we perform non-maximum suppression and keep most
objectness boxes.

2. Fine-tuning for Local Classification

We acquire the global classification by avarging result of multi
crops(eg. 144crops per image[1]) using the pre-trained googlenet model.
On the other hand, we finetune the googlenet to indicate the 1000
categories within the box, which we call "local classification". The
training sets are sampled around ground truth and resized to the network
input.

At test time, we clip images from the regression boundary of the
most objectness boxes, and put them into local classification network.
Classes with high probability from local and global will be retaining.

3. Combine Information and Pair Select

So far, we have got objectness and offsets regression for some
boxes, classification results for both local and global. For each
possible pair (class, box), all these information form a feature vector,
and is trained by SVM.

At test time, the pairs that reach top5 confidence in SVM would become our final result.

It may be surprising that the simple strategy can be more accurate
even without models ensemble. We believe that the object boundary has
something to do with low level vision, such as corner, edge, shape. As
the network goes deeper, the features become more and more abstract. It
is questionable to train a box regression while retaining 1000
categories information, especially when there are only 500 images per
category on average. Our solution avoid seeking box directly, but take
full advantage of classification capability of deep network.

We train our network models using modified cxxnet[6].

[1]Ioffe S, Szegedy C. Batch Normalization: Accelerating Deep
Network Training by Reducing Internal Covariate Shift. In JMLR, 2015.

[2]Simonyan K, Zisserman A. VGG Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.

[3]Sermanet P, Eigen D, Zhang X, et al. Overfeat: Integrated
recognition, localization and detection using convolutional networks. In
ICLR, 2014.

[4]Girshick R. Fast R-CNN. In NIPS, 2015.

[5]Uijlings J R R, van de Sande K E A, Gevers T, et al. Selective
search for object recognition. International journal of computer vision,
2013, 104(2): 154-171.

[6]cxxnet: https://github.com/dmlc/cxxnet

We trained three CNNs on the Places2 datasets using the GoogleNet
[1]. Specifically, Caffe [2] is used for training. Among the three
models, the first model is trained on by fine-tuning the GoogleNet model
trained on Places dataset [3]. The classification accuracy of this
model is: top-1 accuracy = 42.96%, top-5 accuracy = 75.35%. This model
is obtained after 3,320,000 mini-batches, and the batch size is 32.
Learning rate is set to 0.001, and gamma is set to 0.1, with a step size
of 750,000.

The second and the third models are trained using the "quick" and
the default solvers provided by GoogleNet, respectively, and both models
are trained from scratch. Specifically, for the second model, we use a
base learning rate of 0.01, gamma = 0.7, and step size = 320,000. We run
this model for 4,500,000 mini-batches, and each mini-batch is of size
32. For this model, our result on the validation set is: top-1 accuracy =
43.41%, top-5 accuracy = 75.37%. In more detail, we only change the
architecture of GoogleNet to have 401 blobs in the last fully connected
layer. The average operation is done after the softmax calculation.

For submission, we submit results of each model as the first three
runs (run 1, run 2, and run 3). Then, run 4 is the averaged result of
fine-tuned GoogleNet + quick GoogleNet. Run 5 is the averaged result of
all three models.

[1] Szegedy, Christian, et al. "Going deeper with convolutions." arXiv preprint arXiv:1409.4842 (2014).

[2] Jia, Yangqing, et al. "Caffe: Convolutional architecture for
fast feature embedding." Proceedings of the ACM International Conference
on Multimedia. ACM, 2014.

[3] Zhou, Bolei, et al. "Learning deep features for scene
recognition using places database." Advances in Neural Information
Processing Systems. 2014.

[4] Places2: A Large-Scale Database for Scene Understanding. B.
Zhou, A. Khosla, A. Lapedriza, A. Torralba and A. Oliva, Arxiv, 2015

(1)The Third Research Institute of the Ministry of Public Security, China

(2)Fudan university

(3)New York University Shanghai

(4)Huazhong University of Science & Technology

(The Third Research Institute of the Ministry of Public Security, P.R. China.)

Object localization:

Different data augmentation methods were used, including random
crops, multiple scales, contrast and color jittering. Some models were
trained by maintaining the aspect ratio of input images, while others
were not. In the test phase, whole uncropped images are densely
processed for various scales. Further generate the fusion classification
result according to the scores and labels jointly. On the localization
side, we refer to the framework of Fast R-CNN. An iterative scheme like
detection task is used. Then select top-k regions and averaging their
coordinates as output. Results from multiple models are fused in
different ways, using the model accuracy as weights.

Object detection from video:

We use same models as object detection task. Part of these models
were fine-tuned using VID data. We also try main object constraint by
considering whole snippet rather than single frame.

Scene classification:

Based on both MSRA-net and BN-GoogLeNet, and plus several
improvements: 1) Choose subset data in a stochastic way at each epoch,
which ensures each class has roughly equal images. This can both
accelerate training and increase model diversity. 2) To utilize the
whole image and part object information simultaneously, three different
size patches of image (whole image with 224x224, crop of 160x160, and
crop of 112x112) were feed into network and concatenate at the last
convolution layer. 3) Enlarge the MSRA-net to 25 layers, and change some
BN-net input from 224x224 to 270x270. 4) Use dense sample and
multi-crop (50x3) for testing.

* - equal contribution

First, we find image regions that may contain objects via bounding
boxes (i.e. proposals) using the fully convolutional region-proposal
network (RPN) of [1]. The network’s topology is similar to the
Zeiler-Fergus (ZF) network [2]. Once the RPN has been trained, we train
another neural network (RCNN) [1] that tries to classify the object in a
given proposal. For this phase, we use a VGG16-style [3] network that
was pre-trained on the ImageNet Classification and Localization Data
(CLS) and only fine-tune the last fully-connected layer. Instead of
trying to classify 200 objects, the layer has been altered to classify a
proposal as being one of 30 classes. Both the RPN and RCNN are trained
using the initial training release.

During testing, we pass an image to the RPN which extracts 500
proposals. We then apply the RCNN to each proposal and extract 30
confidence scores and 30 refined bounding boxes. We then consider one of
two possible post-processing algorithms: (i) NMS on each frame (ii)
sequence-NMS (SEQ-NMS) on each video snippet.

Regular frame-wise NMS operates on a single frame by iteratively
selecting a class’ most confident detection in the frame and removes
detections in the vicinity that have sufficient overlap. SEQ-NMS, on the
other hand, operates on each video snippet. It iteratively selects the
sequence of boxes over time that has maximum score and then suppresses
detections that overlap with any of the members in the selected
sequence. We accomplish this by constructing a graph over the snippet’s
frames and doing dynamic programming. We score each sequence using
either the average (NMS-AVG) or the max score (NMS-MAX) of the boxes. We
then re-score each box in the kept sequence with the overall sequence
score. One of our models selects whether to score using the average or
the max for each class depending on each method's performance on the
initial validation set.

References

[1] – Ren, Shaoqing, Kaiming He, Ross Girshick, and Jian Sun.
"Faster r-cnn: Towards real-time object detection with region proposal
networks." arXiv preprint arXiv:1506.01497 (2015).

[2] - Zeiler, Matthew D., and Rob Fergus. "Visualizing and
understanding convolutional networks." In Computer Vision–ECCV 2014, pp.
818-833. Springer International Publishing, 2014.

[3] - Simonyan, Karen, and Andrew Zisserman. "Very deep
convolutional networks for large-scale image recognition." arXiv
preprint arXiv:1409.1556 (2014).

The idea of filter map is inspired by epitome [1], which is
developed in the computer vision and machine learning literature for
learning a condensed version of Gaussian Mixture Models (GMMs). In
epitome, the Gaussian means are represented by a two dimensional matrix
wherein each window in this matrix contains parameters of the Gaussian
means for a Gaussian component. The same structure is adopted for
representing the Gaussian covariances. With almost the same parameter
space as GMMs, the epitome possesses significantly more number of
Gaussian components than its GMMs counterpart since much more Gaussian
means and covariances can be extracted densely from the mean and
covariances matrices of the epitome. Therefore, the generalization and
representation capability of epitome outshines GMMs with almost the same
parameter space, while circumventing the potential overfitting.

The above characteristics of epitome encourage us to arrange filters
in a way similar to epitome in the FPCNNs. More precisely, we construct
a three dimensional matrix named filter panorama for the convolutional
layer of FPCNNs, wherein each window of the filter map plays the same
role as the filter in the convolutional layer of CNNs. The filter
panorama is designed such that the number of non-overlapping windows in
the filter panorama is almost equal to the number of filters in the
corresponding convolutional layer of CNNs. By densely extracting
overlapping windows from the filter panorama, there are many more
filters in the filter panorama that vary smoothly in the spatial domain,
and neighboring filters share weights in their overlapping region.
These smoothly varying filters tend to activate on more types of
features that also exhibit small variations in the input volume,
increasing the chance of extracting more robust deformation invariant
features by the subsequent max-pooling layer [2].

In addition to the superior representation capability, filter
panorama inherently enables a better visualization of the filters by
forming a “panorama” of the filters such that adjacent filters changes
smoothly across the spatial domain, and similar filters group together.
This feature would benefit the design of the networks and make it easier
to observe the characteristics of the filters learnt in the
convolutional layer.

References:

[1] N. Jojic, B. J. Frey, and A. Kannan. Epitomic Analysis of
Appearance and Shape. In Pro-ceedings of IEEE International Conference
on Computer Vision (ICCV), pp. 34-43, 2003.

[2] K. Jarrett, K. Kavukcuoglu, M. Ranzato, and Y. LeCun. What is
the best multi-stage archi-tecture for object recognition? In Proc.
ICCV, pages 2146–2153, 2009. 1

For the classification model which predicts the confidence score of
each object class in the viewing window, we ensembled four deep neural
networks two which are similar with [1](CNN) and two of which are their
variants. The variants models(CLSTM) replaces last two convolution
layers of [2] with 2D-LSTM [3] layer, which provides robustness to
object warping and contextual information.

For localization model which predicts the location of bounding box,
we used per-class-regression(PCR) approach[3] which replaces last 1000D
softmax layer with 4000D regression layer. Additionally, we also
replaced SPP layer of CNN and CLSTM with max-pooling layer. The results
obtained from two combinations of the models are submitted: CNN-CNN,
CNN-CLSTM.

For training, we used scale jittering strategy in [3] where the
scale is sampled from fixed set of scales. For testing, we used
multi-scale testing and merged all bounding boxes obtained from each
scale.

All experiments were performed using our own deep learning library called VunoNet on GPU server with 4 NVIDIA Titan X GPUs.

[1] He, Kaiming, Xiangyu Zhang Shaoqing and Ren Jian Sun ”Delving
Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet
Classification.” arXiv preprint arXiv:1502.01852 (2015).

[2] Alex Graves, Santiago Fernandez and Jurgen Schmidhuber.
Multidimensional recurrent neural networks. In ´ Proceedings of the 2007
International Conference on Artificial Neural Networks, Porto,
Portugal, September (2007).

[3] Simonyan Karen, and Andrew Zisserman. ”Very deep convolutional
networks for large-scale image recognition.” arXiv preprint
arXiv:1409.1556 (2014).

In this competition, we submit five entries. The first is a single
model (model A), which achieved 16.33% top-5 error on validation
dataset. The second is a single model (model B), which achieved 16.36%
top-5 error on validation dataset. The third is a combination of
multiple CNN models with the averaging strategy. The fourth is the
combination of these CNN models with a product strategy. The fifth is
the combination of multiple CNN models with learnt weights.

[1] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR 2015

[2] K. He, X. Zhang, S. Ren and J. Sun. Spatial pyramid pooling in
deep convolutional networks for visual recognition. In ECCV 2014.

References:

[1] M. Jain, J.C. van Gemert, T. Mensink, and C.G.M. Snoek.
Objects2action: Classifying and localizing actions without any video
example. In ICCV, 2015.

[2] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov,
D. Erhan, V. Vanhoucke, A. Rabinovich. Going Deeper with Convolutions.
CVPR, 2015.

[3] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean.
Distributed representations of words and phrases and their
compositionality. In NIPS, 2013.

[4] B. Thomee, D. A. Shamma, G. Friedland, B. Elizalde, K. Ni, D.
Poland, D. Borth, L.-J. Li. The New Data and New Challenges in
Multimedia Research. arXiv:1503.01817, 2015.