You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Network In Network, ICLR, 2014
Details
This paper proposes replacing feature maps with neural networks. The defined structure mlpconv, is an MLP which takes local patch as input and gives an output feature vector. The MLP operator is equivalent to the conventional 1x1 convolutional layer. Secondly the paper replaces the fully connected layers in the end of the network which global average pooling layers. The fully connected layers have a lot of parameters which need to be learned and they can easily overfit thus being less dependent on various regularisation tricks.
Striving for Simplicity: The All Convolutional Net, ICLR, 2014
Details
This paper analyses the basic design pipeline of CNN and proposes to replace the pooling layers with a convolutional layer with a non unit stride. The pooling layer can be seen as a convolutional layer with a p-norm operator being applied to each element instead of linear dot product.
Deep Learning Using Linear Support Vector MachinesDetails
The paper proposes use of svm layer instead of softmax layer in the neural network architectures. They use L2-SVM loss function to train the network.
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , ICML, 2015
Details
The paper proposes a method to accelerate the training of neural networks. It observes the fact that fact keeping input distribution same accelerates training of the system , also applies to the inputs of hiden layers. Changing input distribution to hidden layers makes training of hidden difficult. So batch normalisation layer is proposed which normalizes each scalar output in a minibatch independently of all other features values. Each batch normalisation layer has two parameters vectors $ \gamma , \beta $ which scale and shift the normalised value respectively.
During training, the mean and variance vectors are stored for determining the value of mean and variance during test time.
The layer is applied before the non-linearity layer. The batch norm lauer enables higher learning rates and better performance. For CNN, the mean and variance is computed over whole feature map output.
LSTM: A search space odyssey ,IEEE-TNNLS, 2016
Details
8 variations of LSTM are analysed to determine desirable properties in design of LSTM. The variations were evaluated on three tasks, acoustic modelling(TIMIT dataset), handwriting recognition(IAM Online) and polyphonic music modelling(JSB Chorales). Network with single hidden layer with sigmoid was used for music modelling task. Bi-RNN was used for other two tasks.The experiments yielded following results:-
Removal of forget gate or output activation function hurts performance.
Coupling input and forget gates and removing peephole connection didn't lead to degradation of performance.
Full gate recurrence didnt result in performance gains given the increase in the number of trainable parameters.
The imapct of hyperparameters was also studied. High and low learning rates lead to sub-optimal results. So a recommended strategy is to start learning rate with 1 and reduce it by 10 until final performance stop improving. Increasing number of hidden layers led to better results but it lead to substantial increase in training times. So there is a tradeoff and needs to be balanced carefully. Momentum in gradient updates didnt affect network performance and neither reduced the training time. Moreover, hyperparameters do not interact with each other . Changing two or more together didnt augment the affect of each other's change. Therefore, one can tune them separately and the resulting parameters would give a decent performance.
Dropout: a simple way to prevent neural networks from overfitting, JMLR, 2014
Details
A regularisation technique Dropout is introduced. Dropout samples networks from a given network by turning off hidden nodes with a probability p. At each training step, a network is sampled and trained with gradient updating parameters of each which were sampled. During test time, whole network is retained with each node's output being multiplied with p.
The parameters of all sampled networks are shared. This sampling procedure help to reduce overfitting which is caused by hidden nodes co-adapting to each other. In the conventional setup, the nodes try to correct errors made by other nodes thereby inducing strong corelations. But these corelations might not be present in unseen data, thus leading to overfitting. Dropout tries to make each node learn independently. Dropout has led to improvement over a whole class of problems in vision, speech, text etc. Dropout alongwith Max normalisation has worked best in many settings. The value of p is found by validation accuracy, but mostly it works best in range [0.4,0.8]. Dropout can be extended to multiplying gaussian noise ~ N(1,1) to activations instead of bernoulli random variable. One of the major drawbacks of Dropout is that it increases training time by 2-3 times.
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , ICCV, 2015
Details
A variation of relu activation function is introduced, namely Parameteric Relu, which instead of having zero output for negative values, output a non-zero value. The functions take the form f(x) = max(0,x) + a*min(0,x). The parameter is learned while training.
The paper also explores into the initialisation method and improves upon xaviers initialisation method. It concentrates on the fact , that non-linearities would affect the distribution of input to layers and analyses, which is skipped by xaviers method and the std deviation of distribution according to which weights are initliased is estimated by keeping the above mentioned fact in mind. They observe that the network converges faster with their estimates of std dev and the initialisation helps them to train deeper networks.
Maxout Networks, ICML, 2013
Details
A new activation function extending relu is proposed, namely maxout. A maxout is defined as hi(x) = max(zij), where zij = xWij + bij. The W and b are learned while training. Effectively, a maxout models a piecewise linear function. The maxout model with two hidden units arbritary number of linear components is universal approximator. Maxout networks are designed to work well with dropout regularisation. The guiding principle is to reduce the approximation error of dropout while averaging the output of neural network. Dropout does exact averaging in deep architectures when the layers are linear in the input space generated by applying various masks. Maxout units tend to have have linear outputs in regions of inputs generated by various masks.
Apart from the performance benefits, the maxout networks with dropout are easier to train, with networks error blowing slowing with increase in layers than networks with pooled rectifiers. Moreover, maxout networks do not face dying gradients probelem, as there is always a gradient flowing back in contrast to relus which can die.
Net2net: Accelerating learning via knowledge transfer, ICLR, 2016
Details
This paper explores transfer learning for neural networks. The paper designs a function preserving transformation of a smaller neural network to a larger neural network. The basic motivation of the paper is to increase the capacity of neural models on increase in availability of data without going through the tedious training process of the whole network. Advantages of function preserving transformation :-
Largeer network has at least same performance as the small network.
All the parameters are trainable, in contrast to some approaches such as cascade correlation.
Two tranformations are proposed :-
Net2WiderNet :- Addition of more units in a dense layer or more filters in conv layer. The weights of larger network are as follows :-
where g is the mapping function which maps columns of smaller network weight matrix to columns of large one., where n in the number of columns in smaller network's weight matrix. The weighting is done is next layer parameters to preserve function of the smaller network in large network.For conv layers , the above math would work per channel instead of per unit.
Net2DeeperNet :- Addition of layers to the network to make the network deep. So the layer hi = f(h(i-1)*Wi) is replaced with two layers hi = f(Ui*f(h(i-1)*Wi)) where f is a non-linearity. U is an identity matrix. All activations can't be used for function preserving transformation. Relu and maxout can be used , but sigmoid, tanh cannot be used since the tranformation doesnot preserve the outputs. For batch normalisation layer, some changes are required in U to undo the changes of normalisation.
The experiments revealed that training expanded nets are cheaper to train than training the nets for starting. Another major observation was that there was no loss of accuracy when comparing expanded nets and network trained from start.
Reinforcement Learning
Neural Architecture Search with Reinforcement LearningDetails
The paper focuses on problem of generating state of the art neural network architectures for various problems.RNN is used for predicitng the structure of neural network. The RNN is trained with re-inforcement learning to maximise the expected accuracy of the generated network.
The network predicts one paramter of network at a time. For eg, for a convolutional layer, the parameters like filter width and height, stride width and height, no. of filters are predicted one at a time with the previous predicted parameters being input to the RNN for next parameter prediction.
The list of parameters predicted is analogous to actions in a re-inforcement learning problem framework and the accuracy of the generated network as the reward. Policy gradient method is used to update parameters of RNN.
The framework was used to generate networks for both vision and language tasks. Various structures like cell structure for recurrent neural networks, skip connections in cnns and rnns are predicted by the framework. The cell architecture for rnn found is given in tensorflow contrib module as NASCell. The networks generated perform very well on CIFAR-10 and PTB(Penn Treebank) language modelling task.
Generative Models for Vision
Generative adversarial nets, NIPS, 2014 [GAN]
Details
The paper lays down a new framework of generative models. The basic idea is based on adversarial method. The framework has two parts, a discriminator and a generator. The parts compete against each other. The discriminator takes in an input and tries to determine whether the input comes from the data distribution or the distribution learned by the generator. Generator tries to learn the data distribution by trying to fool the discriminator. Generator takes noise variables as input and models the noise into images by using a multilayer perceptron. The discriminator takes image as input and outputs a scalar denoting the probability that the input is from data distribution. Discriminator is trained to maximise the probability of assigning correct labels to both training examples and examples generated from generator. Generator is trained to maximise, discriminator's error on generated samples. This is a formulation of minimax game with the following objective function;-
minGmaxD O(D,G) = Ex~p(data)[log(D(x))] + Ex~p(x)[log(1-D(G(x)))]
where D, G are discriminator and generator functions resp. and p(x) denotes the prior distribution over the noise variables.
This game is implemented in a loop by optimising D for some number(k) of iterations and one iteration of optimisation for generator. Ideally, D should be optimised completely before optimising G, but this is computationally prohibitive.
ImageNet Competition Winners
Imagenet classification with deep convolutional neural networks, NIPS,2014, [AlexNet]
Details
This paper describes the 2012 ILSVRC-2012 competition winner which performly very well as compared to other entries(15.3% vs 26.2% 2nd bestentry test error). The network was 8 layer deep with 5 layers of convolutional layers and 3 fully connected layers. They used relu non-linearity. Local normalisation was used which normalizes each activation value wrt to all values at the same spatial position but different channels. This work has been superseded by many better architectures , for eg. VGGNet, GoogLeNet, Resnet etc.
Visualizing and Understanding Convolutional Networks ECCV, 2014 [ZfNet]
Details
A visualisation techinique is introduced to study the layers of deep cnn. A deconvolutional operation is used to to map filter maps to input space.The network architecture is kept same as the one being studied. The deconvnet layers use the transposed version of the featured maps.Unpooling operation records the max elements chosen during maxpool and performs accordingly during backward step. Relu units have no change. The paper gives insights on the type of feaures learnt by cnns. The study was performed on alexnet and some changes were made to it based on observation of different layers , which performed better than alexnet on imagenet task.
Very Deep Convolutional Networks for Large-Scale Image Recognition, ICLR, 2015 [VGGNet]
Details
It explores the effect of depth on accuracy of image recognition task. They experiment with networks of depth upto 19 with convolutional filters of size 3x3. The network performed well in imagenet challenge and secured 1st and 2nd position in localisation and clasification tasks respectively.
Going Deeper with Convolutions, CVPR, 2015 [GoogLeNet]
Details
They introduced deep neural network architecture with 12x less parameters than AlexNet. Inception module is introduced whcih is stacked one above the other to give GoogLeNet. The main idea behind inception module is to cluster inputs with high correlation together. Each module has a set of 1x1, 3x3 and 5x5 convolutional filters to account for various scales of cluster over image. 1x1 filters carry forward the features from local regions and 3x3 and 5x5 tend to find spatially distributed features. All the extracted features are concatenated and passed on to above layer for processing. To avoid computational blowups due to ever increasing number of channels in input to each layer, 1x1 convolutional filters are used to reduce number of channels in output and in input to 3x3 and 5x5 convolutional operator.
Deep Residual Learning for Image Recognition, CVPR, 2016 [ResNet]
Details
The paper proposes a network architecture to solve the problem of training of very deep networks.Shortcut connections are introduced in the networks which help network to easily learn identity mapping if extra layers are not required. In addition, the shortcut connections also facilitate gradient flow. Each module of layers learn a residual function F(x) .
Object Detection/Classification
Rich feature hierarchies for accurate object detection and semantic segmentation, CVPR, 2014
Details
The paper was one of the earliest works to demonstrate power of CNN's for object detection tasks. The main idea was to use a CNN to extract features from images which would then be used for classification and detection instead of traditional features like SIFT, HOG etc. and their derivatives.
The basic components of the detection framework are :-
Region proposals :- Selective Search algorithm.
CNN architecture :- AlexNet.
Classifier :- SVM
The training process has three stages :-
Firstly the CNN is trained over ImageNet dataset.
The CNN is fine tuned over pascal voc.
The individual SVM classifiers are trained.
A bounding box regressor is implemented to improve the bounding box localisation errors.
Fast R-CNN , ICCV, 2015
Details
Fast RCNN is an extension of RCNN for object detection.They propose several improvements to RCNN framework:-
Higher mAP than RCNN
The CNN feature extraction network and the classifier networks from RCNN have been merged into one network, and therefore the training process becomes single stage using a multi task loss.
Faster running time as compared to RCNN.
The fast rcnn framework overcomes the computational wastage of multiple cnn forward passes for each object proposal in rcnn. A single pass is made for each image and for each of the object proposal generated from selective search, corresponding part from feature map is selected and given to ROI pooling layer which then outputs a fixed size vectior irrespective of the size of input. The framework uses networks which are pre-trained on imagenet, and the last pooling layer of the pretrained network is replaced with a ROI pooling layer.
The last fully connected layer is replaced by two layers one for object classification and one for bounding box regression. The loss of the network is defined as the combination of these two layers. This network is trained on a dataset to fine tune its weights.
Faster R-CNN: Towards real-time object detection with region proposal networks, NIPS, 2015
Details
This paper proposes an improvement to Fast RCNN by introducing a network to propose object proposal. In Fast RCNN, while predicting the objects in the image, most of the time is taken by Selctive Search to propose object proposals. Faster RCNN does away with the requirement of an external object proposal method and introduces a RPN(region proposal network) to give candidate boxes for object localisation.
The RPN takes the 3 x 3 spatial window of convolution feature map as input, and produces a feature vector of 512 length(for VGG, varies for different architectures). This feature vector is fed to two fully connected layers for bounding box regression and classifcation(whether the proposal contains an object or not). For each spatial window location, a set of k object proposals is produced according to anchor boxes(denoting size and aspect ratio of bounding box).The RPN is trained using a multitask loss giving equal weight to classifcation(log loss) and regression(smooth L1) loss.
This network can be trained in various ways:-
Alternating Training :- Training RPN first and then Fast RCNN part in a loop.
Approximate Joint Training :- Training whole network at once combining the gradient of RPN and Fast RCNN for shared convolutional layer.
4-step alternating training:- The network is trained in four parts :-
Train RPN.
Train Fast RCNN by using proposals from RPN trained in the first step.
Fix shared conv layers between Fast RCNN and RPN and fine tune the layers specific to RPN.
Fine Fast RCNN layers keeping shared conv layers constant.
Tomasi, and Jianbo Shi. “Good Features to Track.”, CVPR, 1994
Details
The papers addresses three issues in image tracking in an image sequence.First they demonstrate that pure transaltional models are not measures to measure dissimilarity in features across frames. Affine image changes perform decently to measure dissimilarity. Secondly they propose a stable numerical method using Newton-Raphson minimization procedure to track features.Thirdly, they propose a feature selection criterion based on tracker's accuracy. This is an old paper and has been superseeded.
Mikolajczyk, Krystian, and Cordelia Schmid. “A Performance Evaluation of Local Descriptors.”, PAML, 2005
Details
This work analyses various descriptors and describes the performance wrt the region on interest. The evaluation criteria they used was recall and precision. They carried out tests for various image transformations. The descriptors they comapred include SIFT, spin images, complex filter, moment invariants etc.
Identify points of interests by using a difference of gaussian function.
Filter points based on location, scale and ratio of principal curvatures.
Assign orientation to each point based on direction of local image gradients.
Determine the features by computing image gradients relative to orientation and weigh them by using a gaussain function. Create a gradient histogram with each bar representing the magnitude of gradients in a particular direction.
Visual Tracking with Online Multiple Instance Learning, CVPR, 2009
Details
This paper proposes a method to track objects in an image sequence using MIL(Multiple instance learning) based object classifier. At each time step, a bag of image crops around the object is labelled as positive and passed to the classifier for training. Then a location within a radius s which has the highest probability of having the object is marked as the object location for the next time step.
Mean shift: a robust approach toward feature space analysisDetails
This paper gives an approach to analyse large dimensioanl feature spaces. The main idea used is of the density estimation using kernels. So basically, the features are projected into a d dimensional space and density function is defined over all the projected points. Mean shift procedure(basically gradient descent on density function) is used to find extrema's of the density estimate. This processed feature space is used for discontinuity preserving smoothing and segmentation problems.
Torr, Philip H. S., et al. “Struck: Structured Output Tracking with Kernels., ICCV, 2011
Details
The paper focuses on the problem of tracking objects by detection in am image sequence. The paper focused on coupling the object cllasifier loss function to the loss metric of the tracker. They propose using structured output SVM model which is learnt online. They also describe a budgeting mechanism for support vectors of SVMs as online learning SVM is prone to have an unbounded number of support vectors.
Mikolajczyk, Krystian, et al. “Tracking-Learning-Detection.”, PAML, 2012
Details
The paper tackles the problem of tracking objects in videos. It proposes a TLD(Tracking Learning Detection) approach. Their method has three components. First component, the tracker predicts the position of object in next frame assuming limited frame to frame motion. Second component, Detector takes input the image at time t and analyses it independently. Third component is a PN learning system which evaluate the performance of both tracker and detector and generates training examples to correct them. The P expert checks for false negatives and adds them to positive training set for the detector. The N expert checks for false positives and adds them to the negative training set. The PN expert system is designed in such a way such that the errors in PN system are complementary and they compensate errors of each other in generating training examples. The object classifer is a three stage with the last stage being nearest neighbour classifier.The tracker is based in Median-flow tracker extended with failure detection.
, where n in the number of columns in smaller network's weight matrix. The weighting is done is next layer parameters to preserve function of the smaller network in large network.For conv layers , the above math would work per channel instead of per unit.
The basic components of the detection framework are :-
