Exploiting Invariance in Training Deep Neural Networks

by   Chengxi Ye, et al.

Inspired by two basic mechanisms in animal visual systems, we introduce a feature transform technique that imposes invariance properties in the training of deep neural networks. The resulting algorithm requires less parameter tuning, trains well with an initial learning rate 1.0, and easily generalizes to different tasks. We enforce scale invariance with local statistics in the data to align similar samples generated in diverse situations. To accelerate convergence, we enforce a GL(n)-invariance property with global statistics extracted from a batch that the gradient descent solution should remain invariant under basis change. Tested on ImageNet, MS COCO, and Cityscapes datasets, our proposed technique requires fewer iterations to train, surpasses all baselines by a large margin, seamlessly works on both small and large batch size training, and applies to different computer vision tasks of image classification, object detection, and semantic segmentation.



page 1

page 2

page 3

page 4


AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks

Training deep neural networks with Stochastic Gradient Descent, or its v...

Neither Quick Nor Proper -- Evaluation of QuickProp for Learning Deep Neural Networks

Neural networks and especially convolutional neural networks are of grea...

Large Batch Training Does Not Need Warmup

Training deep neural networks using a large batch size has shown promisi...

Robust Differentially Private Training of Deep Neural Networks

Differentially private stochastic gradient descent (DPSGD) is a variatio...

Towards Stabilizing Batch Statistics in Backward Propagation of Batch Normalization

Batch Normalization (BN) is one of the most widely used techniques in De...

Spherical Motion Dynamics of Deep Neural Networks with Batch Normalization and Weight Decay

We comprehensively reveal the learning dynamics of deep neural networks ...

Accelerating Natural Gradient with Higher-Order Invariance

An appealing property of the natural gradient is that it is invariant to...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.