Memory-efficient training with streaming dimensionality reduction

04/25/2020
by   Siyuan Huang, et al.
52

The movement of large quantities of data during the training of a Deep Neural Network presents immense challenges for machine learning workloads. To minimize this overhead, especially on the movement and calculation of gradient information, we introduce streaming batch principal component analysis as an update algorithm. Streaming batch principal component analysis uses stochastic power iterations to generate a stochastic k-rank approximation of the network gradient. We demonstrate that the low rank updates produced by streaming batch principal component analysis can effectively train convolutional neural networks on a variety of common datasets, with performance comparable to standard mini batch gradient descent. These results can lead to both improvements in the design of application specific integrated circuits for deep learning and in the speed of synchronization of machine learning models trained with data parallelism.

READ FULL TEXT

page 3

page 5

research
08/03/2022

Robust PCA for Anomaly Detection and Data Imputation in Seasonal Time Series

We propose a robust principal component analysis (RPCA) framework to rec...
research
09/25/2018

Graph filtering for data reduction and reconstruction

A novel approach is put forth that utilizes data similarity, quantified ...
research
03/05/2019

Streaming Batch Eigenupdates for Hardware Neuromorphic Networks

Neuromorphic networks based on nanodevices, such as metal oxide memristo...
research
10/24/2021

Micro Batch Streaming: Allowing the Training of DNN models Using a large batch size on Small Memory Systems

The size of the deep learning models has greatly increased over the past...
research
01/04/2020

Distributed Stochastic Algorithms for High-rate Streaming Principal Component Analysis

This paper considers the problem of estimating the principal eigenvector...
research
07/28/2022

One-Pass Learning via Bridging Orthogonal Gradient Descent and Recursive Least-Squares

While deep neural networks are capable of achieving state-of-the-art per...
research
07/06/2017

High-Performance FPGA Implementation of Equivariant Adaptive Separation via Independence Algorithm for Independent Component Analysis

Independent Component Analysis (ICA) is a dimensionality reduction techn...

Please sign up or login with your details

Forgot password? Click here to reset