Low-Precision Stochastic Gradient Langevin Dynamics

06/20/2022
by   Ruqi Zhang, et al.
6

While low-precision optimization has been widely used to accelerate deep learning, low-precision sampling remains largely unexplored. As a consequence, sampling is simply infeasible in many large-scale scenarios, despite providing remarkable benefits to generalization and uncertainty estimation for neural networks. In this paper, we provide the first study of low-precision Stochastic Gradient Langevin Dynamics (SGLD), showing that its costs can be significantly reduced without sacrificing performance, due to its intrinsic ability to handle system noise. We prove that the convergence of low-precision SGLD with full-precision gradient accumulators is less affected by the quantization error than its SGD counterpart in the strongly convex setting. To further enable low-precision gradient accumulators, we develop a new quantization function for SGLD that preserves the variance in each update step. We demonstrate that low-precision SGLD achieves comparable performance to full-precision SGLD with only 8 bits on a variety of deep learning tasks.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/09/2018

High-Accuracy Low-Precision Training

Low-precision computation is often used to lower the time and energy cos...
research
03/04/2019

Learning low-precision neural networks without Straight-Through Estimator(STE)

The Straight-Through Estimator (STE) is widely used for back-propagating...
research
04/26/2019

SWALP : Stochastic Weight Averaging in Low-Precision Training

Low precision operations can provide scalability, memory savings, portab...
research
01/04/2023

On the Convergence of Stochastic Gradient Descent in Low-precision Number Formats

Deep learning models are dominating almost all artificial intelligence t...
research
11/09/2022

Variants of SGD for Lipschitz Continuous Loss Functions in Low-Precision Environments

Motivated by neural network training in low-bit floating and fixed-point...
research
11/27/2018

Reliable uncertainty estimate for antibiotic resistance classification with Stochastic Gradient Langevin Dynamics

Antibiotic resistance monitoring is of paramount importance in the face ...
research
03/08/2019

Accelerating Generalized Linear Models with MLWeaving: A One-Size-Fits-All System for Any-precision Learning

Learning from the data stored in a database is an important function inc...

Please sign up or login with your details

Forgot password? Click here to reset