DeepAI
Log In Sign Up

Sharpness-Aware Training for Accurate Inference on Noisy DNN Accelerators

11/18/2022
by   Gonçalo Mordido, et al.
0

Energy-efficient deep neural network (DNN) accelerators are prone to non-idealities that degrade DNN performance at inference time. To mitigate such degradation, existing methods typically add perturbations to the DNN weights during training to simulate inference on noisy hardware. However, this often requires knowledge about the target hardware and leads to a trade-off between DNN performance and robustness, decreasing the former to increase the latter. In this work, we show that applying sharpness-aware training by optimizing for both the loss value and the loss sharpness significantly improves robustness to noisy hardware at inference time while also increasing DNN performance. We further motivate our results by showing a high correlation between loss sharpness and model robustness. We show superior performance compared to injecting noise during training and aggressive weight clipping on multiple architectures, optimizers, datasets, and training regimes without relying on any assumptions about the target hardware. This is observed on a generic noise model as well as on accurate noise simulations from real hardware.

READ FULL TEXT
06/24/2020

Bit Error Robustness for Energy-Efficient DNN Accelerators

Deep neural network (DNN) accelerators received considerable attention i...
04/16/2021

Random and Adversarial Bit Error Robustness: Energy-Efficient and Secure DNN Accelerators

Deep neural network (DNN) accelerators received considerable attention i...
05/03/2022

MemSE: Fast MSE Prediction for Noisy Memristor-Based DNN Accelerators

Memristors enable the computation of matrix-vector multiplications (MVM)...
05/07/2022

Impact of L1 Batch Normalization on Analog Noise Resistant Property of Deep Learning Models

Analog hardware has become a popular choice for machine learning on reso...
11/25/2022

TAOTF: A Two-stage Approximately Orthogonal Training Framework in Deep Neural Networks

The orthogonality constraints, including the hard and soft ones, have be...
06/10/2022

Real-time Hyper-Dimensional Reconfiguration at the Edge using Hardware Accelerators

In this paper we present Hyper-Dimensional Reconfigurable Analytics at t...
12/20/2022

Walking Noise: Understanding Implications of Noisy Computations on Classification Tasks

Machine learning methods like neural networks are extremely successful a...