A Transistor Operations Model for Deep Learning Energy Consumption Scaling Law

05/30/2022
by   Chen Li, et al.
7

Deep Learning (DL) has transformed the automation of a wide range of industries and finds increasing ubiquity in society. The high complexity of DL models and its widespread adoption has led to global energy consumption doubling every 3-4 months. Currently, the relationship between the DL model configuration and energy consumption is not well established. At a general computational energy model level, there is both strong dependency to both the hardware architecture (e.g. generic processors with different configuration of inner components- CPU and GPU, programmable integrated circuits - FPGA), as well as different interacting energy consumption aspects (e.g., data movement, calculation, control). At the DL model level, we need to translate non-linear activation functions and its interaction with data into calculation tasks. Current methods mainly linearize nonlinear DL models to approximate its theoretical FLOPs and MACs as a proxy for energy consumption. Yet, this is inaccurate (est. 93% accuracy) due to the highly nonlinear nature of many convolutional neural networks (CNNs) for example. In this paper, we develop a bottom-level Transistor Operations (TOs) method to expose the role of non-linear activation functions and neural network structure in energy consumption. We translate a range of feedforward and CNN models into ALU calculation tasks and then TO steps. This is then statistically linked to real energy consumption values via a regression model for different hardware configurations and data sets. We show that our proposed TOs method can achieve a 93.61

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/13/2023

Energy-Efficient GPU Clusters Scheduling for Deep Learning

Training deep neural networks (DNNs) is a major workload in datacenters ...
research
08/23/2023

FECoM: A Step towards Fine-Grained Energy Measurement for Deep Learning

With the increasing usage, scale, and complexity of Deep Learning (DL) m...
research
11/14/2022

Understanding the Energy Consumption of HPC Scale Artificial Intelligence

This paper contributes towards better understanding the energy consumpti...
research
06/12/2020

Power Consumption Variation over Activation Functions

The power that machine learning models consume when making predictions c...
research
06/06/2021

From DNNs to GANs: Review of efficient hardware architectures for deep learning

In recent times, the trend in very large scale integration (VLSI) indust...
research
09/16/2021

Machine learning methods for modelling and analysis of time series signals in geoinformatics

In this dissertation is provided a comparative analysis that evaluates t...
research
08/01/2018

Energy-based Tuning of Convolutional Neural Networks on Multi-GPUs

Deep Learning (DL) applications are gaining momentum in the realm of Art...

Please sign up or login with your details

Forgot password? Click here to reset