Carbon Emissions and Large Neural Network Training

04/21/2021
by   David Patterson, et al.
17

The computation demand for machine learning (ML) has grown rapidly recently, which comes with a number of costs. Estimating the energy cost helps measure its environmental impact and finding greener strategies, yet it is challenging without detailed information. We calculate the energy use and carbon footprint of several recent large models-T5, Meena, GShard, Switch Transformer, and GPT-3-and refine earlier estimates for the neural architecture search that found Evolved Transformer. We highlight the following opportunities to improve energy efficiency and CO2 equivalent emissions (CO2e): Large but sparsely activated DNNs can consume <1/10th the energy of large, dense DNNs without sacrificing accuracy despite using as many or even more parameters. Geographic location matters for ML workload scheduling since the fraction of carbon-free energy and resulting CO2e vary  5X-10X, even within the same country and the same organization. We are now optimizing where and when large models are trained. Specific datacenter infrastructure matters, as Cloud datacenters can be  1.4-2X more energy efficient than typical datacenters, and the ML-oriented accelerators inside them can be  2-5X more effective than off-the-shelf systems. Remarkably, the choice of DNN, datacenter, and processor can reduce the carbon footprint up to  100-1000X. These large factors also make retroactive estimates of energy cost difficult. To avoid miscalculations, we believe ML papers requiring large computational resources should make energy consumption and CO2e explicit when practical. We are working to be more transparent about energy use and CO2e in our future research. To help reduce the carbon footprint of ML, we believe energy usage and CO2e should be a key metric in evaluating models, and we are collaborating with MLPerf developers to include energy usage during training and inference in this industry standard benchmark.

READ FULL TEXT

page 3

page 5

page 6

page 11

page 13

page 14

page 16

page 17

research
04/11/2022

The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink

Machine Learning (ML) workloads have rapidly grown in importance, but ra...
research
11/03/2022

Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model

Progress in machine learning (ML) comes with a cost to the environment, ...
research
07/06/2020

Carbontracker: Tracking and Predicting the Carbon Footprint of Training Deep Learning Models

Deep learning (DL) can achieve impressive results across a wide variety ...
research
10/12/2022

Energy Consumption-Aware Tabular Benchmarks for Neural Architecture Search

The demand for large-scale computational resources for Neural Architectu...
research
02/09/2022

Neural Architecture Search for Energy Efficient Always-on Audio Models

Mobile and edge computing devices for always-on audio classification req...
research
11/19/2019

Energy Usage Reports: Environmental awareness as part of algorithmic accountability

The carbon footprint of algorithms must be measured and transparently re...
research
06/05/2019

Energy and Policy Considerations for Deep Learning in NLP

Recent progress in hardware and methodology for training neural networks...

Please sign up or login with your details

Forgot password? Click here to reset