While a decade ago, only a few ML pioneers were training neural networks on GPUs (Graphical Processing Units), in recent years powerful GPUs have become increasingly accessible and used by ML practitioners worldwide. Furthermore, new models often need to beat existing challenges, which entails training on more GPUs, with larger datasets, for a longer time. This expansion brings with it ever-growing costs in terms of the energy needed to fuel it. This trend has been the subject of recent studies aiming to evaluate the climate impact of AI, which have predominantly put the focus on the environmental cost of training large-scale models connected to grids powered by fossil fuels Strubell et al. (2019); Schwartz et al. (2019). While these models are not necessarily representative of common practice, we believe that it is important to continue this conversation further and work towards defining the tools and steps that we need to assess the carbon emissions generated by the models we train, as well as to propose ways to reduce those emissions.
In this work, we present our Machine Learning Emissions Calculator (https://mlco2.github.io/impact/
), a tool for our community to estimate the amount of carbon emissions produced by training ML models. We accompany this tool with a presentation of key concepts and an explanation of the factors impacting emissions. Finally, we end our article with some recommendations of best practices for the overall ML research community, as well as for individual researchers.
2 Quantifying Carbon Emissions in Neural Network Training
In order to quantify carbon emissions, we use CO2-equivalents (CO2eq), which is a standardized measure used to express the global-warming potential of various greenhouse gases as a single number, i.e. as the amount of CO2 which would have the equivalent global warming impact Eggleston et al. (2006). We will use this single metric to compare the factors and choices that impact overall amount of emissions produced by training an ML model in the sections below.
2.1 Type of Energy Used
Practically speaking, it is hard to estimate exactly the amount of CO2eq emitted by a cloud server in a given location because the information regarding the energy grid that it is connected to is rarely publicly available. However, if we assume that all servers are connected to local grids at their physical location, we are able to make an estimation of the amount of CO2eq that they emit using public data sources To and Lee (2017); Brander et al. (2011) . Therefore, in order to create our emissions calculator, we gathered data regarding CO2eq emissions of different grid locations and cross-referenced them with known GPU server locations from the three major cloud providers: Google Cloud Platform, Microsoft Azure and Amazon Web Services 111The data can be found at https://github.com/mlco2/impact/tree/master/data. Our aim in doing this is to illustrate the degree of variability that exists depending on the location of a given server. For instance, in Figure 1, we show the distribution and variation in carbon emissions depending on geographical region. It can be noted that a large amount of variation can be found within a single region; for instance, servers located in North America can emit anywhere between 20g CO2eq/kWh in Quebec, Canada to 736.6g CO2eq/kWh in Iowa, USA Brander et al. (2011).
2.2 Computing Infrastructure and Training Time
Another, more subtle, factor in carbon emitted by a neural network is the computing infrastructure used and training time of the model. In terms of performance the number of floating point operations per second (FLOPS) of GPUs has been steadily increasing in recent years, from 100 Giga FLOPS per second in 2004 to up to 15 Tera FLOPS per second in recent hardware Harju et al. (2012). However, with neural network architectures becoming deeper and more complex, recent state-of-the-art models are often trained on multiples GPUs for several weeks (or months) to beat benchmark performance, requiring more and more energy.
Finally, when it comes to defining a training procedure for ML architectures, there are several elements to consider: for starters, whether it is necessary to train a model from scratch or whether fine-tuning is adequate for the task at hand. Notably, recent research has shown that using pre-trained models with task-specific fine-tuning performs as well as training from scratch, while being more robust, for tasks in image recognition Tajbakhsh et al. (2016); Yanai and Kawano (2015) and NLP Howard and Ruder (2018)
. Furthermore, when it comes to hyperparameter search, it has been proven both empirically and theoretically that random hyperparameter search is more efficient than grid search for hyperparameter optimizationBergstra and Bengio (2012), and there is much research being done on ways to improve the efficiency of hyperparameter optimization Falkner et al. (2018a); Feurer and Hutter (2019), which makes it possible to continue choosing the right hyperparameters for new models without incurring superfluous computing and energy costs.
3 ML Emissions Calculator and Actionable Items
It is difficult to provide clear-cut guidelines for ML researchers to follow in order to reduce the carbon emissions, or specific benchmarks for the training time that a given model or task warrants. Nonetheless, we think that there are certain best practices and actionable items that can be adopted by our community to reduce environmental impact of the ML domain. We present some of these, along with our ML emissions calculator, in the current section.
Quantify Your Emissions
Being informed regarding the factors that impact the quantity of carbon emissions produced by ML research is the first step to making positive changes. It is for this reason that we created our ML Emissions Calculator. This tool, currently in its alpha version, takes as input the details regarding the training of an ML model: the geographical zone of the server, the type of GPU, and the training time, and gives as output the approximate amount of CO2eq produced. We collected publicly available data for the 4 main variables of this computation: (i) the energy consumption of hardware (see "Choose More Efficient Hardware" below), (ii) the location of providers’ regions of compute – which we assumed to be connected to their local grid, (iii) the region’s CO2eq emissions per kWh and (iv) potential offsets bought by the provider.
We intend to adopt an open and transparent approach: the data we used is publicly available, debatable and editable through Github issues and pull requests. We are therefore open to updating data as more information becomes available. Since this paper’s core goal is to raise awareness around the carbon emissions of ML, we have also included two educational sections in the website: one about learning the main notions and concepts related to this domain (e.g. RECs, carbon neutrality, etc.), the other about actionable items an individual or an organization can leverage to mitigate their carbon impact.
Choose Cloud Providers Wisely
In recent years, many cloud providers have defined ambitious sustainability goals and are offsetting their emissions through Renewable Energy Certificates (RECs) in an effort to become carbon neutral, a term used to indicate a net zero carbon footprint of an organization. Each REC bought attests that 1 MWh of renewable energy has been added to the energy grid and can be used to offset an equivalent amount of non-renewable energy. For instance, Google Cloud Platform is certified carbon neutral and funds solar and wind farms directly on local grids through RECs Google (2018). Microsoft Azure is also certified carbon neutral and 44% of its electricity consumption directly comes from renewable energy, according to a 2016 estimate Microsoft (2018). Finally, to the best of our knowledge, while not yet 100% carbon neutral on an organizational level, Amazon Web Services is also funding renewable energy projects and some of their data centers are powered by renewable energy Services (2019).
Another major energy consumption factor of server installations is the power usage effectiveness (PUE) of the centers where the GPUs are hosted, which represents the percentage of energy consumption that is used for cooling, power conversion, and other auxiliary tasks, and can vary immensely. For example, Google Cloud Services has an average PUE of 1.1, meaning that only 11% of their total energy usage is not used for the servers themselves, a ratio that they have been steadily reducing using Reinforcement Learning9; 10. Finally, if you rely on a local private compute infrastructure, it is also possible to engage with administrators about quantifying and offsetting the emissions produced, as well as improving the efficiency of your grid – this may help bring your organization toward carbon neutrality and have a significant impact at scale.
Select Data Center Location
While many cloud providers are carbon neutral, some of their data centers may still be carbon intensive due to the local grid that they are connected to, whereas others will be low carbon and powered solely by renewable energy sources. Hence, selecting the data center location where an algorithm will be trained has a large impact on its direct carbon emissions. This choice can be achieved by consciously selecting the server location before dispatching your jobs. As we illustrated in previous sections, this single choice can make the direct emissions of an algorithm vary by a factor of 40, from 20g CO2eq/kWh in a location that uses renewable energy sources to 820g CO2eq/kWh in a location that solely relies on fossil fuels Brander et al. (2011). For a model such as VGG Simonyan and Zisserman (2014) or BERT Devlin et al. (2018), which are trained on multiple GPUs for several weeks, this can correspond to avoiding emitting several hundreds of kilograms of CO2eq by training on a server powered by hydroelectricity instead of fossil fuels.
Reduce Wasted Ressources
Grid search is still often used in practice, in spite of its low efficiency both in terms of model performance and environmental impact. However, it has been shown that random search (and others) not only is a straightforward replacement but also has potential to significantly accelerate hyperparameter search Li et al. (2016, 2018); Falkner et al. (2018b), consequently reducing carbon emissions. Also, while failed experiments are a common part of ML research and are sometimes unavoidable, their number can often be reduced with careful design such as unit tests, integration tests, and extensive and early debugging. Uninformative experiments are also frequent (sometimes unknowingly) – they can be caused by unstable learning algorithms requiring averaging results over many random seeds. Taking the time to carry out a literature review and to understand the potential sources of noise before launching large-scale hyperparameter searches increases the chance of obtaining reproducible and statistically significant results. Hence, reducing the need to extend the experiment cycles.
Choose More Efficient Hardware
The choice of computing hardware can also have a major impact on ML emissions. To perform a comparison between different devices, their compute efficiency can be estimated in FLOPS/W. This estimation is based on their theoretical peak performance with respect to their Thermal Design Power (TDP)222Empirical measurement of GFLOPS/W on various ML architecture would provide more accurate numbers but we are only interested in approximate values to compare classes of devices.. Using this approach, it can be found that CPUs can be 10 times less efficient than GPUs while TPU 3 can be 4 to 8 times more efficient than GPUs Teich (2018) (refer to Table Appendix B: Hardware Efficiency for details). Interestingly, in contexts where low power consumption and efficiency are important, e.g., for embedded applications, GPUs such as the Jetson AGX Xavier can be 10 to 20 times more efficient than traditional GPUs.
The factors that we discussed in the current work give ML practitioners a certain amount of control over the environmental impact produced by the training of their models. We are aware that these choices are not always possible to make in practice – for instance, the choice of server location can be limited due to privacy considerations in the case of applications in the medical or financial domain, and large amounts of data may be needed to produce most robust models. However, we find that our emissions calculator is a good starting point to estimate the impact that small choices in model training can have on direct carbon emissions resulting from ML research.
Despite our best efforts, our calculator remains simply an approximation of the true emissions produced by ML training for several reasons: to start with, there is the issue of global load balancing, i.e. if a majority of practitioners choose to run their models in a low-carbon location, the servers will get saturated and other servers will still need be used. In that perspective, the global gain will not be a 40-fold reduction of emissions, but much smaller. Furthermore, there is a lack of transparency with regards to the true quantity of emissions produced by organizations, so while we use the current best publicly-available sources, there is still a large margin of error with regards to the exact quantity of energy consumed and carbon produced – we remain open to additional data sources and numbers. Finally, while in the current version of our tool, we focus on quantifying the emissions of training ML models, there is still the issue of deploying them, since the inference process is also energy-expensive, especially if done continuously and on a large scale. This is something that should be taken into account by ML practitioners in their products that are deployed in real-world settings, for instance by using energy-efficient architectures Cai et al. (2017) and computing infrastructure.
There are also more far-reaching discussions to be had regarding the environmental value of scientific knowledge in general and of ML research in particular. On one hand, there is valuable research to be done in ML especially with regards to tackling climate change Rolnick et al. (2019); Schmidt et al. (2019), whereas on the other hand, the emissions of the field of ML are growing quickly Strubell et al. (2019). We do not propose the solution to this problem, but we believe that there are steps to be taken, for instance by using efficiency as an evaluation criterion (as proposed by Schwartz et al. (2019)) or by taking concrete steps to reduce emissions (as proposed by the current paper). We hope that our work, along with others, will open the door for these conversations and debates to take place, to quantify the environmental impact of our field, and for positive changes that can be made to reduce it.
-  (2012) Random search for hyper-parameter optimization. Journal of Machine Learning Research 13 (Feb), pp. 281–305. Cited by: §2.2.
-  (2011) Electricity-specific emission factors for grid electricity. Ecometrica, Emissionfactors. com. Cited by: §2.1, §3.
Neuralpower: predict and deploy energy-efficient convolutional neural networks. arXiv preprint arXiv:1710.05420. Cited by: §4.
-  (2018) Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. Cited by: §3.
-  (2006) 2006 ipcc guidelines for national greenhouse gas inventories. Vol. 5, Institute for Global Environmental Strategies Hayama, Japan. Cited by: §2.
-  (2018) BOHB: robust and efficient hyperparameter optimization at scale. arXiv preprint arXiv:1807.01774. Cited by: §2.2.
-  (2018) BOHB: robust and efficient hyperparameter optimization at scale. arXiv preprint arXiv:1807.01774. Cited by: §3.
-  (2019) Hyperparameter optimization. In Automated Machine Learning, pp. 3–33. Cited by: §2.2.
-  (2014) Machine learning applications for data center optimization. Cited by: §3.
-  (2019) Google Data Centers efficiency: how we do it. Note: https://www.google.com/about/datacenters/efficiency/internal/Accessed: 2019-08-23 Cited by: §3.
-  (2018) Google environmental report 2018. External Links: Cited by: §3.
-  (2012) Computational physics on graphics processing units. In Proceedings of the 11th international conference on Applied Parallel and Scientific Computing, pp. 3–26. Cited by: §2.2.
-  (2018) Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146. Cited by: §2.2.
-  (2018) Massively parallel hyperparameter tuning. arXiv preprint arXiv:1810.05934. Cited by: §3.
-  (2016) Hyperband: a novel bandit-based approach to hyperparameter optimization. arXiv preprint arXiv:1603.06560. Cited by: §3.
-  (2018) Beyond carbon neutral. white paper. External Links: Cited by: §3.
-  (2019) Tackling climate change with machine learning. arXiv preprint arXiv:1906.05433. Cited by: §4.
-  (2019) Visualizing the consequences of climate change using cycle-consistent adversarial networks. CoRR abs/1905.03709. External Links: Cited by: §4.
-  (2019) Green ai. arXiv preprint arXiv:1907.10597. Cited by: §1, §4.
-  (2019) AWS & sustainability. External Links: Cited by: §3.
-  (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556. Cited by: §3.
Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.02243. Cited by: §1, §4.
-  (2016) Convolutional neural networks for medical image analysis: full training or fine tuning?. IEEE transactions on medical imaging 35 (5), pp. 1299–1312. Cited by: §2.2.
-  (2018) Tearing apart google’s tpu 3.0 ai coprocessor. Note: https://www.nextplatform.com/2018/05/10/tearing-apart-googles-tpu-3-0-ai-coprocessor/ Cited by: §3.
-  (2017) GHG emissions from electricity consumption: a case study of hong kong from 2002 to 2015 and trends to 2030. Journal of cleaner production 165, pp. 589–598. Cited by: §2.1.
-  (2015) Food image recognition using deep convolutional network with pre-training and fine-tuning. In 2015 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), pp. 1–6. Cited by: §2.2.
Appendix A: Energy Grid Data Used for the ML Emissions Calculator
For clarity purposes, the data presented in this appendix contains fewer columns than what can be found in our public database: https://github.com/mlco2/impact/tree/master/data. For instance we did not include the sources in these tables. For future reference, the data’s commit hash at the time of publication is e692e28.
Google Cloud Platform
Amazon Web Services
|eu-central-1||Germany||Frankfurt am Main||615|
|eastasia||Hong Kong||Wan Chai||702|
Appendix B: Hardware Efficiency
|RTX 2080 Ti||250||13.45||26.90||53.80||107.60|
|GTX 1080 Ti||250||11.34||0.17||45.36||0.68|
|Intel Xeon E5-2699||145||0.70||0.70||4.83||4.83|