The Cost of Training NLP Models: A Concise Overview

04/19/2020
by   Or Sharir, et al.
9

We review the cost of training large-scale language models, and the drivers of these costs. The intended audience includes engineers and scientists budgeting their model-training experiments, as well as non-practitioners trying to make sense of the economics of modern-day Natural Language Processing (NLP).

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 2

02/15/2022

A Survey on Model Compression for Natural Language Processing

With recent developments in new architectures like Transformer and pretr...
04/30/2022

EasyNLP: A Comprehensive and Easy-to-use Toolkit for Natural Language Processing

The success of Pre-Trained Models (PTMs) has reshaped the development of...
04/21/2019

NeuronBlocks -- Building Your NLP DNN Models Like Playing Lego

When building deep neural network models for natural language processing...
12/22/2016

Continuous multilinguality with language vectors

Most existing models for multilingual natural language processing (NLP) ...
08/28/2016

What to do about non-standard (or non-canonical) language in NLP

Real world data differs radically from the benchmark corpora we use in n...
03/04/2022

Deep Lexical Hypothesis: Identifying personality structure in natural language

Recent advances in natural language processing (NLP) have produced gener...
09/29/2020

Utility is in the Eye of the User: A Critique of NLP Leaderboards

Benchmarks such as GLUE have helped drive advances in NLP by incentivizi...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Costs: Not for the faint hearted

The cost of floating-point operations (FLOPs), the basic Neural Network (NN) operation, has been decreasing. For example, Google reported GoogleTPU a 38% cost decrease in ResNet-50 training costs

222It also reported a dramatic 27 decrease in training time. While training time is not our focus, it is relevant indirectly: Compressed time makes it realistic to train larger models, which costs more.. This was achieved with optimized hardware (moving from GPUs to TPUs) coupled with framework-level optimizations, exploiting parallelism opportunities. This kind of cost reduction isn’t an isolated occurrence – we’re seeing the costs of training large models fall as hardware innovations and training techniques improve. Despite this, overall costs have increased, and can run into the millions. We’ll explain why this is occurring and what factors play a significant role in the costs of training333There is a whole other discussion to be had on the costs of NLP models at inference time. These are quite related to the training costs, but deserve a separate discussion. In particular, the inference phase allows for post-training model optimizations, for example via model distillation sanh2019distilbert,jiao2019tinybert. This discussion is beyond the scope of this article. NLP models.

Just how much does it cost to train a model? Two correct answers are “depends” and “a lot”. More quantitatively, here are current ballpark list-price costs of training differently sized BERT devlin2019Bert models on the Wikipedia and Book corpora (15 GB). For each setting we report two numbers - the cost of one training run, and a typical fully-loaded cost (see discussion of "hidden costs" below) with hyper-parameter tuning and multiple runs per setting (here we look at a somewhat modest upper bound of two configurations and ten runs per configuration).444The following figures are based on internal AI21 Labs data. They can be somewhat lower due to discounts, or using preemptible versions of the system. The figures also assume the use of cloud solutions such as GCP or AWS, and on-premise implementations are sometimes cheaper. Still, the figures provide a general sense of the costs.

  • $2.5k - $50k (110 million parameter model)

  • $10k - $200k (340 million parameter model)

  • $80k - $1.6m (1.5 billion parameter model)

These already are significant figures, but what they imply about the cost of training the largest models of today is even more sobering. Exact figures are proprietary information of the specific companies, but one can make educated guesses. For example, based on information released by Google, we estimate that, at list-price, training the 11B-parameter variant

555 With context lengths of 512 for both encoding and decoding, 128 attention heads, and 65k-dimensional feed-forward layers. of T5 2019t5 cost well above $1.3 million for a single run. Assuming 2-3 runs of the large model and hundreds of the small ones, the (list-)price tag for the entire project may have been $10 million666These $ figures come with substantial error bars, but we believe they are in the right ballpark..

Not many companies – certainly not many startups – can afford this cost. Some argue that this is not a severe issue; let the Googles of the world pre-train and publish the large language models, and let the rest of the world fine-tune them (a much cheaper endeavor) to specific tasks. Others (e.g., CloudAndInnovation) are not as sanguine.

2 Cost Drivers: Size Matters

We are not aware of a formula that tells you how many FLOPs are needed in a given NLP setting to achieve a given performance777 It is worth noting the work of kaplan2020scaling, which analyzes the impact of various variables, including model size and amount of compute, on performance, as measured by perplexity. Although the paper does not directly address the question we are after, the methodology it offers may provide useful hints. Other relevant papers include dodge-etal-2019-show,li2020train,rosenfeld2020a. . However, there are several variables that impact this number, all of which have increased dramatically in the past few years, far surpassing the once-deemed “massive” vision-focused ML models.888

Although computer vision is not our focus here, the contrast with NLP is striking, and we discuss it briefly in Appendix 

A.

Here are some of the relevant variables, which fall into three categories: (a) size of dataset, (b) model size (we use the number of parameters as a proxy), and (c) training volume (we use as proxy the total number of tokens processed during pre-training). The top row applies to all models, and the bottom row zooms in on transformer-based models.

The exact ways in which these increases impact the number of FLOPs are subtle, and depend on the specific training scheme and architecture. For example, fewer FLOPs are needed when training BERT-style models versus GPT-2 radford2019language models with comparable model and data sizes, and training steps. Other training schemes can introduce additional factors that dictate cost; for example, the adversarial training scheme of ELECTRA Clark2020ELECTRA uses an additional “generator” model during training. This increases the relative per-step costs, but requires fewer steps, thus reducing the overall costs. Despite these subtleties, however, it is clear that all these growing numbers correlate with an overall trend towards a greater number of FLOPs, which determine the bottom line.

On top of the above, there are also additional hidden costs, which are often overlooked. Each model must be trained multiple times – this is in order to minimize random effects (each run is inherently stochastic), and to search over a combinatorially large hyper-parameter search space. This means there can be a large multiple over the cost of a single training episode (although significant cost savings can be had by conducting most of the experiments on the smaller models first, before training the large models in the optimized configuration).

3 The Future

The reason the community has adopted the mega-scale, brute-force statistical approach is that it works; it has yielded better performance than any alternative. And since NLP has substantial economic value, no cost is too high in pursuit of good performance. We do not see an end to the use of large NN models operating on massive corpora, and one can imagine the costs escalating further, as the community develops more elaborate architectures in pursuit of more ambitious tasks. As you go from sentences to whole documents and beyond, you can imagine the need for more dimensions per token, longer contexts, and potentially more layers. Adding external knowledge sources, although potentially reducing the sole reliance on the network (see below), could also contribute to expanding the size of the network in order to reflect the external knowledge in the embedding space. Indeed, there is already discussion SambaNovaModel of 100B-parameter models. That said, we see several factors that may help tame this explosion and prevent things from getting out of hand. In increasing order of importance:

  • Further reduction of raw-compute prices due to increased competition. According to this (admittedly self-interested) blog post AWSCosts, the prices on AWS were reduced over 65 times since its launch in 2006, and by as much as 73% between 2014 and 2017. We expect the same trend for AI-oriented compute offerings.

  • More efficient NN architectures, driven in part by economics and partly by environmental considerations. For example, the Reformer Kitaev2020Reformer architecture uses heuristics to reduce the complexity of the attention mechanism of transformers from quadratic to

    . Similarly, ALBERT Lan2020ALBERT achieves better accuracy with fewer parameters by factorizing the embedding matrix and weight sharing across layers. We expect to see more of this.

  • Ending the State-of-the-Art (SOTA) race. There is increasing recognition in the community that significant amount of compute is sunk into reaching the top of leaderboards of the many challenge datasets, often involving many (in some reported cases, thousands) of runs, just so that one instance will luck into first place. Such overfitting is of course of little value, and we expect to see less of it.

  • Maxing out on useful data. There is just that much (useful) text that has been written, or that will be. At some point, we will have trained on Borges’ Universal Library.

  • Useful as NNs are, there is a school of thought that holds that statistical ML is necessary but insufficient, and will get you just that far. Instead, the thinking goes, you need to incorporate structured knowledge and symbolic methods into the mix, and that in turn depends on brain rather than (only) brawn. This is a view we subscribe to at AI21 Labs (see levine2019sensebert as an example).

Appendix A NLP versus CV

With a few notable exceptions999There have been a few attempts to create “mega-models” for CV, e.g., FixResNet FixResNet has 830M parameters and was trained on nearly a billion weakly-labeled images. However, the gains are not as great compared to the added costs, and such approaches have not become the norm just yet., you do not see in computer vision (CV) the large numbers and cost escalations you do in NLP, and it is natural to ask why. We enter this discussion with some trepidation. While some of the folks at AI21 Labs have experience in CV, it is not our core competence as a company. Furthermore, some of the CV experts with whom we spoke did not have firm opinions here, and the opinions they did have did not always agree with each other. Still, since we have been asked this question we feel we should address it, but please treat the following more as a beginning of a discussion rather than definitive answers.101010See also this article NLPImageNet for an interesting discussion circa 2018.

We believe that there are fundamentally two reasons why training CV models is cheaper than training NLP models:

  • Images versus sentences. Images are smooth and local, in that by and large the value of a pixel depends mostly on its close neighborhood and less so on other parts of the image, and furthermore the value does not change drastically from one pixel to its neighbor. Moreover, images are iconic, by which we mean that "what you see is what you get"; an image of chair and a desk represents a chair and desk. Language is very different. Words far apart can be coupled probabilistically, and language is compositional; the way you string words together carries as much meaning as the semantic content of the words themselves.

  • Object recognition versus – what? The canonical problem in computer vision – object recognition/classification – is, while by no means trivial, relatively simple. It has no direct analog in NLP. One could argue that topic- or sentiment-analysis are somewhat analogous at the document level, and word-sense disambiguation is at the sentence level. But the analogy is weak, and neither of these plays the same central role that object recognition does in vision. Another telling analogy is between object identity in vision (is the person seen in this image the same as the person in this other image?) and noun-phrase co-reference in NLP (does “the president” refer to the same entity as “Mr. Trump”?). Here the separation between vision and language is stark; object identity is close to being a solved problem, while co-reference is still unsolved. And this is leaving aside the issue that even once solved, co-reference on its own would not bring the same value that object recognition does in CV.

These differences manifest themselves in several ways, including these:

  • CNNs versus transformers.

    CV problems lend themselves to Convolutional Neural Networks (CNNs), while the canonical NLP approach has centered around transformer models, which are inherently more expensive than CNNs. The different choice of architecture is directly related to the differences between images and sentences; the locality property matches with the local windows of convolutional layers, and smoothness with the sub-sampling operation in pooling layers. Since language does not enjoy these properties, we must use a more general, but less efficient, architecture such as the transformer.

  • Supervised versus semi-supervised versus self-supervised learning.

    NLP and computer vision employ all of these learning regimes, but the balance is different. Unlike in computer vision, most of the training time of NLU models is devoted to self-supervised learning of language itself, and only a small portion is devoted to (supervised) fine-tuning of the model to solve a specific task. This is related to the inherent complexity of the structure of language and the nature of NLU and NLG tasks. Much larger datasets are needed in order to provide useful signal, and, just as bad, the tasks are inherently more ambiguous and the data is harder for people to annotate than in image classification; it is easier to answer the question “Is that a person or a car” than “Does this sentence imply that sentence”. Furthermore, data augmentation is much more successful in vision than in NLP, and semi-supervised learning aided by data augmentation has led to many recent SOTA results in vision. In contrast, NLP has been driven toward purely self-supervised learning (“the NLP revolution will not be supervised!”). This in turn translates into larger training datasets compared to the supervised setting, as well as longer training cycles.

Again, important caveats apply to all of the above. Even in object recognition, the larger context of the image can matter when determining what is depicted in a given image patch. Furthermore, object recognition is not the sole focus of CV, and more elaborate tasks, such as scene understanding Geman3618, certainly do not have the smooth, local properties mentioned (to use a famous example, object recognition techniques do not tell you the interesting part about an image depicting a piano dropping through the air and about to land on someone’s head). As another example, in the area of image synthesis, which often requires accommodating complex logical, real-world constraints in the synthesized image, CNNs give way to inherently more expensive models such as GANs.

Despite these important caveats, we feel the above analysis is fair, for two reasons. First, it is the case that among CV technologies, object recognition has brought the most commercial value to date, and CNNs have been the main driver behind its success. And second, more ambitious tasks such as scene understanding are getting close to NLP in being less well defined and less well solved. They also call for the same commonsense reasoning as does NLP, and thus are likely require the elaborate techniques – and costs – currently associated with NLP.