Training Costs

The Cost of Training NLP Models: A Concise Overview

AI21 Labs' 2020 overview of the financial and compute cost of training large NLP models, tracing the exponential growth in training expenditure from BERT through GPT-3. An early quantitative lens on the economics of scale in language model development.