The Future of AI Is Expensive

Why large AI models have been costing more and more, and why they’ll only get more expensive

September 2023  | subscribe here

1)  Why did AI explode in the last few years? Big models got even bigger

• AlexNet (2012):  One of the first “big models” and was made for image classification. 60M parameter was big at the time (0.2% the size of the largest models today) and made waves by beating state-of-the-art. 

• Resnet (2015):  An even bigger model, which improved accuracy by another 10% on top of AlexNet, but it cost 10x more expensive to train– a prescient pattern of compute’s diminishing returns on performance. 

• Transformer (2017): Released by Google with their famous “Attention is All You Need” paper. TLDR – with a fancy new algorithm, NLP models could be much better at learning human language. 

• GPT 2 (2019): OpenAI took note of Transformers and decided they were the “innovation” local maxima. Instead of inventing a more clever algorithm, OpenAI stuck with transformers, but dumped enormous compute to make models that were much, much bigger. GPT 2 was the result – it required 8x more training compute than Google’s Transformer model. 

• GPT 3 (2020): OpenAI kept building bigger models, and released GPT 3, which needed a whopping 1,662x more compute than GPT 2. 
  

2) Performance gains have been costing more and more

• OpenAI’s GPT 3 vs. GPT 2: 70% improvement, but cost ~$3M more
  

• Meta’s LLaMA vs GPT 3: 17% improvement but cost ~$3M more. 

• Google’s PaLM vs. LLaMA; 20% improvement but cost ~$25M more.

• GPT 4 vs. PaLM 2: 6% improvement but cost ~$100M more.

3a) Why models can’t get bigger

• 2020-2021: Models got much bigger, without using much more data. Compared to GPT 2, GPT 3 had a 100x increase in parameter size but only a 15x increase in training token data. And it worked back then – GPT 3 was much better than GPT 2. 

• Late 2021: Models got enormous. MT-NLG, Nvidia and Microsoft’s overweight baby, was a 530x increase in parameter size but only 13x increase in training token data. It was a goliath model that barely used any training data. This model underperformed across almost all benchmarks.

• 2022: Reset to smaller models with more data.  Google’s Chinchilla paper showed that a compute-efficient model required a higher ratio of training data to parameter size. This makes sense conceptually. An AI model is only as good as the training data you dump into it. 

• 2023: Large models with proportional amounts of data.  OpenAI took note of these new scaling laws. The two largest iterations, GPT-4 and PaLM, are both huge, but also use a lot more training data.

3b) More data then, but where?

• Medical records

• Private M&A agreements

• Corporate presentations and memos

• Financial underwriting documents 

• Conversations that aren’t even recorded as data! 

3c) A better algorithm?

• Why we needed this: “Regular” LLMs predict the next word of any given input text, if encountered on the internet. When a model needs to follow specific instructions (e.g., write a rap song in the voice of Scooby-Doo), it needs to be “taught” how to use its knowledge to do more useful tasks instead of pure prediction. 

• How does it work? You have a model generate 10 responses, have a human rank the responses from best to worst, and then use that human feedback to train a separate “reward model” to learn human preferences and adjust the base model. 

• Cost: OpenAI’s InstructGPT is a 100x smaller model than GPT3 (1.3B vs. 175B parameters) and it still required 20,000 hours of human feedback. Assuming an average Philippines outsourced labor cost of $30/hr, that’s $600k. That’s 17% of GPT3’s total training cost (~$3.6M). 
  x Scale AI, one of OpenAI’s largest data labeling partners, is rumored to have ~$350-400M ARR, mostly all data labeling for the ~6 foundational companies. OpenAI could be spending up to ~$50M on data labeling a year. In fact, OpenAI has begun insourcing data labeling from Scale AI – it’s a big enough financial cost that they don’t want to pay the 50% margin that Scale is charging. 
  x

• Overfitting: Sometimes the model anchors too much on the reward model. When OpenAI over-optimized a model to focus on positive sentiment, it learned that wedding parties were overwhelmingly positive. The model’s response to any arbitrary prompt would inevitably end up describing a wedding party. 

• Task evaluation: for models to be very smart, they’ll need to answer questions that are hard for humans to evaluate. For example, if a model produces an answer to a complicated scientific or mathematical question, it’ll be hard (and expensive) to find the right PhD graduate to assess if the model is correct. 

• Active learning: model labels it’s own training data (taking into account cost and benefit, according to a reward model)

• Self distillation:  use of a larger "teacher" model to provide a training signal for a smaller "student" model.

• Recursive Reward Modeling: OpenAI is very excited about this one. The idea is to break down a complicated human evaluation task into smaller, stepwise chunks. An AI can then evaluate each smaller step, which is also easier for a human to evaluate. 
  
  First, OpenAI first explored this technique with book summarization. Evaluating the summary of a whole book is hard for people who haven’t read it. But if humans can trust AI summaries of individual chapters, the whole book summary is easier to evaluate. 
  
  Second, My friends Hunter and Bowen at OpenAI implemented this idea with the Let’s Verify Step by Step paper. They broke down complex math problems into individual steps –
  
  
   
  It’s easier for a human to evaluate each step, and also improves the AI’s performance. So much so that it can do very complicated math: 
     
  
  However, once again, cost is a problem. The dataset they used contains about 800,000 step-level labels over 75,000 solutions. These humans here are more expensive too, those with advanced math education. I wouldn’t be surprised if this dataset cost several million. 
  

4) So, what does this all mean?

Sources