Google's Breakthroughs in AI Design
Mixture of Depths Meets Mixture of Experts
Think about how traditional language models work.
They gobble up sentences word-by-word, crunching through complex calculations on each word equally. But let's be honest, not every word deserves the VIP treatment. Some are keys to the whole conversation, others are filler.
This seems wasteful.
…and expensive!
On April 2, 2024 Google DeepMind released their Mixture of Depths (MoD) design to solve for this.
Mixture of Depths is like hiring a super-efficient manager for your language model. This manager has a fixed budget of computational resources to spend.
They get to decide which words in a sentence are worth the heavy investment and which can get by with a lighter touch. MoD allows the model to dynamically change this resource allocation throughout its layers.
The Secret Weapon: Top-k Routing
LLMs rely on a concept called attention to process info…
