Life in the Singularity

Life in the Singularity

Google's Breakthroughs in AI Design

Mixture of Depths Meets Mixture of Experts

Matt McDonagh's avatar
Matt McDonagh
Apr 06, 2024
∙ Paid
1
1
Share

Think about how traditional language models work.

They gobble up sentences word-by-word, crunching through complex calculations on each word equally. But let's be honest, not every word deserves the VIP treatment. Some are keys to the whole conversation, others are filler.

This seems wasteful.

…and expensive!

On April 2, 2024 Google DeepMind released their

This post is for paid subscribers

Already a paid subscriber? Sign in
© 2025 Matt McDonagh
Privacy ∙ Terms ∙ Collection notice
Start writingGet the app
Substack is the home for great culture