The Emergence Phenomenon
Here's where it gets interesting—and concerning:
Some capabilities don't improve smoothly. They emerge suddenly.
At small scale: The model can't do multi-step arithmetic.
At medium scale: Still can't.
At large scale: Suddenly can.
This "emergence" happens when capabilities require some threshold of model capacity. Below the threshold, the capability is absent. Above it, the capability appears—often quickly.
Documented Emergent Capabilities
Research has identified capabilities that emerge abruptly:
Three-digit addition: Absent in small models, suddenly present above ~10^22 training FLOPs.
Multi-step reasoning: Chains of inference that require holding context across many steps.
Code generation: Writing functional programs from descriptions.
Theory of mind: Reasoning about beliefs and knowledge of others.
Each capability has a compute threshold. Below it, the capability doesn't exist. Above it, the capability often appears suddenly.
The Implication
This means future AI systems may suddenly gain capabilities we can't currently anticipate.
We know the rough compute at which certain capabilities emerged historically. But we don't know what capabilities await at compute levels we haven't yet reached.
The next 10x might bring something surprising.
Chinchilla and the Data Wall
In 2022, DeepMind's "Chinchilla" paper refined the scaling laws with a crucial insight:
Previous models were undertrained.
GPT-3 had 175 billion parameters but was trained on "only" 300 billion tokens. The optimal compute allocation would use more data and fewer parameters.
Chinchilla showed that a 70-billion parameter model trained on 1.4 trillion tokens outperformed GPT-3—with less compute.
The Data Wall
This created a new problem: We might run out of data.
Chinchilla-optimal training of large models requires trillions of tokens. High-quality text data on the internet might be 10-50 trillion tokens total.
As models require more data, they approach the limit of available high-quality text. This is the "data wall."
Responses to the Data Wall
Several approaches are being explored:
Synthetic data: Use AI to generate training data. Early results are promising but create feedback loop concerns.
Multi-modal data: Train on images, video, audio—not just text. More total data available across modalities.
Higher quality, less quantity: Better filtering and curation to extract more learning from existing data.
Test-time compute: Shift capability from training to inference, where compute is more flexible.
The data wall is a real constraint, but not an insurmountable one.
Predictions Based on Scaling Laws
Let me share what scaling laws suggest about AI's trajectory:
Short-Term (1-3 Years)
Compute investments will continue increasing. Training runs costing $500 million to $1 billion are coming. The scaling laws justify these investments.
Capabilities will continue improving predictably. Each generation of models will be 2-5x more capable than the previous, as compute increases.
Emergence events will surprise us. Some capabilities will appear suddenly at compute thresholds we haven't yet reached.
Medium-Term (3-7 Years)
The data wall will force innovation. Pure text scaling will exhaust available data. New approaches will dominate.
Efficiency improvements will compound. Better architectures (like MoE) will extract more capability per unit compute, extending the scaling trajectory.
Human-competitive performance in most cognitive tasks. If scaling continues, models will match human performance across most benchmarks.
Long-Term (7-15 Years)
Physical limitations may become binding. Energy, chip manufacturing, and infrastructure constraints may limit compute growth.
Recursively self-improving systems become possible. If models can contribute to their own improvement, a qualitative shift occurs.
The scaling laws may break. Either we hit fundamental limits, or something changes that makes current predictions invalid.
The Uncomfortable Questions
Scaling laws raise questions we don't have good answers for:
What happens when AI exceeds human capability across domains?
Not in narrow tasks—that already happened. But in general cognitive ability. The scaling laws suggest this is a matter of when, not if.
Can we control systems smarter than us?
Intelligence doesn't guarantee benevolence. As AI capability increases, the alignment problem becomes more urgent.
Who decides how much compute to deploy?
A few companies control the largest training runs. Their decisions—about what to build and how to deploy it—have global implications.
Does society have time to adapt?
If capabilities emerge suddenly and scaling continues, societal adaptation may lag technological capability. This creates instability.
Why You Should Pay Attention
Let me be direct about why this matters to you personally:
Career Planning
Scaling laws suggest AI capability will increase dramatically within your career lifetime. Planning around current AI limitations is increasingly risky.
Investment Decisions
Companies that understand scaling laws have made fortunes. Understanding what's predictable about AI helps evaluate opportunities.
Civic Engagement
Democratic societies will need to make decisions about AI development. Understanding scaling laws is foundational to informed participation.
Personal Adaptation
The world is changing in predictable ways. Understanding the trajectory helps you position yourself advantageously.
The Final Word
Here's what scaling laws have taught us:
AI improvement isn't magic or luck—it's engineering. Given enough compute, data, and parameters, capability follows predictable laws.
We're early on the curve. Current systems represent a small fraction of the capability these laws predict is possible.
The future is more predictable than we thought—and more uncertain. We can predict aggregate capability. We can't predict specific emergent abilities.
Choices made now matter. The companies and societies that understand these dynamics will shape the future. Those that don't will be shaped by it.
The scaling laws are a map of AI's trajectory. Reading the map doesn't tell you exactly where you'll end up—but it shows you the terrain you'll travel through.
Choose your path wisely.
Want to understand where AI is heading? Subscribe to Absomind Blog for expert analysis of the trends shaping our technological future.
Comments · 0
Beta: comments are stored locally on your device and not visible to other readers.
No comments yet. Be the first to share your thoughts.