Emmett Shear on AI Alignment, AGI, and Why Most AI Companies Are Wrong

Emmett Shear explains AI alignment, why AGI requires theory of mind, and why most AI startups are building the wrong products.

In this episode of the Founders in Arms podcast, we sit down with Emmett Shear, founder and CEO of Softmax, former CEO of Twitch, and former interim CEO of OpenAI, to talk about AI alignment, AGI, and how intelligent systems actually learn.

Emmett argues that most discussions about AI alignment are fundamentally confused, and that alignment is not about enforcing rules—but about building systems that can continuously learn, adapt, and understand other agents.

The conversation also explores why AGI and alignment are the same problem, why current AI approaches may be heading in the wrong direction, and what founders are getting wrong about building AI products.

This conversation dives deep into:

• What AI alignment actually means

• Why alignment is not about rules or morality

• Theory of mind as the foundation of intelligence

• Continuous learning and why it’s so hard

• Why AGI and alignment are the same problem

• The risks of “aligned to me” AI systems

• Why singleton AI is likely the wrong model

• How AI will integrate into human society

• Why most AI startups are building the wrong products

• Treating AI as systems to train, not tools to prompt

In this episode, we cover:

(00:00) The dangerous truth about alignment

Emmett explains that alignment is not inherently “good.”

The same capability that allows coordination and cooperation can also enable large-scale harm.

True alignment increases both the potential for positive and negative outcomes.

(01:13) What Softmax is building

Softmax is focused on understanding alignment as a general problem across systems—not just AI.

The goal is to create environments where agents can learn how to align with each other over time.

(02:05) Why most people misunderstand alignment

Emmett argues that most discussions of alignment lack a clear definition.

The key question is:

Aligned to what?

Without answering that, the concept of alignment is meaningless.

(03:35) Building training environments for alignment

Instead of building a single “aligned model,” Softmax is building learning environments.

These environments:

• Are multiplayer

• Allow cooperation and competition

• Require agents to model each other

This helps develop theory of mind.

(04:00) Theory of mind as a prerequisite

To align with others, an agent must understand them.

That requires:

• Modeling other agents’ goals

• Predicting behavior

• Inferring intent

Without this, alignment is fragile and accidental.

(04:45) Continuous learning is required

Aligned systems cannot be static.

They must:

• Continuously adapt

• Learn from new experiences

• Update their understanding of the world

The world is non-stationary, so alignment must be ongoing.

(06:30) Multiplayer environments vs static training

Softmax uses open-ended environments where agents:

• Compete

• Cooperate

• Solve evolving problems

This better reflects the real world compared to static datasets.

(07:13) Alignment is always relative

Alignment is not universal.

It depends on:

• The agent

• Its relationships

• What it identifies with

Humans align with family, community, and society in different ways.

AI will face the same problem.

(10:55) Cooperation and competition both matter

Real-world environments are not purely cooperative.

Agents must learn to:

• Collaborate when beneficial

• Compete when necessary

Training environments need both dynamics.

(12:59) AGI and alignment are the same problem

Emmett argues that building AGI inherently requires solving alignment.

Key missing capabilities today:

• Theory of mind

• Self-modeling

• Learning from experience

Without these, systems remain fragile.

(15:20) Alignment enables both good and evil

A key insight:

The ability to align systems also enables large-scale coordination.

That coordination can be used for:

• Positive outcomes

• Harmful outcomes

Alignment is a capability, not a guarantee of safety.

(17:14) Why “aligned AI” can be dangerous

When someone says they are building aligned AI, it often means:

Aligned to them.

This creates a concentration of power and risk.

(17:50) Why a single super AI may not dominate

Emmett argues that a “singleton AI” is unlikely.

Instead, we may see:

• Many independent AIs

• Distributed learning systems

• AI societies

This could create more robustness.

(20:44) Alignment must be built through relationships

AI cannot align to “humanity” in the abstract.

It must align to:

• Individual humans

• Direct interactions

• Real relationships

This mirrors how humans develop alignment.

(21:54) Why current AI approaches may be wrong

Many AI companies focus on:

• Bigger models

• More compute

• Longer training

Emmett compares this to building a bigger jet engine instead of designing a new kind of system.

(37:29) The core problem with continuous learning

Training on your own outputs leads to “mode collapse.”

Systems reinforce their own behavior until they become repetitive and less useful.

Solving this requires:

• Better feedback signals

• Understanding what “good” means

(39:30) Why emotions are part of intelligence

Humans rely on:

• Emotions

• Intuition

• Subjective signals

These act as training signals for decision-making.

Pure reasoning is not enough.

(43:07) Advice for AI founders

Emmett’s key advice:

Stop treating AI like a magic tool.

Instead:

• Treat it as something to train

• Build systems that improve over time

• Focus on creating value, not saving labor

(45:10) Why most AI startups are wrong

Many startups focus on:

• Making AI easier

• Reducing effort

• Automating tasks

But the real opportunity is:

Creating entirely new capabilities.

(45:45) The Twitch lesson: people want quality, not ease

At Twitch, making streaming easier didn’t drive growth.

Making it better did.

The same applies to AI:

• Users don’t want easy outputs

• They want high-quality results

(51:01) The “AI slop” problem

Low-effort AI-generated content is a temporary trend.

These products will either:

• Evolve into higher-quality tools

• Or disappear

Key Takeaways for Founders

Alignment is not a fixed goal

It is an ongoing process of adapting to changing environments and relationships.

AGI requires self-awareness

Systems must understand themselves and others to become truly intelligent.

Continuous learning is the core challenge

Training models on static data is not enough for real intelligence.

Alignment increases both risk and potential

The same capability enables both cooperation and large-scale harm.

AI should be trained, not just prompted

The biggest opportunities come from building systems that improve over time.

Focus on value, not automation

The most important AI products will create new capabilities—not just reduce costs.

About the Guest

About Emmett Shear

Emmett Shear is the founder and CEO of Softmax, an AI research company focused on alignment and learning systems.

He previously co-founded Twitch, which was acquired by Amazon, and later served as interim CEO of OpenAI.

Listen to Founders in Arms

Founders in Arms is a podcast for ambitious builders—covering startups, AI, marketplaces, leadership, and the realities of building enduring companies.

🎙 Subscribe to Founders in Arms on your favorite platform
💬 Join the conversation at TribeChat.com
🚀 Discover more insights from founders and operators shaping the future of technology