Mira Murati's TML-Interaction-Small hits 0.4s latency — 3x faster than GPT-Realtime-2.0. Full architecture breakdown and what it means for voice agent developers.
GPT-Realtime-2.0 averages 1.18 seconds from the end of your utterance to first audio token. Gemini 3.1 Flash Live clocks 0.57 seconds. Thinking Machines Lab’s first public model — TML-Interaction-Small — hits 0.40 seconds. But the latency number is almost the wrong frame for what Thinking Machines actually shipped.
The architecture Mira Murati’s lab announced on May 11, 2026, is not a faster realtime API. It is a fundamentally different model class. An interaction model does not wait for you to stop talking before it processes what you said. It listens and reasons simultaneously. It can interrupt, redirect, and respond mid-sentence — the same way a human conversation partner does. Every major voice AI system today is half-duplex masquerading as conversation. Thinking Machines shipped something that is architecturally full-duplex.
Here is what this means for voice and multi-modal agent architecture, what FD-bench actually measures, and what developers can — and cannot — build with it right now.
Try it yourself: Free AI Token Counter — free, no signup, runs in your browser.
What “Interaction Models” Actually Are
The canonical request-response loop that underlies every AI assistant — user speaks, AI listens, AI processes, AI responds — is a carry-over from text interfaces. Text is naturally turn-based. Audio is not. Human conversation is full-duplex: both parties process incoming signals while simultaneously preparing responses.
The current generation of voice AI works around this by stitching together a voice activity detector (VAD), a transcription model, an LLM, and a TTS pipeline. The stitching introduces latency at every seam. More importantly, the end-to-end system fundamentally cannot respond until your turn ends. The VAD has to detect silence, the transcription model has to convert audio to tokens, and only then does the LLM start processing.
Thinking Machines Lab’s position is that this stitched architecture is not an implementation problem that can be optimized away. The constraint is structural. A model trained on discrete text tokens cannot natively reason over streaming audio — it has to wait for the stream to be transcribed first, and that transcription step is where latency accumulates regardless of how fast each component runs.
Their interaction model is trained to reason over raw streaming audio and video in real time. No transcription preprocessing step. The output is also generated as a native audio stream, not synthesized from text tokens after the fact. Both input and output stay in the continuous-signal domain throughout processing. The model’s context is a rolling temporal window rather than a fixed token buffer.
The Dual-Component Architecture
Thinking Machines describes the system as two coupled components that share full conversation context throughout a session. Understanding how these two components divide responsibility is the key to understanding why the architecture works.
TML-Interaction-Small (foreground model): A 276-billion parameter Mixture-of-Experts architecture with 12 billion parameters active at any inference step. This is the component that maintains presence — it processes incoming audio and video continuously, tracks conversational state, handles turn-taking and interruption detection, and generates the immediate response stream. At 12B active parameters, it runs fast enough to sustain the 200ms micro-turn latency that enables genuine overlap between listening and speaking.
Background Model (asynchronous reasoning): A second, larger model that handles sustained reasoning, tool use, web search, and longer-horizon tasks. When a query requires genuine deliberation — a calculation, a code generation request, looking up current information — the foreground model hands off context to the background model, which works asynchronously. The foreground model maintains the conversational thread (acknowledging, asking clarifying questions, providing partial responses) while the background model works on the harder task.
# Conceptual architecture (from Thinking Machines Lab blog post)
# Source: thinkingmachines.ai/blog/interaction-models/
User audio/video stream
│
▼
TML-Interaction-Small (12B active, MoE)
│
├── Turn-taking, interruption, immediate response (200ms micro-turns)
│
└── Routes complex requests ──────────────────▶ Background Model
│
└── Tool calls, search,
code gen, reasoning
(async, full context)This separation is the architectural insight. Previous realtime APIs tried to compress both presence and reasoning into a single model under a single latency budget. The result is a forced tradeoff: either the model is fast (shallow reasoning) or thorough (slow to respond). Thinking Machines decoupled the two concerns entirely. The foreground model is optimized for presence and latency. The background model is optimized for reasoning quality. They share context but operate on separate constraints.
The analogy that clarifies this: the foreground model is a skilled conversationalist who keeps the room engaged while a colleague does the research. The colleague’s quality of research is independent of how fast they deliver the answer to the room.
Comments · 0
Beta: comments are stored locally on your device and not visible to other readers.
No comments yet. Be the first to share your thoughts.