Google TPU 8t and TPU 8i: Why Splitting Training and Inference Into Two Chips Changes Everything

TPU 8i: Built for Inference at Scale

Specifications

The TPU 8i is designed for one purpose: serving millions of concurrent inference requests with the lowest possible latency.

• On-chip SRAM: 384 MB — three times more than the TPU 8t
• HBM: 288 GB per chip at 8.6 TB/s memory bandwidth
• Compute: 10.1 petaFLOPS of FP4 per chip
• Pod scale: 1,152 chips per pod
• New components: Collectives Acceleration Engine (CAE) + Boardfly network topology

The 384 MB of on-chip SRAM is the headline number. On-chip SRAM is an order of magnitude faster to access than HBM — lower latency, no bus contention, no DRAM timing overhead. For a large model serving billions of daily queries, the fraction of the model that fits in on-chip SRAM determines tail latency. Attention layers and frequently accessed MLP blocks can live in SRAM during serving, avoiding the slow trip to HBM on every request.

The TPU 8i also carries more HBM bandwidth (8.6 TB/s) than the TPU 8t (6.5 TB/s), despite having less total HBM capacity. This reflects inference’s memory-bandwidth-bound nature: inference reads weights once per forward pass and is often not reusing cached values the way training does, so raw bandwidth matters more than total capacity.

Boardfly and the Collectives Acceleration Engine

Google introduced two new components for the TPU 8i that have no equivalent in previous generations:

Boardfly is a new network topology specifically designed for serving workloads. Training topologies optimize for all-reduce operations across large superpods, where all chips share gradient updates simultaneously. Inference has a different pattern: each incoming request touches a specific subset of chips, needs a fast response, and then the chips move to the next request. Boardfly minimizes the communication latency for these scatter/gather patterns, reducing the per-request overhead that accumulates at high concurrency.

The Collectives Acceleration Engine (CAE) offloads collective communication operations from the main tensor cores entirely. Without CAE, collective operations (like the all-gather needed during tensor-parallel inference) stall the compute units while they wait for data. CAE runs these collectives on dedicated silicon in parallel with tensor core computation, reducing effective inference latency by eliminating those stalls.

The Virgo Network: Connecting It All

Both chips run on Google’s new Virgo Network, a megascale data center fabric announced alongside the TPUs at Cloud Next 2026. Virgo is designed to allow over one million TPUs to operate as a single logical cluster — not across the public internet, but within Google’s private data center network with performance characteristics closer to on-chip interconnect than to WAN networking.

From a developer perspective, Virgo is largely invisible. You do not configure it; you benefit from it through Google Cloud’s AI Hypercomputer platform. But it matters because it is the reason the TPU 8t superpod’s 9,600-chip scale is actually usable in practice. When the inter-pod interconnect is fast enough, scaling from 1,000 chips to 9,600 chips delivers near-linear throughput gains. When it is not, scaling efficiency degrades rapidly and the additional chips produce diminishing returns.

Google claims that Virgo-backed AI Hypercomputer clusters can achieve linear scaling efficiency up to the full one-million-chip scale, which would be an unprecedented result if it holds in production workloads.

What This Means for AI Developers

The TPU 8t/8i split has different implications depending on which part of the AI lifecycle you care about.

If You Train Foundation Models or Run Large Fine-Tunes

The TPU 8t’s 3x training throughput is the most cost-relevant announcement for your workload. Timeline compression at this scale has compounding effects: faster iteration means more experiments per dollar, which means better models per unit of budget. Teams training at the 100B+ parameter scale on Google Cloud should be on the TPU 8t waitlist now.

If You Run Production Inference

The TPU 8i’s 384 MB of on-chip SRAM is the number that matters most. If you serve Gemini-class or similar large models, first-token latency is directly tied to how much of the model’s hot path fits in fast memory. At 3x the on-chip SRAM of the training chip, the 8i can hold substantially more of the model in fast memory than previous generations. Pair that with the Boardfly topology’s lower per-request overhead, and p99 latency at high concurrency improves meaningfully.

If You Use Google Cloud Managed AI Services

Google runs Gemini API, Vertex AI predictions, and its other managed AI services on its own TPU fleet. As the 8th generation rolls out internally, the services you call will become faster and cheaper without any changes on your end. The 16 billion tokens per minute that Google Cloud already processes will scale further as the new generation comes online.

If You Are Currently on NVIDIA GPUs

This announcement is not an immediate reason to switch. The H200 NVL and upcoming Blackwell Ultra are well-matched competitors, and the CUDA ecosystem’s maturity, library coverage, and developer familiarity remain real advantages. PyTorch on TPUs has improved significantly in 2025–2026 but is not identical to native CUDA performance. For teams already heavily invested in CUDA tooling, the switching cost is real.

That said, the TPU 8t/8i performance-per-dollar numbers are competitive in a way that previous generations were not always clearly competitive. If your team is starting a new large training run or building a new inference-at-scale system and is not yet locked into NVIDIA, it is worth comparing the current pricing once Google Cloud rates are published.

Google vs. NVIDIA: What the Chip War Looks Like in 2026

The framing of Google’s TPU announcements as “competing with NVIDIA” is accurate but slightly incomplete. Google is competing with NVIDIA for cloud AI workloads that Google Cloud can capture directly. But it is also competing against the scenario where enterprises build private GPU clusters rather than using managed cloud at all.

NVIDIA’s advantage is ecosystem depth. CUDA, cuDNN, NVLink, and the broader software stack have years of tooling, benchmarks, and developer familiarity. The TPU ecosystem requires JAX or TensorFlow for optimal performance, and while PyTorch via XLA has improved substantially, there is still a learning curve for teams moving from pure CUDA workflows.

Google’s advantage at the top end is scale. If you need a 9,600-chip training run with 2 petabytes of shared HBM, operating on a single logical cluster via a high-bandwidth fabric, the number of technically viable options in the world is very small. Google Cloud with AI Hypercomputer is one of them. For the organizations running the largest training workloads on Earth — which increasingly means the organizations building the next generation of frontier models — Google has made a compelling case that TPU 8t is the best option available.

How to Get Access

Both chips are available through Google Cloud’s AI Hypercomputer platform. Here is the current access path:

• Interest registration: Google has opened a sign-up page at cloud.google.com/resources/tpu-interest for both the TPU 8t and 8i
• Priority access: Existing Cloud TPU quota holders will receive priority access as the generation rolls out
• GA timeline: General availability for both chips is expected in the second half of 2026
• Supported frameworks: TensorFlow, JAX, and PyTorch via XLA are all supported with the full managed stack
• Indirect access now: Gemini 3.1 Ultra (which runs on TPU 8i internally) is available via the Gemini API today

The Bottom Line

The TPU 8t and TPU 8i represent a genuine architectural maturation in Google’s chip strategy. The decision to split training and inference into dedicated hardware is not about throwing more transistors at the problem — it is a recognition that the two workloads have diverged to the point where the compromises of a unified chip are no longer acceptable at scale.

The numbers make the case concretely: 3x faster training, 80% better performance per dollar, 384 MB of on-chip SRAM for inference, and a new network topology designed specifically for the communication patterns of serving at million-query-per-second scale. These are not marginal improvements on existing chips — they are the result of several years of workload specialization research finally reflected in silicon.

For developers, the practical signal is simple: training and inference now have different optimal hardware, and the gap will only grow. Teams that size their Google Cloud infrastructure without distinguishing between the two workloads will leave performance and budget on the table. The era of one chip for all AI is ending. Build your infrastructure accordingly.