Google TPU 8t and TPU 8i: Why Splitting Training and Inference Into Two Chips Changes Everything

For seven generations of Google’s Tensor Processing Units, the same chip handled both training large models and running them in production. That approach made sense when models were smaller and the two workloads had broadly similar compute profiles. At Google Cloud Next 2026 on April 22, Google announced that this era is over.

Google unveiled its eighth-generation TPUs as two separate chips: the TPU 8t, purpose-built for training, and the TPU 8i, purpose-built for inference. Each is independently optimized for the radically different computational demands of its workload. The result is up to three times faster model training, 80% better performance per dollar for inference, and the ability to run over one million TPUs in a single logical cluster via the new Virgo Network.

This guide covers what changed, why the architectural split matters, what the actual specifications look like, how these chips compare to NVIDIA’s offerings, and what developers should do right now.

The Problem with One Chip for Two Workloads

Training a large language model and running it in production are fundamentally different computational problems, and that gap has been widening for three years.

Training is throughput-bound. You want to process as many tokens, gradients, and weight updates as possible per second, across as many chips as you can synchronize efficiently. You accept longer job runtimes — days or weeks — in exchange for maximizing throughput. The critical bottlenecks are inter-chip communication bandwidth, memory bandwidth for reading and writing large weight tensors, and scale-out efficiency as you add more chips to the cluster.

Inference is latency-bound. When a user submits a query, the first token must arrive within milliseconds. Subsequent tokens stream at a rate that feels natural, and the system must serve thousands of concurrent users without degrading. The critical bottlenecks are on-chip SRAM (to avoid slow HBM reads on frequently accessed attention patterns), per-request collective communication latency, and network topology optimized for scatter/gather rather than all-reduce.

A single chip optimized for both workloads means accepting compromises on both ends. Training wants more scale-up bandwidth and interchip interconnect; inference wants more on-chip SRAM and a topology that eliminates latency on individual requests. By splitting the two into dedicated chips, Google can optimize each without compromise — and the specifications of the 8t and 8i make clear just how different those optimizations actually are.

TPU 8t: Built for Training at Scale

Specifications

The TPU 8t is designed for one purpose: training the largest models that exist, at the fastest possible throughput.

• HBM: 216 GB per chip at 6.5 TB/s memory bandwidth
• On-chip SRAM: 128 MB
• Compute: Up to 12.6 petaFLOPS of 4-bit floating-point (FP4) per chip
• Chip-to-chip interconnect: 19.2 Tbps — double the previous generation (Ironwood)
• Superpod scale: Up to 9,600 chips with 2 petabytes of aggregate shared HBM

The chip-to-chip interconnect doubling is the most architecturally significant number. Synchronizing gradients across thousands of chips during backpropagation is one of the hardest scaling bottlenecks in distributed training. The previous generation’s interconnect became the ceiling on how efficiently a large superpod could run all-reduce operations. At 19.2 Tbps, that ceiling has been substantially raised.

Superpod Scale

A single TPU 8t superpod now accommodates up to 9,600 chips with two petabytes of aggregate shared high-bandwidth memory. That is a cluster large enough to train models with hundreds of billions of parameters without inter-node communication becoming the primary bottleneck. Google claims this configuration delivers up to three times faster model training compared with Ironwood, and up to 2x better performance per watt.

To put that in practical terms: a training run that took three weeks on the previous generation completes in approximately one week on TPU 8t. A pre-training job that cost $3M on Ironwood would cost under $1.5M for the same result on TPU 8t, assuming comparable pricing. The exact per-chip-hour rates have not yet been announced, but the 80% performance-per-dollar improvement is Google’s stated number.