Google's two TPUs make the training–inference split explicit

One generation, two priorities

At Cloud Next in April 2026, Google introduced TPU 8t for training and TPU 8i for inference. That split makes a broader industry development visible: building a model and serving it are different infrastructure problems, even when they belong to the same product. The announcement is a platform introduction, not evidence that every configuration is generally available to every customer today. Google: TPU 8t and TPU 8i ↗

Training typically asks how quickly a defined run can finish within a resource budget. A live inference service must also care about response times, unpredictable demand and the cost of serving an acceptable answer. There are exceptions and overlaps, but the purchasing questions are different enough to deserve separate evaluation.

Throughput is not the whole service

Imagine two inference deployments with similar maximum output. One reaches that output only by allowing requests to wait in a large batch. The other handles fewer simultaneous requests but responds within the application's latency target. Which is better depends on whether the work is an overnight job or an interactive service.

That example explains why an impressive accelerator benchmark is not a complete service comparison. Input and output lengths, concurrency, model precision and acceptable response time all need to match. Otherwise, the apparent price advantage may depend on delivering a different experience.

A cloud architecture changes the buying decision

TPUs are also a reminder that some important alternatives to GPUs are obtained as cloud services. Comparing them with an owned server requires more than converting an hourly rate into a hardware purchase price.

Include engineering work, committed-use terms, storage, data movement and the cost of retaining an exit option. Establish whether the intended frameworks and model implementations are supported. Check what happens when demand exceeds the capacity you reserved, or falls below it.

An owned system has its own constraints: capital tied up in equipment, installation work and a finite amount of capacity. A fair comparison states these differences explicitly instead of assuming either ownership or rental is automatically economical.

Divide the workload before choosing the platform

A business does not have to train and serve on identical infrastructure. It does need a reliable process for moving models between environments and checking that quality and performance remain acceptable.

The practical response to Google's two-platform approach is to write two sets of requirements. One should describe development and training; the other should describe the production service. If those requirements lead to different hardware or providers, that can be a deliberate architecture choice. The useful question is what the complete workflow costs and how reliably it runs.