Google’s Tunix system lifts the choke point that has kept large-scale agentic reinforcement learning (RL) from using TPUs efficiently. By separating the work of generating interaction data from the work of updating the policy, Tunix drives TPU utilization from single-digit percentages up to near-full capacity, cutting compute waste dramatically.

The bottleneck in agentic RL

Agentic RL differs from the more familiar “next-token” language-model training. An agent must send API calls, execute code, or step through a simulated environment, then react to the result. The training loop is therefore synchronous: the model produces an action, the environment runs, the result returns, and only then does the model receive a gradient update. When a single environment step takes several seconds, the costly TPU hardware sits idle, and reported utilization can dip below 10 %. The inefficiency translates directly into higher cloud bills and slower research cycles.

Tunix’s decoupled architecture

Tunix attacks the problem by pulling the two stages—trajectory generation and policy optimization—onto separate hardware pools.

  • Asynchronous actors run on inexpensive CPUs or GPUs. Each actor continuously interacts with its assigned environment, records actions and observations, and streams the resulting trajectories into a shared store.
  • Continuous learners occupy dedicated TPU Pods. The learner pulls batches from the central buffer and performs gradient updates without waiting for any single actor to finish a rollout.
  • High-throughput buffer sits in the middle, acting as a staging area for trajectories. Because the learner can read as fast as the buffer can supply data, the TPU never stalls.

The net effect is a training pipeline where TPUs stay busy almost all the time, pushing utilization toward 100 %.

Technical hurdles and how Tunix overcomes them

Variable-length episodes and XLA recompilation

JAX’s XLA compiler optimizes for fixed tensor shapes. Agentic tasks, however, produce sequences of differing lengths, which would normally trigger costly recompilations. Tunix packs shorter sequences together and groups episodes of similar length into buckets, keeping shapes stable long enough for XLA to reuse compiled kernels. The result is steady throughput without the compiler overhead that would otherwise cripple performance.

Scaling massive models across many TPU chips

Training agents with more than 70 billion parameters requires spreading weights and data across multiple TPU nodes. Tunix uses JAX’s ShardMap primitive to shard both model parameters and activations, allowing the learner to keep the entire model in memory while still feeding it data at high speed. This sharding strategy makes it possible to train models that were previously out of reach for a single TPU pod.

Stale gradients from decoupled pipelines

When actors run ahead of the learner, the data they supply can become “stale” relative to the current policy. Tunix mitigates this drift with two mechanisms: importance-sampling reweights older samples to reflect their relevance, and a configurable staleness threshold discards trajectories that exceed a preset age. Together they keep learning stable even as the pipeline runs asynchronously.

What adopters need to watch

  • Latency audit – The benefit of decoupling hinges on the environment’s response time. Teams should measure end-to-end latency and ensure that actor pools are sized to keep the buffer well-filled.
  • Worker pool design – Cheap CPUs or GPUs can host many actors, but oversubscribing them can cause contention on network or storage. A balanced pool that matches the buffer’s ingest rate is essential.
  • Buffer robustness – The central store must handle high write and read rates without becoming a new bottleneck. Choosing a storage system with low tail latency and sufficient bandwidth is a non-negotiable part of the architecture.

Potential downsides

The split architecture introduces more moving parts: separate hardware fleets, a persistent buffer, and coordination logic to enforce staleness limits.

Takeaway

Tunix shows that the dominant cost in agentic RL is not the model itself but the idle time caused by synchronous interaction loops. By offloading rollout work to cheap hardware and feeding a continuously learning TPU pod from a high-throughput buffer, Google has turned a sub-10 % utilization problem into a near-full-capacity workflow.