For prefill/decode disaggregation, Google says median KV transfer latency for Qwen3 235B fell from several seconds to about 230 ms, with average TTFT reductions of 60% for that model and 90% for a 480B model at low concurrency. For prefix caching, it reports up to 90% lower average TTFT on 64K-context workloads.
Developers get Python wheels for JAX and PyTorch, with KV cache manager, KV store, and weight synchronizer APIs. TPU Sync acts as the KV connector for vLLM-TorchTPU through vLLM's KVConnector interface, as the disaggregation and hierarchical-cache backend for SGLang-TorchTPU, and pairs with llm-d 1.0 on GKE for prefix-cache-aware routing.
The RL weight-sync module for Pathways and multi-controller JAX is experimental. The GitHub repository, which absorbed the earlier TPU Raiden codebase, warns that APIs are changing quickly and that only development builds are on PyPI, and recommends pinning to its last-known-good commit.
