Google Cloud has released llm-d 1.0, the first stable version of its open-source distributed inference stack. Unveiled at the PyTorch Conference North America, the Kubernetes-native layer unifies vLLM and SGLang model servers into a coordinated pool, targeting self-hosted agent and reinforcement learning workloads where traditional round-robin load balancing fails.
The software was developed by a coalition including Red Hat, IBM, Google Cloud, CoreWeave, and NVIDIA, and operates as a CNCF Sandbox project. Version 1.0 brings multi-tenant quality of service, zero-downtime upgrades, agent-aware session routing, async inference processing, and expanded TPU support. It relies on a router built on the Kubernetes Gateway API Inference Extension, which tracks cached prefixes and routes requests to replicas where context already lives.
In self-reported benchmarks across multi-region deployments, the system achieved significant performance gains. On an Ironwood TPU workload, prefix-aware routing and host memory offloading cut median time-to-first-token by over 85% and increased throughput by 2.4x. Adopters such as Snap and Zenteiq reported higher cache hit rates and multi-fold throughput improvements.
This release matters because it provides production-ready infrastructure to optimize distributed TPU and GPU fleets, dramatically reducing latency and operational costs for self-hosted AI agents.