Linkerd Maker Buoyant Open-Sources Kinference, a Control Plane That Cut Median TTFT 95 Percent on Shared-Prefix Traffic

Buoyant, the company that built the Linkerd service mesh, has open-sourced Kinference, an inference control plane that turns Kubernetes plus vLLM into a multi-tenant serving platform. The project, released under Apache 2.0 at version v0.1.0-alpha.7 and written in Rust, draws a clean line: vLLM is the data plane doing the generation, Kinference is the control plane deciding where each request goes, applying routing policies, token rate limits, KV-cache-aware load balancing and per-tenant metrics in one layer above the model servers.

Architecturally, a traffic gateway works with an inference handler that exposes a gRPC interface: for each request it returns a chosen endpoint, and may rewrite the request body along the way. The controller watches vLLM's KV-cache events and engine metrics, currently using NVIDIA's dynamo-kv-router crate to process cache events, and predicts time to first token for each candidate endpoint from cache overlap and queue length. Configuration lives in five Kubernetes custom resources, and policies can be kept in git and validated offline with a kinfctl CLI.

The documented policy patterns read like a field guide to real inference operations: route low-complexity prompts to a smaller model unless cache overlap makes the big one faster; skip workers with more than two queued requests and fall back to external providers such as Bedrock; fail closed and keep traffic local when a PII tag fires; admit batch jobs only on idle GPUs below 70 percent KV-cache usage; and publish one stable model name backed by several backends.

Buoyant's benchmark numbers are self-reported and deserve the usual caution. On a shared-prefix workload, median TTFT fell from 6.88 seconds to 0.31 seconds, about 95 percent, and on multiturn traffic from 1.70 seconds to 0.59 seconds, about 65 percent; peak goodput meeting the tested latency SLOs rose 7.4x and 3.5x respectively. But on prompts with no shared prefix, median TTFT showed no improvement at all, 6.92 seconds versus a 6.83 second baseline. The test used two pools of three vLLM 0.31.0 pods, one L40S GPU each, serving Qwen3.8-27B-FP8, and the baseline was plain random routing through a Kubernetes Service, not another cache-aware router.

The alpha status matters too: token rate limiting and semantic classifiers are not yet implemented despite appearing in policy examples, Anthropic-compatible API support is pending, and the controller runs as a single replica. Even so, the layer Kinference occupies is where inference economics increasingly get decided, and the company's argument is that teams today solve routing with hand-written proxies nobody else can reuse. Whether a general-purpose control plane can beat the tuned stacks inside model gateways is the comparison the first serious third-party benchmarks will settle.