Self-hosted inference wastes money in a specific, repeatable way. A long agent session resumes on a different GPU, and that GPU holds no key-value state for the history it is being handed, so it recomputes attention keys and values for every prior token before generating one new one. Local prefix caching only helps while session affinity keeps the request on the same host; routing changes, autoscaling and failures break that affinity several times an hour. On November 20, AWS published a fix packaged as an agent skill rather than a product.
The skill lives in the Agent Toolkit for AWS and is designed to be driven by coding agents that implement the Agent Skills specification, including Kiro, Claude Code and OpenAI Codex. It walks through four stages: a read-only fit check that inspects the GPU fleet, counts Elastic Fabric Adapter interfaces and collects what the API cannot report, such as engine, tensor-parallel size, KV data type and prompt repetition rate; an economic gate that replays recorded cache events through a tool called KVTrace before any hardware is bought; the integration itself, which adds the open-source LMCache connector to an existing vLLM or SGLang deployment including workers under llm-d and Ray Serve; and troubleshooting, starting with the RDMA write failures that a misconfigured security group produces.
The numbers AWS attaches are ceilings, not promises. Distributed KV caching with Amazon ElastiCache can cut inference cost by up to 40% and time to first token by up to 80%, on workloads with long contexts and heavy prompt reuse. The company's worked example is a legal assistant serving 64,000-token case histories on eight p5 instances with a 13-second time to first token, where the skill proposes two cache.i8ge.48xlarge nodes in the same availability zone and connects them over RDMA so any worker can load the state directly.
The prerequisites are narrow enough to matter. GPU instances must support EFA RDMA write, which in practice means p5-class hardware; the connector needs LMCache 0.5.8rc1 or later; the serving engine needs vLLM 0.11.1 or later, or SGLang 0.5.13 or later; and the cache tier needs ElastiCache for Valkey 9.1 or later on cache.i8ge.48xlarge nodes. Teams without that volume are better served by a managed API, and AWS says as much implicitly by putting a savings gate in front of provisioning: if the modelled savings do not cover the cache nodes, the agent is supposed to stop.
Two guardrails deserve attention because they point at failure modes that do not announce themselves. KVTrace records block hashes and counts rather than token IDs or prompt text, which keeps the ROI estimate from becoming a logging incident. And the skill forces a single KV ID hash mechanism and an explicit KV data type across all servers, because a mismatch produces the worst possible outcome - healthy hit counters serving the wrong cached state. Before production traffic it empties local tiers and verifies that two cache-loaded runs return identical output. Every create, change or delete requires explicit human approval.
There are also cases where the cache loses. Below a cutover prompt length, reading cached state is slower than recomputing it, which is why the sizing step exists. Multi-tenant deployments can leak information through timing, so AWS recommends separate clusters or tenant-specific hash salts for sensitive workloads. And the entire saving only becomes money when a higher hit rate turns into fewer instances or more traffic on the same footprint.
The strategic read is less about caching than about who operates infrastructure. AWS has shipped a procedure - diagnose, model, provision, validate - as a skill an agent executes conversationally, with the human reduced to an approval gate on resource changes. That pattern will spread, and it moves the cloud competition from how many GPUs a provider can offer to how much redundant work it can talk a customer out of doing. It also introduces a new trust boundary: the thing that decides how much hardware you buy is now a model reading your fleet.
Comments (0)
Log in to join the discussion
Log InNo comments yet