Opinion OpenAI Scores Its Own Models on 99 Drug-Chemistry Problems, and Compound Selection Is Where They Stall Coding Assistants AWS Puts Distributed KV Caching Behind an Agent Skill: Up to 40% Cheaper Inference and 80% Faster First Token AI Agents Tencent and King's College London Build an Agent Memory Test From 80,299 Real Group-Chat Messages News Tesla Starts FSD v15 With a Generative Reality View and a 25x Intervention Claim It Has Not Published Voice & Audio OpenAI Opens GPT-Live-1.1 to the API and Prices Silence at 2 Cents a Minute Coding Assistants Replit Pitted 32 Claude Agents Against NumPy Tuning: 18.9x Speedup While GPT-6.2 Swarms Collapsed at Scale Security GAO Covert Tests: All Four Top Chatbots Produced Viable Attack Plans in Under 12 Hours Once Guardrails Fell Meta Meta Puts Its 2030 Net-Zero Target Under Review as AI Data Centers Push Net Emissions Up 45 Percent Opinion OpenAI Scores Its Own Models on 99 Drug-Chemistry Problems, and Compound Selection Is Where They Stall Coding Assistants AWS Puts Distributed KV Caching Behind an Agent Skill: Up to 40% Cheaper Inference and 80% Faster First Token AI Agents Tencent and King's College London Build an Agent Memory Test From 80,299 Real Group-Chat Messages News Tesla Starts FSD v15 With a Generative Reality View and a 25x Intervention Claim It Has Not Published Voice & Audio OpenAI Opens GPT-Live-1.1 to the API and Prices Silence at 2 Cents a Minute Coding Assistants Replit Pitted 32 Claude Agents Against NumPy Tuning: 18.9x Speedup While GPT-6.2 Swarms Collapsed at Scale Security GAO Covert Tests: All Four Top Chatbots Produced Viable Attack Plans in Under 12 Hours Once Guardrails Fell Meta Meta Puts Its 2030 Net-Zero Target Under Review as AI Data Centers Push Net Emissions Up 45 Percent

AWS Puts Distributed KV Caching Behind an Agent Skill: Up to 40% Cheaper Inference and 80% Faster First Token

AWS Puts Distributed KV Caching Behind an Agent Skill: Up to 40% Cheaper Inference and 80% Faster First Token

AWS shipped an Agent Toolkit skill that has coding agents size, provision and validate a shared KV cache on ElastiCache beside self-hosted vLLM or SGLang fleets. AWS cites up to 40% lower inference cost and 80% lower time to first token, on workloads with heavy prompt reuse.

Self-hosted inference wastes money in a specific, repeatable way. A long agent session resumes on a different GPU, and that GPU holds no key-value state for the history it is being handed, so it recomputes attention keys and values for every prior token before generating one new one. Local prefix caching only helps while session affinity keeps the request on the same host; routing changes, autoscaling and failures break that affinity several times an hour. On November 20, AWS published a fix packaged as an agent skill rather than a product.

The skill lives in the Agent Toolkit for AWS and is designed to be driven by coding agents that implement the Agent Skills specification, including Kiro, Claude Code and OpenAI Codex. It walks through four stages: a read-only fit check that inspects the GPU fleet, counts Elastic Fabric Adapter interfaces and collects what the API cannot report, such as engine, tensor-parallel size, KV data type and prompt repetition rate; an economic gate that replays recorded cache events through a tool called KVTrace before any hardware is bought; the integration itself, which adds the open-source LMCache connector to an existing vLLM or SGLang deployment including workers under llm-d and Ray Serve; and troubleshooting, starting with the RDMA write failures that a misconfigured security group produces.

The numbers AWS attaches are ceilings, not promises. Distributed KV caching with Amazon ElastiCache can cut inference cost by up to 40% and time to first token by up to 80%, on workloads with long contexts and heavy prompt reuse. The company's worked example is a legal assistant serving 64,000-token case histories on eight p5 instances with a 13-second time to first token, where the skill proposes two cache.i8ge.48xlarge nodes in the same availability zone and connects them over RDMA so any worker can load the state directly.

The prerequisites are narrow enough to matter. GPU instances must support EFA RDMA write, which in practice means p5-class hardware; the connector needs LMCache 0.5.8rc1 or later; the serving engine needs vLLM 0.11.1 or later, or SGLang 0.5.13 or later; and the cache tier needs ElastiCache for Valkey 9.1 or later on cache.i8ge.48xlarge nodes. Teams without that volume are better served by a managed API, and AWS says as much implicitly by putting a savings gate in front of provisioning: if the modelled savings do not cover the cache nodes, the agent is supposed to stop.

Two guardrails deserve attention because they point at failure modes that do not announce themselves. KVTrace records block hashes and counts rather than token IDs or prompt text, which keeps the ROI estimate from becoming a logging incident. And the skill forces a single KV ID hash mechanism and an explicit KV data type across all servers, because a mismatch produces the worst possible outcome - healthy hit counters serving the wrong cached state. Before production traffic it empties local tiers and verifies that two cache-loaded runs return identical output. Every create, change or delete requires explicit human approval.

There are also cases where the cache loses. Below a cutover prompt length, reading cached state is slower than recomputing it, which is why the sizing step exists. Multi-tenant deployments can leak information through timing, so AWS recommends separate clusters or tenant-specific hash salts for sensitive workloads. And the entire saving only becomes money when a higher hit rate turns into fewer instances or more traffic on the same footprint.

The strategic read is less about caching than about who operates infrastructure. AWS has shipped a procedure - diagnose, model, provision, validate - as a skill an agent executes conversationally, with the human reduced to an approval gate on resource changes. That pattern will spread, and it moves the cloud competition from how many GPUs a provider can offer to how much redundant work it can talk a customer out of doing. It also introduces a new trust boundary: the thing that decides how much hardware you buy is now a model reading your fleet.

Comments (0)

Log in to join the discussion

Log In

No comments yet