AWS Splits One GPU Into Seven Isolated Partitions on SageMaker, and a Half H100 Beat a Whole A100 by Up to 89%

Amazon SageMaker Inference can now carve a single GPU into as many as seven hardware-isolated partitions, each running a different model with its own compute, memory and memory bandwidth. AWS announced the capability on November 6. It applies only to endpoints that use inference components, and it is configured through two fields rather than a new API surface.

The controls are AcceleratorPartitionLayouts on the production variant in CreateEndpointConfig, which declares how each GPU is carved, and AcceleratorPartitionType on each component, which picks the slice. On an ml.p4d.24xlarge, a layout of mig-4g.20gb, mig-2g.10gb and mig-1g.5gb on each A100 yields 24 partitions across the instance. There are no new IAM actions or API operations, carving adds roughly 17 seconds to provisioning, and the layout is fixed when the endpoint is created, so changing it means building a new endpoint.

Supported hardware is specific: P4d with A100, P5 with H100, G7 with RTX PRO 4500 Blackwell at two partitions, and G7e with RTX PRO 6000 Blackwell at four partitions, in seven regions. AWS says Salesforce helped define the requirements and validated the first implementation.

The headline number comes from AWS's own genai-bench run with Llama-3.1-8B-Instruct. A single mig-3g.40gb partition of an H100 delivered 53 to 89 percent more output tokens per second than a whole A100-40GB at the load where its cost advantage peaked, with 5.6 to 47.5 percent lower time to first token and 18 to 34 percent lower cost per million output tokens above the crossover point. Below that crossover, at concurrency 64 for 100-token prompts and 4 for 8,000-token prompts, the undivided A100 was the cheaper option. These are vendor-published figures and have not been independently reproduced.

The economics explain why the feature exists. Teams increasingly route hard requests to a large model and bulk traffic to a small one, and a dedicated accelerator per model wastes silicon on the small one. Partitioning with a hard boundary also answers a question that time-slicing cannot: another tenant's model cannot read this model's memory, because the split is enforced in hardware rather than by a scheduler.

The constraints are the flip side. Because the layout is locked at creation and only inference-component endpoints qualify, capacity planning has to happen before traffic arrives, and an undersized partition cannot be resized in place. Multi-Instance GPU has existed on bare EC2 instances for years; what changed on November 6 is that it became a managed endpoint option with per-model routing and the console, billing and quotas that come with it.

For teams deploying agents that mix a frontier model with a cheap router or classifier, the practical result is that the cheap tier no longer needs its own card. Whether the 53-to-89-percent gain holds for other model sizes and prompt shapes is exactly the kind of claim that only outside testing settles.