SageMaker's GPU slices move the cost test to workload shape

AWS now lets real-time SageMaker endpoints divide supported GPUs into isolated partitions. The saving depends on concurrency, context length and a fixed endpoint layout, so benchmark the slice as its own capacity unit.
AWS added Multi-Instance GPU partitioning to SageMaker Inference on November 6. In its test, half of an H100 delivered 53–89% more output tokens per second than a full A100 at the load points where its cost benefit was largest. That result changes the capacity comparison, but it is not a universal discount.
An inference component can now reserve a fixed share of compute and GPU memory. Under AWS's illustrated layout, a P4d instance with eight A100s can expose 24 partitions, allowing more separately loaded models than physical accelerators. The gain is useful for portfolios of smaller models that would otherwise leave a full device idle.
The crossover belongs in the benchmark
The AWS comparison ran Llama-3.1-8B-Instruct on an H100 `mig-3g.40gb` partition and a whole A100-40GB across five traffic shapes. Above its reported crossover, the partition was 18–34% cheaper per million output tokens. For 100-token prompts, the crossover came at concurrency 64; for 8,000-token prompts, at concurrency 4. Below those points, the whole A100 was cheaper in this test.
Those numbers are a sizing example, not a price promise. The company derived unit cost from the full instance price, while customers still pay for the full instance. A team must fill the other slices, or the theoretical partition saving can disappear in idle capacity.
A cheap GPU slice is only cheap when the endpoint's remaining slices have useful work.
Memory and layout set the real limit
A model's weights are only part of GPU memory demand. The AWS guide calls out KV cache growth with context length and concurrent sequences, plus runtime overhead. A partition that loads an 8B model may still fail under production traffic if it was sized only for the weights. Test first-token latency, output throughput, request failures and p99 latency at the expected mix of prompts and outputs.
The carve is fixed when the endpoint is created. Changing the layout requires a new endpoint and traffic migration; moving a model to a different profile requires a new component. A stable set of models suits this design better than a portfolio that changes every day.
Treat isolation as a scoped claim
MIG separates GPU compute and memory in hardware. The AWS post is explicit that models on the same instance still share host CPU, memory, storage, networking and endpoint routing. Tenant reviews therefore need to map both the GPU partition and those shared resources.
The practical decision is to list partition profiles beside whole accelerators as distinct deployment options. Measure each against the real request distribution, account for the full instance bill and migration cost, then choose the layout that keeps the entire endpoint productive.
Sources
Written by the Devence Lab research team.
