13 inference launches do not make one serving stack. Optimise for the bottleneck you can measure.

AWS says SageMaker AI has shipped 13 inference launches so far in 2026 across managed endpoints and HyperPod Inference. The list spans capacity-aware pools, tiered KV caching, intelligent routing and disaggregated prefill and decode.

The obvious response is to assemble the features into a faster serving stack. AWS's own documentation points to a stricter rule: each optimisation targets a different bottleneck, and some add overhead when the workload does not exhibit it.

Cache reuse is a workload property

HyperPod's managed tiered KV cache keeps reusable attention state locally and in a cluster-wide tier, while routing can steer requests towards replicas that already hold relevant prefixes. That is useful for long-context, multi-turn and shared-system-prompt traffic because it avoids recomputing prefixes.

It is not free acceleration. AWS's prescriptive guidance says slower cache tiers add retrieval latency and warns that low-prefix-reuse workloads can lose more to that overhead than they gain. A cache decision therefore starts with measured prefix reuse and hit rate, not a feature flag.

Inference optimisation should follow the measured stall, not the service catalogue.

Prefill and decode only need separating when they interfere

Disaggregated prefill and decode moves compute-bound prompt processing and memory-bandwidth-bound token generation onto separate GPU pools. AWS recommends it for dense models around 70B parameters or larger, long inputs of roughly 4,000 tokens or more, sustained concurrency and moderate or long outputs.

For short prompts, low concurrency or smaller models, AWS says a conventional colocated deployment is simpler and performs well. Disaggregation also introduces another transfer path: KV state has to cross between prefiller and decoder, so queue depth, transfer time and routing decisions become production signals.

Turn inference architecture into a benchmark decision

Before enabling an optimisation, classify the workload with the same trace set: input-token distribution, output length, request concurrency, repeated-prefix rate, time to first token and per-token latency. Then change one serving mechanism and measure the effect against that baseline.

For caching, retain cache hit rate and retrieval latency beside time to first token. For disaggregation, record prefill and decode queues, KV transfer time and the share of requests actually routed through the split path. Cost per successful request belongs beside the latency figures.

The 13 launches matter because inference platforms now expose several independent control knobs. Production teams should resist turning them into a default stack. The architecture decision is which bottleneck the workload proves it has, which mechanism removes it, and which metric will show when that assumption stops being true.

Written by the Devence Lab research team.