A robot's GPU is not always where inference belongs. Treat placement as a control loop.

Microsoft Research found that smaller onboard GPUs slowed mapping and planning by as much as 383% against an A100 in its physical-AI tests. Navigation lost 30% of timely obstacle detections, while vision-language-action accuracy fell by 50% in the tested constrained configurations. The obvious answer is a bigger GPU on every robot. The research points somewhere more useful: inference placement should be a runtime decision.

Onboard compute is a constraint, not an architecture

Robots have a hard power, thermal and weight budget. The study compared onboard, edge and cloud inference across mapping and planning, navigation and manipulation workloads. Some smaller GPUs could not fit the mobile-manipulation stack at all. Moving inference away from the robot let the system use larger accelerators without carrying their full power draw.

The battery result makes the trade-off operational rather than academic. Researchers replaced onboard GPU compute with a Raspberry Pi 5 that shipped data to remote inference and measured longer operating time. Larger onboard GPUs, including Jetson Thor, could consume enough additional power to cut useful robot runtime by hours in the tested platforms.

Inference placement is part of robot control architecture, not a deployment detail.

Offloading creates a new failure boundary

Remote inference is not automatically better. A manipulation loop that depends on a congested network has exchanged a compute constraint for a communications constraint. The research explicitly identifies network latency, bandwidth and available GPU resources as part of the offload trade-off.

That changes what production evaluation needs to measure. Average model latency is insufficient. Teams need task-level budgets for sensor upload, queueing, inference and action return, then a defined behaviour when any leg misses its deadline. A robot that can call a larger model but cannot fail safely when the link disappears is not a better system.

Benchmark placement per workload

The practical design is hybrid. Keep latency-critical control and safe fallback local; offload workloads whose accuracy or model capacity improves enough to justify the network dependency. Mapping, planning and heavier perception may have different thresholds from manipulation or emergency avoidance.

The released Physical AI Toolchain includes example offload projects for SO-101 and UR10e robots. The useful evaluation is therefore not edge versus cloud as a platform choice. It is a placement matrix: workload, maximum round-trip latency, bandwidth floor, local fallback, accelerator requirement and battery cost.

For physical-AI teams, inference routing should become observable policy. Measure task success and energy alongside latency, then move each model to the execution point that satisfies the robot's control envelope. The robot is the endpoint. It does not have to be the data centre.

Written by the Devence Lab research team.