Distributed AI inference does not reduce to a single tokens-per-second number. Production serving mixes prompt prefill, token-by-token decode, bursty traffic, long contexts, model routing, and a latency budget that users actually notice. A benchmark can still be rigorous, but only if the deployment details travel with the score.
Distributed AI inference needs more than peak throughput
MLPerf Inference exists to make system comparisons more disciplined. Its datacenter suite defines workloads, quality targets, latency constraints, and rules for submitting results. That matters because a score without a workload is marketing, not an engineering decision. MLCommons separates scenarios such as Server and Offline because a service answering independent requests behaves differently from a throughput-oriented batch job.
Take AMD’s published MLPerf Inference 6.0 figure for Instinct MI355X: 100,282 tokens per second in one configuration, with 3.1x the throughput of its earlier MI325X submission. That is a useful published data point. It does not mean an application will return its first token at that rate, nor that a smaller cluster, a different model, or a stricter service-level objective will match it. The result must be read with the model, precision, accelerator count, software stack, scenario, and quality setting attached. See AMD’s MLPerf Inference 6.0 report and the MLPerf Inference Datacenter benchmark for the methodology behind those labels.
There are two broad phases in LLM serving. Prefill processes the input prompt and tends to reward compute, memory bandwidth, and efficient attention kernels. Decode produces the next token repeatedly. Decode often becomes more sensitive to batch shape, key-value cache movement, and request scheduling. A system tuned to post a huge offline total can still feel slow in an interactive coding assistant if time to first token or inter-token latency misses the product target.
Why mixture-of-experts makes topology visible
Model architecture makes the gap wider. A dense model activates most parameters for each token. A mixture-of-experts model uses a router to select a subset of experts. That can lower compute per token, but the selected tokens and activations may need to cross device or node boundaries. In distributed AI inference, that communication path becomes part of the model.
The practical distinction is simple: eight accelerators inside one server do not behave like eight accelerators spread across racks. Within a node, high-bandwidth links and shared switch fabric can move tensors quickly. Across nodes, the serving engine must contend with network adapters, switches, collective communication, congestion, and imperfect placement. AMD’s single-node and distributed MI355X analysis makes this separation explicit. That is the right habit: report topology instead of treating GPU count as the whole configuration.
Long context adds another constraint. The key-value cache grows with sequence length, layer count, hidden size, and active requests. Even when model weights fit comfortably, cache pressure can force paging, reduce effective batch size, or push engineers toward more devices. That changes both cost and tail latency. A 128,000-token prompt test says something important, but it is not interchangeable with a short chat workload.
What developers should measure before choosing a stack
Start from the user action, not the accelerator spec sheet. For a chat product, record time to first token, inter-token latency, p50 and p95 end-to-end latency, request success rate, and generated tokens per second. For document extraction, add prompt length, output length, and pages or documents per hour. For an agent, measure tool-call stalls and queue time too. The same model can look fast or slow depending on which of those constraints matter.
Run a small workload matrix. Use at least a short prompt and a long prompt, low and high concurrency, and the model precision intended for production. Pin the serving runtime version, tokenizer, model revision, and sampling settings. Report the concurrency level rather than hiding it inside a batch size. That turns a one-off benchmark into a result another engineer can reproduce or challenge.
Questions that expose an incomplete claim
| Question | Why it changes the result |
|---|---|
| What model, revision, and precision ran? | Weights, quantization, and kernels alter memory use and quality. |
| Which scenario and latency target applied? | Offline throughput and interactive serving optimize for different behavior. |
| How many nodes and what fabric linked them? | Distributed AI inference exposes network and routing overhead. |
| What were prompt length, output length, and concurrency? | They determine cache pressure, batching, and queueing. |
| Which percentile was reported? | A mean can hide the slow requests that users remember. |
What infrastructure teams should ask
Infrastructure teams need capacity plans rather than a headline result. Separate accelerator utilization from network utilization and host-side bottlenecks. Watch cache hit rate, queue depth, request cancellations, and the fraction of time devices wait on communication. If an MoE workload starts to saturate the fabric before compute, buying more accelerators without revisiting placement may produce a disappointing curve.
There is also a reliability angle. A larger distributed AI inference deployment has more failure domains and more work to rebalance after a device or node disappears. Measure recovery time, not just steady-state speed. Test whether the scheduler keeps sessions near their cached state and whether a rolling upgrade causes a latency spike. Those checks matter for a service that operates all day, not just a benchmark run.
The useful comparison is a workload map: throughput, p95 latency, time to first token, context length, power or rack limits, topology, and quality. One score can signal progress. It cannot make the architecture decision alone. Test the real traffic shape, publish the configuration, and treat the network as part of the inference system. Try visual models such as Nano Banana 2 on Wiro to see how output workflows shape the serving experience.
