AI inference hardware is becoming its own product category, not a leftover use for training GPUs. Serving a modern reasoning model means loading large weights, managing a growing KV cache, and returning the first token quickly enough that a user does not feel the system stall. That changes what infrastructure teams buy and how developers measure success.
Why AI inference hardware changed
Training still consumes enormous clusters, but it is a bounded job. Inference repeats for every user request after a model ships. A useful assistant may read a long chat history, retrieve documents, call tools, then generate hundreds or thousands of reasoning tokens. The machine has to serve that work at a predictable latency while many other requests compete for the same memory and network.
That is why peak FLOPS no longer tells the whole story. MLPerf Inference v5.0 added a 405-billion-parameter Llama 3.1 workload with a six-second time-to-first-token target and a 175 ms time-per-output-token target. Its Llama 2 70B Interactive scenario tightens those targets to 450 ms and 40 ms, or 25 tokens per second per user. Those numbers make the product problem concrete: a system can have huge aggregate throughput and still feel slow if queueing or token latency misses the service target. NVIDIA’s MLPerf v5.0 breakdown explains the scenarios and constraints.
AI inference hardware therefore gets judged on a different scorecard. Teams care about tokens per second at a stated concurrency, time to first token, tail latency, memory headroom, and energy per useful response. A low-precision tensor core helps, but only if the runtime can batch requests, place model shards sensibly, and move state without turning the interconnect into a traffic jam.
The rack is now the computer
The clearest evidence sits in rack-scale systems. Microsoft Azure’s production GB300 NVL72 cluster for OpenAI connects more than 4,600 Blackwell Ultra GPUs. Each liquid-cooled GB300 rack combines 72 GPUs and 36 Grace CPUs. NVIDIA lists 37 TB of fast memory and 1.44 exaflops of FP4 Tensor Core performance for the system.
The topology matters as much as the headline figures. Inside a GB300 NVL72, the NVLink Switch fabric provides 130 TB/s of all-to-all bandwidth across the 72 GPUs. Beyond the rack, the Azure deployment uses Quantum-X800 InfiniBand, with 800 Gb/s per GPU. That split is purposeful: scale-up traffic keeps a tightly coupled model responsive within a rack, while scale-out traffic connects many racks for larger capacity. NVIDIA’s Azure cluster overview details the system.
This is a different design target from putting the fastest available card in a standard server. A large model may split its layers across accelerators. Mixture-of-experts routing, attention state, and collective operations can all introduce communication. Every slow boundary shows up as idle compute or worse user latency. Rack-scale AI inference hardware treats the GPU, CPU, switch, network interface, power delivery, and cooling loop as one system because the request crosses all of them.
Why that changes product behavior
Model quality decides whether an answer is useful. Infrastructure decides whether that answer arrives fast enough to support an interactive workflow. Faster first-token delivery makes chat and coding tools feel present. Stable decode speed makes longer reasoning traces less frustrating. Better throughput lets a product admit more simultaneous users before it must shed load or shorten context. The same logic applies to image editing and video generation, where large reference inputs and repeated iterations can make waiting the dominant part of the experience.
Memory is the quiet constraint in AI inference hardware
Inference often hits memory capacity or bandwidth before it exhausts arithmetic throughput. Model weights must be read repeatedly. The KV cache holds prior attention keys and values so the model can generate the next token without recomputing the entire conversation. Longer prompts, retrieved documents, and agent traces raise that memory demand. Quantization reduces weight size, but it does not make context state disappear.
It helps to separate a request into two phases. Prefill processes the prompt and builds the KV cache. It tends to need high compute and memory bandwidth. Decode generates one token at a time from that state, so response latency becomes sensitive to memory movement, batching policy, and cache placement. Mixing long prefill jobs with latency-sensitive decode traffic on one queue can hurt both. Modern serving stacks often separate or schedule the phases differently for that reason.
Microsoft’s Maia 200 makes the memory-first design visible. The accelerator pairs FP8 and FP4 tensor cores with 216 GB of HBM3e, 7 TB/s of memory bandwidth, and 272 MB of on-chip SRAM. Microsoft says the chip uses more than 140 billion transistors and targets token-generation economics, not a generic graphics workload. Microsoft’s Maia 200 announcement lays out those specifications.
What developers and infrastructure teams should do
Start with the request shape, not a chip shortlist. Record prompt-token percentiles, output-token percentiles, context length, concurrency, and latency targets. A support bot with short replies has a different bottleneck from a coding agent that carries a large repository context. One average tokens-per-second result cannot represent both.
- Measure TTFT and TPOT separately. Report p50 and p95, then test at the concurrency expected in production. Aggregate throughput alone hides a bad interactive experience.
- Track KV-cache pressure. Watch cache occupancy, eviction rate, and the share of requests with long prompts. Those signals explain latency spikes better than GPU utilization by itself.
- Test precision on the actual model. FP4, FP8, and other formats can improve capacity and speed, but evaluate answer quality, tool-call reliability, and structured-output accuracy after conversion.
- Design for failure domains. Keep a single rack or node failure from taking every replica offline. Capacity planning needs headroom for maintenance and traffic bursts, not only a perfect benchmark run.
- Compare systems with the same service-level objective. State the model, context length, batch policy, concurrency, accuracy target, power boundary, and software version. Without those details, token-cost claims are hard to compare.
The practical point is simple. AI inference hardware is not just a faster accelerator. It is a computer organized around the life of a request: prefill, cache, decode, route, and repeat. Teams that profile those stages will make better architecture choices than teams that shop by peak compute alone.
Want to feel the application side of faster generative inference? Run Nano Banana 2 and other models on Wiro.
