Crusoe's MLPerf Inference v6.1 results on NVIDIA GB200 NVL72

Today, MLCommons published the MLPerf® Inference v6.1 results, and with them Crusoe's first-ever MLPerf submission. We submitted results for gpt-oss-120b and Qwen3-VL-235B-A22B running on NVIDIA GB200 NVL72 on Crusoe Cloud, validating that the NVIDIA Accelerated Computing Platform, from NVIDIA CUDA® and NVIDIA TensorRT-LLM through NVIDIA Dynamo, NVIDIA NVLink™, and NVIDIA Quantum InfiniBand, runs exactly as our customers consume it.

Three results matter more than the raw numbers:

• We exceeded NVIDIA's published single-node reference for Qwen3-VL-235B-A22B. NVIDIA's guidance for a 4x GB200 NVL72 node was roughly 56 samples/s Offline and 46 queries/s Server. Our submission measured 58.40 and 49.70. Our gpt-oss-120b submission is likewise the highest per-GPU GB200 NVL72 result in the closed division, at roughly 12,700 tokens per second per GPU.
• Scaling from one node to four was effectively linear. Qwen3-VL Server throughput at 16 GPUs came in at 4.00x the 4-GPU result, and Offline landed within a fraction of a percent of the linear extrapolation. Rack-scale NVLink and NVIDIA Quantum InfiniBand make the fabric a non-factor for this workload shape.‍
• A topology tuning pass produced a ~2.5x Interactive throughput gain on identical hardware. By rebalancing the prefill:decode GPU split in a disaggregated NVIDIA Dynamo deployment from 75:25 to 50:50, Qwen3-VL Interactive throughput on 16 GPUs went from 15.85 to 39.85 queries/s. Same GPUs, same fabric, dramatically more serving capacity.

This post covers the results, the methodology behind them, and the engineering it took to get a multi-node prefill/decode-disaggregated serving topology through MLPerf's closed-division rules. A companion post covers our AMD MI355X submission.

MLPerf Inference v6.1 results on NVIDIA GB200 NVL72

All results are MLPerf Inference v6.1, closed division, on NVIDIA GB200 NVL72 on Crusoe Cloud (gb200-186gb-nvl-ib.4x instances, 4x GB200 per node, 1200W TGP per GPU).

gpt-oss-120b works out to roughly 12,700 tokens per second per GPU in both scenarios. MLPerf measures latency-bounded throughput in the Server scenario, so the interesting number is the ratio: our Server result lands within 0.6% of Offline. The latency SLA costs almost nothing on this platform. The same node that maxes batch throughput also holds p99 time-to-first-token and time-per-output-token under the v6.1 limits, which is the property that matters when sizing a production deployment.

All runs cleared closed-division accuracy: gpt-oss-120b at 83.69% exact-match (Offline) and 83.64% (Server) against an 82.30% floor, and Qwen3-VL between 78.56% and 78.82% hierarchical F1 against a 78.24% floor. gpt-oss-120b passed compliance audits TEST07 and TEST09.

Published results are available in the MLCommons MLPerf Inference v6.1 closed-division results.

Software configuration

Prefill/decode disaggregation: a 2.5x gain from topology alone

The 16-GPU Qwen3-VL deployment is the most architecturally interesting system we submitted. Rather than running 4 independent node-sized replicas, we deployed NVIDIA Dynamo in a prefill/decode-disaggregated topology: prefill workers (TP2) handle prompt processing, decode workers (TP1) handle token generation, and KV cache moves between them over NVIDIA Inference Xfer Library (NIXL).

Disaggregation matters because prefill and decode have opposite hardware appetites. Prefill is compute-bound and parallelizes across long prompts; decode is memory-bandwidth-bound and wants maximum batch residency. Serving both phases on the same GPUs forces a compromise; splitting them lets each pool run at its natural operating point. This is the same architecture behind large-scale production inference systems, and MLPerf's Interactive scenario, with its tight p99 latency limits, is precisely where it pays off.

Our first Interactive run used a 75:25 prefill:decode GPU split and measured 15.85 queries/s. Profiling showed decode was the bottleneck: prefill GPUs finished prompts faster than decode GPUs could stream tokens under the p99 limit. Rebalancing to 50:50 and binary-searching the sustainable query rate brought the submission to 39.85 queries/s, a roughly 2.5x improvement with zero hardware changes. The final topology was 6 prefill workers across 3 nodes and 4 decode workers on the fourth. To date, that result is the highest Qwen3-VL Interactive score in the round on the GB200 NVL72 system.

Two takeaways for anyone deploying disaggregated serving:

• The prefill:decode ratio is workload-dependent and worth sweeping. The optimal split for a latency-bound Interactive workload was very different from the throughput-oriented defaults.
• Scaling was linear. Offline throughput at 4 nodes (230.09 samples/s) matched the linear extrapolation from the 3-node run (172.35 predicted 229.8). Server hit 198.94 against a target of 200 on the first attempt after setup fixes, 99.5% of the configured rate.

Running MLPerf the way production runs

We deliberately ran these benchmarks on the orchestration services that power our customers’ workloads today.

The single-node gpt-oss-120b runs executed on Crusoe Managed Kubernetes, the same platform as our other MLPerf work, with the same observability through Crusoe Managed Metrics. The multi-node Qwen3-VL runs executed on a Crusoe Managed Slurm cluster using NVIDIA's sflow launcher, with the model repository on VAST-backed shared NFS and staged to per-node NVMe so the storage hot path never crossed the network during timed runs.

What comes next

This submission demonstrates the NVIDIA Accelerated Computing Platform on Crusoe at two scales: a single GB200 node exceeding published reference throughput, and a multi-node disaggregated topology scaling linearly across NVLink and NVIDIA Quantum InfiniBand. Every Crusoe cluster is validated end to end during bring-up before a customer touches it, and MLPerf is one instrument in that process.

We are continuing to work with NVIDIA on performance and software co-design across GB200 NVL72, GB300 NVL72, and beyond, and we will keep publishing results for both Inference and Training workloads.

Ready to serve at this scale? Talk to our team about running inference on NVIDIA GB200 NVL72 on Crusoe Cloud.