Serving GLM-5.3 coding agents faster on NVIDIA GB200 NVL72
We benchmarked GLM-5.3 serving coding agents on a 16-VM NVIDIA GB200 NVL72 fleet on Crusoe Cloud, letting a dstack preset agent pick the layout. Aggregated serving won at 48 agents, and prefill/decode disaggregation kept scaling to 192.

This post was written in partnership with Andrey Cheptsov, founder and CEO of dstack.
An NVIDIA GB200 NVL72 fleet on Crusoe Cloud served GLM-5.3 coding agents faster than the median of 47 OpenRouter endpoints on all four speed measures we compared. At 48 concurrent agents, the median first token arrived in 0.29 seconds, against 0.99 seconds for the median endpoint.
We did not hand-tune the fleet. We gave a dstack preset agent the workload and let it design the deployment.
- Throughput: the agent raised output from 2,188 to 5,114 tokens per second in its best trial, up 134%.
- First token: median time to first token (TTFT) fell 23%, from 383 to 293 ms.
- Where the gain came from: almost all of it came from layout decisions, not engine tuning. That means whether to disaggregate prefill and decode, how many engines to run, how wide each one is, and whether to keep part of the KV cache in host memory.
- What changes at scale: the best layout depends on load. Aggregated serving wins at 48 agents, and prefill/decode (PD) disaggregation keeps scaling to 192 agents.
For additional test details and full benchmark results, see the dstack team's technical post.
This post starts with how to set up Crusoe VMs on an NVIDIA GB200 NVL72 rack, so you can reproduce the run. Then it covers the workload, the method, the results for both layouts, and the speed comparison.
Set up Crusoe VMs on an NVIDIA GB200 NVL72 rack
To reproduce the run, you need 16 NVIDIA GB200 NVL72 VMs in one NVLink™ domain, NVIDIA IMEX running on each VM, and a dstack fleet on top. This section walks through each step.
1. Check prerequisites
- A Crusoe Cloud project with NVIDIA GB200 NVL72 access. Some VM types are restricted by region, so contact Crusoe sales if the types below do not appear.
- The Crusoe CLI, installed and authenticated, and an SSH public key registered on your account.
- Quota and capacity for 16 VMs in one rack. A full NVL72 rack holds 18 VMs with 4 GPUs each, so 16 VMs gives you 64 GPUs within one NVLink domain.
2. Pick the instance type and location
The specs and location come from the Crusoe VM overview. One rack is enough for this run, because NVLink already connects every GPU in it, so no InfiniBand partition is required.
3. Create the VMs
Create all 16 VMs in the same NVLink domain, so every GPU reaches every other over NVLink. VMs in one domain share a pod ID, which you can check with the Crusoe CLI. Confirm the current placement options with crusoe compute vms create --help, and check the image name on the Crusoe Images page.
The NVIDIA GB200 NVL72 image is ARM64, which matches the Grace CPUs. Wait until every VM shows Running before you move on.
4. Enable IMEX on every VM
IMEX lets GPUs in the NVLink domain export and import memory across VMs. The nvidia-imex binaries are already on the NVIDIA GB200 NVL72 image. Run these steps on each VM, following the Crusoe IMEX guide.
First, check that the service is enabled.
Next, list the private IPs of all 16 VMs in /etc/nvidia-imex/nodes_config.cfg, one per line, then restart the service.
Then create an IMEX channel that survives reboots. Add the option below to /etc/modprobe.d/nvidia.conf, rebuild the initramfs, and reboot.
5. Validate the NVLink fabric
Run nvidia-imex-ctl -N on any VM. Every node should show C for connected toward every other node. An I means invalid, and sudo journalctl -u nvidia-imex shows the reason.
Then run an NCCL all-reduce across all 64 GPUs. On our rack, an all-reduce over 64 ranks spanning 16 nodes averaged 889 GB/s, with a 940 GB/s peak. If your numbers are far lower, fix the fabric before you benchmark. Crusoe also publishes an nvbandwidth how-to for NVIDIA GB200 NVL72 for multi-node checks.
6. Alternative: one-click setup with Terraform and Ansible
To streamline provisioning and eliminate the operational burden of manual node configuration, you can use the automated Terraform and Ansible blueprints from Crusoe’s Developer Hub. This setup handles infrastructure creation, IMEX deployment, local NVMe RAID0 array configuration, and multi-node compute validation in a single workflow. Infrastructure as Code (IaC) ensures your environment remains fully reproducible, robust, and ready for production workloads on Crusoe.
This one-click deployment is what we used to fully set up the NVIDIA GB200 NVL72 rack for this blog. Notably, dstack also supports direct provisioning on Crusoe through its crusoe backend.
7. Register the VMs as a dstack fleet
Install dstack, start the server, and register the VMs as an SSH fleet. List all 16 private IPs under hosts.
Presets use a locally installed Codex or Claude CLI as the optimization agent. Install the dstack skills so the agent knows the CLI and the preset format.
The fleet is ready. The rest of this post covers what we asked the agent to do on it.
The workload and four layout decisions
We replayed real coding agents against the fleet and asked two key questions. How fast can an NVIDIA GB200 NVL72 fleet serve GLM-5.3 coding agents? How many agents does the fleet hold?
The agents come from a recorded Claude Code trace of 393 real coding sessions, published by SemiAnalysis. One agent is one session, and a session can fan out into subagents.
A coding agent resends its whole context on every request and appends to the end. That is why 96% of each prompt is already seen, and why KV cache reuse decides how well a layout performs.
On a multi-node machine, a deployment can take many shapes. Four decisions matter, and none has an obvious answer.
- Disaggregate or not. Serving has two phases, reading the prompt and generating the reply. They can share GPUs, or run in separate prefill and decode pools that hand the cache between them. If you split them, you also pick the ratio of chips for each side.
- How to divide the fleet into engines. An engine is one copy of the model across a group of GPUs. A wide engine holds more cache, but every step waits on more GPUs. Many narrow engines give the router more places to send a session, but each holds a smaller share of the cache.
- How to split the model inside an engine. Tensor parallelism cuts each layer across GPUs. Data parallelism gives each GPU its own requests. Expert parallelism spreads the model's experts out. They are usually combined.
- Where the KV cache lives. It can stay in GPU memory, or spill into host memory when GPU memory runs out.
We chose NVIDIA GB200 NVL72 because it is the one machine where all of these options are available. Every GPU reaches every other over NVLink, across node boundaries. That matters because disaggregation usually fails when the cache handoff from prefill to decode lands on a network that is far slower than memory.
Cache capacity matters just as much. Each node carries 881 GB of host memory joined to its GPUs by NVLink-C2C at 900 GB/s, against 744 GB of HBM across its four GPUs. Once the weights are loaded, host memory adds more cache than the GPUs do, at a speed that makes it a real second tier.
How we ran the optimization agent
A dstack preset hands the workload to an agent and lets it run trials on your fleet. The agent is a locally installed Codex or Claude CLI, and in this study it ran on Claude Opus 5. We supply a preset configuration that sets the model, the workload, the concurrency, the number of trials and the constraints. The preset gives the agent a harness to run trials, benchmark each candidate, and verify the winner as a deployable service. Later sessions can inherit the trial records of earlier ones.
Both layouts used SGLang as the inference backend. The aggregated layout used SGLang Model Gateway as the router, and the PD layout used Dynamo. The router ran on a separate x86 CPU VM with 32 vCPUs and 128 GB of RAM, in the same private network as the GPU VMs.
Our constraints were 48 concurrent agents, a median TTFT under 1,000 ms, a context length of at least 1,000,000 tokens, and five trials per session.
Two modes
The study ran in two modes. In layout unpinned mode, the agent decides the layout itself, from aggregated or disaggregated serving to engine count, width, parallelism, prefill to decode ratio and cache offload. In PD pinned mode, every trial must use prefill/decode disaggregation. The agent still tunes everything inside it, but it cannot fall back to aggregated serving.
The cookbook targets a single GB300 node at TP4. NVIDIA GB200 NVL72 has 186 GB per GPU, which cannot hold 755.7 GB of weights across four GPUs. The minimum adaptation is TP8 across two nodes, replicated seven times to use the fleet.
The two baselines are different kinds of opponent. The first is what you get by following the documentation. The second is what a specialist team published after tuning the hardware themselves.
What stayed fixed
Four things stayed in the prompt in both modes, so nothing outside the layout varied between sessions.
- Benchmark settings. The AIPerf scenario, duration, trajectory window, idle-gap cap and random seed were identical on every trial. Any difference between trials is the agent's change, not different traffic.
- NCCL test results. We measured all-reduce over 64 ranks across 16 nodes at 889 GB/s average and 940 GB/s peak, and gave the agent these numbers. Engines that span node boundaries were a known option, not something to rediscover.
- Grace CPU memory. Each GPU reaches system memory over NVLink-C2C at up to 900 GB/s per superchip, so host memory is a usable second tier for the KV cache.
- Setup issues. We listed the failures our preparation session had hit, with fixes, so no trial was spent rediscovering them.
We also told the agent to treat the whole fleet as the unit being optimized. In preparation runs it used only the minimum nodes needed to load the model and left the rest idle.
Concurrency sweep and benchmark window
To test whether each configuration scales, we took the best service from both modes and swept each across rising agent counts, changing nothing but the agent count. We ran the PD sweep at our accelerated pace, with idle gaps capped at 5 seconds, and the aggregated sweep at SemiAnalysis pacing, so compare the two sweeps with care.
SemiAnalysis runs 3,625 seconds per measurement. We ran 929 seconds and capped the trace's idle gaps at 5 seconds instead of 300. Their window would have made a full session, five trials plus a deployment, take a day instead of six hours. Requests in flight per chip came out close to theirs, at 2.07 against 2.27, so the fleet is under comparable load in a quarter of the time.
Results: aggregated serving wins at 48 agents
Left free to choose, the agent settled on seven two-node TP8 engines with a host-memory cache tier (HiCache). Over three sessions it raised throughput from 2,188 to 5,114 tokens per second, up 134%, and cut median TTFT from 383 to 293 ms.
Each session starts from its own baseline, so the first two gain columns use different starting points. Measured from the session 1 baseline, session 3 is up 133.7% in throughput. The 5,114 tok/s figure is the agent's best optimization trial. The same layout measured as a deployed service gives 5,050 tok/s at 291 ms median TTFT, which is the figure the 48-agent comparison below uses.
Each session taught the agent something about the tradeoffs.
- Data-parallel attention frees cache but costs speed. Assigning sessions to different GPU groups cut duplicate KV copies. It also slowed long prompts and added communication between GPUs.
- Decode context parallelism (DCP) added about 5.3 times the cache per group. Combining partial attention results at every decode step ate the gain. HiCache gave more room while keeping fast TP8 decode steps.
- Smaller engines lost on long prompts. Fifteen single-node TP4 engines decoded faster, but each had less compute and GPU cache. Throughput fell 28%, from 4,623 to 3,338 tokens per second.
- Write-back is the better HiCache policy here. New context stays in GPU memory and is copied to host memory only when it is evicted, which avoids needless copies.
PD at 48 agents
In the first three sessions the agent dropped PD after setup problems blocked a benchmark. In a fourth session we required it to resolve those problems and test PD. The best compliant PD configuration used three TP4/DP2 prefill engines and six TP8/DP4 decode engines across 15 nodes. It reached 3,212 tokens per second at 561 ms median TTFT, against 5,050 tokens per second and 291 ms for aggregated on 14 nodes.
PD lost because its decode engines took longer per token. Data-parallel attention split sessions into smaller groups and added coordination between GPUs.
To check the result against published work, we compared the later 16-node PD service and the aggregated service with SemiAnalysis's 45-agent GB200 result.
Our two columns are deployed service runs, not in-trial figures. That is why PD reads 55.5 tok/s per chip here, against 53.5 to 57.7 for the best 48-agent PD trials. The PD run here is separate from the sweep in “The PD service scales,” so its figures differ slightly.
SemiAnalysis's configuration is handed requests at nearly the rate ours is per chip, 0.053 against 0.058 per second, and delivers nearly the same output per chip, 56.1 against 55.5. Ours matches its throughput per chip with a p50 TTFT 62% lower and a p99 TTFT less than half as long. SemiAnalysis's trace averaged 163,786 input tokens per request here, against about 133,000 to 135,000 for ours, so read the comparison as approximate.
Where aggregated runs out
The winning layout is limited by cache capacity. Inside each engine, all 8 GPUs hold a copy of the same KV cache, and the seven engines cannot borrow cache from one another. We swept the deployed service across 48, 120 and 240 agents under SemiAnalysis pacing, so absolute numbers differ from the trial results above.
Throughput peaks at 120 agents and falls 38% by 240. Cache reuse falls with it, while the share of each prompt available to reuse stays near 96%. The server stopped recognizing context it had already processed.
The median hides the damage. Across a fivefold increase in agents, p50 TTFT rose from 278 ms to 625 ms, while p99 TTFT rose from 5.7 seconds to 690 seconds. A dashboard that tracks only median latency would call this service healthy at 240 agents, when one request in a hundred waits eleven minutes.
Routing is not the cause. Cache-aware and DP-aware routing were both on, and the router sent each session back to the engine that last served it. By then, that engine had already evicted the context. This is a capacity problem.
A run without the host-memory tier confirms it. Reuse dropped to 27.6% at 96 agents and 6.3% at 192, and median TTFT at 96 agents was nearly a minute. Same router, same policy, same workload, less room for cache.
Two layout changes can attack a capacity problem like this. Fewer, wider engines split the cache into fewer, larger pieces, but every decode step then waits on more GPUs. PD disaggregation holds context in a prefill pool, so decode engines keep only what they are generating. The agent never tested PD on its own, because it never got PD to run. That is what the PD-pinned sessions were for.
Results: PD disaggregation keeps scaling to 192 agents
When we pinned PD, the agent landed on four prefill and four decode engines across all 16 nodes. At 48 agents the best PD result was 3,693 tokens per second at 887 ms median TTFT, behind aggregated's 5,114 and 293 ms. Its advantage appeared with load, reaching 10,808 tokens per second at 192 agents.
What the agent learned
- Session 1: give prefill more chips. The first full-fleet layout used a 1:3 prefill to decode split and missed the 1,000 ms limit at 1,447 ms. The rebalancing step reduced median TTFT 47%, from 1,447 to 762 ms.
- Session 2: sharing attention weights has a hidden cost. Pairing decode GPUs cut attention weights from 28.6 GB to 14.3 GB per GPU. But each pair held its own KV copy, so prefill had to transfer two copies per session. Four-way sharing needed four copies and pushed median TTFT over one second.
- Session 3: narrower decode engines won by the largest margin. We started from SemiAnalysis's 128-agent engine shapes, doubled to fit 64 chips. Splitting two wide DEP16 decode engines into four DEP8 engines lifted throughput 30%, from 6,510 to 8,470 tokens per second. The decode step fell from 35.7 ms to 24.5 ms.
- Session 4: more decode KV memory helped throughput but not TTFT. Decode KV capacity grew from 1.34 million to 1.69 million tokens per GPU, and throughput rose 12.5%. Median TTFT stayed above target.
The session 3 result runs against the obvious expectation. A narrower engine gives each GPU a larger share of the experts, so each GPU reads more weight bytes per step, yet the step got faster. The step was never limited by how fast GPUs read weights. It was limited by how long they waited on each other across node boundaries.
Two more decode changes lifted session 3 from 8,470 to 8,599 tokens per second. First, we cut every expert into eight pieces, so each GPU holds a piece of all 256 experts. Second, speculative decoding reached three steps ahead instead of two. Once each GPU reads a piece of every expert anyway, the extra drafted token costs nothing, and the deeper draft pays off for the first time.
How it compares with published PD results
At 128 agents, our PD setup matched SemiAnalysis's per-chip output, at 134.4 against 136.4 tokens per second per chip. Latency is where the two separate. Our p50 TTFT was 42% lower, and p99 TTFT was five times tighter, at 14.8 seconds against 73.1 seconds. The loads are comparable rather than identical, so read these figures that way.
Why PD looks weak at 48 agents
At 48 agents, PD used two more nodes than the aggregated layout and produced 38% less output per chip, 55.5 against 90.2 tokens per second. PD is not broken at that load. It has nothing to do. The prefill pool sits idle, and every chip in it is one the aggregated layout would have used to generate tokens.
That is why the case for PD has to be made where aggregated runs out. It is also why session 3 moved to 128 agents.
The PD service scales
We deployed session 4's configuration and swept it across four loads, changing nothing but the agent count. This sweep ran at our accelerated pace, with idle gaps capped at 5 seconds.
Throughput rises at every step. Four times the load takes the fleet from 3,530 to 10,545 output tokens per second. Unlike the aggregated layout, it never peaks and falls back, because the cache keeps working. The service reused 96.4% of each prompt at 48 agents and 95.5% at 192, against 76.6% for aggregated at 240, though the two sweeps ran at different paces.
What rises is latency. Median TTFT goes from 727 ms to 3,250 ms across the sweep. The service scales, and the price is how long an agent waits for its first token.
Which layout fits your load
For GLM-5.3 on NVIDIA GB200 NVL72, the tested aggregated layout gives faster replies while sessions' caches fit in memory. As more sessions need cache, the tested PD layout keeps more context available for reuse and produces more tokens overall. The tradeoff is that at low load a single session waits longer for its first token and gets later tokens more slowly.
The two sweeps in this post used different agent counts and different pacing, and they show different requests in flight at 48 agents. You cannot read the exact crossover off the two tables. What the data does show is the mechanism, which is cache capacity.
So the practical limit is not a fixed number of agents. It depends on context size and the cache memory available. Measure cache reuse and p99 TTFT as you add agents, not only the median.
The speed against 47 OpenRouter endpoints
At 48 agents, the aggregated layout returns the first token in 0.29 seconds at the median, against 0.99 seconds for the median of 47 GLM-5.3 endpoints on OpenRouter. Its median speed is 137.0 tokens per second, against 81. At 128 agents, the PD layout runs at 60.1 tokens per second with a 1.98 second median first token, which is the price of serving more agents on the same fleet.
Provider speeds are averages over Oct. 19 to 26. We use OpenRouter's own definition of throughput, which counts from the moment a request is sent and includes TTFT, for responses over 50 output tokens. We apply the same definition to our per-request records.
Both PD rows are the deployed service from the concurrency sweep, measured at 48 and 128 agents.
Two results stand out. At 48 agents, the aggregated layout is faster than the median provider on all four measures. At 128 agents, the PD layout is slower than the median provider at p50, on both speed and first token, but it is ahead at the tail, with a p99 speed of 14.7 against 10 tokens per second and a p99 first token of 12.8 seconds against 20.1.
One caution on speed. Provider figures are public measurements averaged over a week, and ours come from a replayed trace on a dedicated fleet, so this is not a controlled test using the same requests. Your own speed depends on load, context length and cache reuse, so re-run the benchmark at your agent count.
Limitations
Read these results with the following limits in mind.
- One model, one trace. The test covers GLM-5.3 and one recorded coding trace. Other models and workloads may land on different layouts.
- A tested layout, not a general rule. The cache limit we saw belongs to the 7 x TP8 configuration we tested. It is not a limit of aggregated serving in general.
- Optimized for 48 agents. The agent tuned for 48 agents. Its behavior at higher loads is a separate question.
- Shorter window. We ran 929 seconds with idle gaps capped at 5 seconds, against SemiAnalysis's 3,625 seconds. Per-chip load is comparable, not identical.
- One PD run missed its target. At 192 agents, PD reached 10,808 tokens per second but missed the 3,000 ms median TTFT target at 3,170 ms.
- Replayed workload, not live traffic. Provider speeds come from public endpoint measurements, so the speed comparison is not a controlled test using the same requests. Real traffic may differ in prompt lengths, cache reuse, timing and concurrency.
- Layout over engine tuning. The agent spent most of its time exploring multi-node layouts, which left less time to tune engine settings or change source code. More engine tuning may improve these results.
- Approximate per-chip comparisons. Our comparisons with SemiAnalysis differ in fleet size, benchmark window and request pacing, so they are not controlled head-to-head tests.
Conclusion
For 48 coding agents on GLM-5.3, an NVIDIA GB200 NVL72 fleet running the agent-optimized aggregated layout returned the first token in 0.29 seconds at the median, and it was faster than the median of 47 OpenRouter endpoints on all four speed measures.
- How fast. At 48 agents, the aggregated layout reached 137.0 tokens per second at the median and a 0.29 second median first token, against 81 tokens per second and 0.99 seconds for the median endpoint.
- How many agents. The aggregated layout keeps working as long as sessions' caches fit and cache reuse stays high. In our sweep it kept gaining throughput up to 120 agents and fell by 240, as reuse dropped. The limit depends on context size and cache memory, not on a fixed agent count. The agent was asked to optimize for 48.
- Which layout. Aggregated is faster at low load. PD keeps scaling past it, at the price of a longer wait for the first token.
The reason is layout. Seven TP8 engines keep fewer sessions on each engine, which shortens generation steps. Cache-aware routing brings sessions back to their saved context. And 140 GB of host cache per GPU rank adds room, until duplicated KV inside each engine becomes the limit.
The agent chose aggregated serving on its own. Later PD-pinned sessions confirmed that aggregated was the better layout at 48 agents, and showed that PD takes over once cache capacity binds. The agent was asked to optimize for 48 agents, so judge its result at that load apart from the limits we measured above it.
Next steps
- Set up your rack. Follow the VM steps earlier in this post, or use the one-click Terraform and Ansible blueprint in Crusoe’s Developer Hub.
- Deploy a preset. Both presets are public. Presets are an experimental dstack feature, so check the dstack presets documentation for current flags.
Aggregated, tuned for 48 agents:
Disaggregated, tuned for 128 agents:
- Re-run at your own agent count. Watch p99 TTFT and cache reuse, not only the median.
- Pick the layout by load. If cache reuse falls as you add agents, test the PD preset. If your sessions’ caches fit, aggregated gives faster replies.
- Run the agent on your own workload. The study covers one model and one trace. Create a preset with your model, your traffic and your constraints, and let the agent search the layout space on your fleet. The dstack team’s technical post has the full benchmark detail.
To run this on your own rack, get started on Crusoe Cloud or contact Crusoe sales to discuss NVIDIA GB200 NVL72 capacity.
Frequently
asked questions
Why split prefill and decode?
Splitting prefill and decode lets decode engines hold only the context they are generating, which keeps KV cache reuse high as load grows. On a 64-GPU NVIDIA GB200 NVL72 fleet serving GLM-5.3, the disaggregated layout scaled from 3,530 to 10,545 output tokens per second between 48 and 192 agents, while prompt reuse stayed above 95%. At low load it loses, because the prefill pool sits idle.
How many tokens does agentic coding use?
In SemiAnalysis' trace of 393 real Claude Code sessions, a coding agent sent about 135,000 input tokens per request and got back about 1,000, because it resends its whole context on every request and appends to the end. About 96% of each prompt had already been seen in the previous request. That ratio is why KV cache reuse decides how well a serving layout performs.
How important is KV cache?
KV cache capacity decides whether a serving layout keeps working as load grows, because coding agents resend nearly all of their context on every request. In the GLM-5.3 benchmark, a run without a host-memory cache tier saw prompt reuse drop to 27.6% at 96 agents and 6.3% at 192, and median time to first token at 96 agents was nearly a minute. Adding 140 GB of host-memory cache per GPU rank was the single largest throughput gain the optimization agent found.



