NVIDIA and Crusoe cut GLM-5.3 startup from 8 minutes to 31 seconds with NVIDIA Snapshot
Once a checkpoint artifact sits on local NVMe instead of NFS, restoring 594 GB of GPU state for GLM-5.3 on NVIDIA HGX B300 drops from 33.3 s to 15.0 s, and the pod is ready in 31 s instead of 50 s. Storage, not CRIU, is the lever.
Starting GPU inference workers is a complex task that requires things like loading weights, initializing NVIDIA CUDA, and warming up the engine. This startup time is required every time a pod scales up or reschedules.
NVIDIA Snapshot, an open source, modular component of the NVIDIA DSX AI Factory Platform, substantially reduces startup overhead. It captures a fully initialized GPU worker, including CPU and GPU memory, and restores it on any compatible node so new pods can become ready in seconds instead of minutes. DSX brings together technologies to help the ecosystem design, build and operate more efficient AI factories, with independently adoptable components such as Snapshot supporting faster recovery and scaling. The project lives in the open at ai-dynamo/snapshot, and we have been running it on Crusoe Cloud.
In close collaboration with NVIDIA, our engineering team demonstrated a 15x speedup, reducing restore times from roughly 8 minutes for a cold start down to just 31 seconds when restoring GLM-5.3 on NVIDIA HGX™ B300. Along the way, we contributed our performance-enhancing code updates upstream to the open-source Snapshot repository for the benefit of the broader community. Together, NVIDIA Snapshot and Crusoe Cloud combine open-source software with accelerated infrastructure to bring inference workers online faster, advancing the NVIDIA DSX vision of more efficient AI factories.
What Snapshot does
Snapshot is a Kubernetes-native checkpoint and restore system for NVIDIA GPU workloads. A privileged node agent runs on every GPU node and performs the work by invoking Checkpoint/Restore In Userspace (CRIU) and the NVIDIA cuda-checkpoint utility against live processes. To restore, a new pod references a previously captured snapshot artifact. Snapshot owns the mechanics of checkpoint and restore. The systems that integrate with it decide when to use them.
This benchmark uses release v0.2.0-rc.2 of ai-dynamo/snapshot with our code in PR #602.
The benchmark
We restored a snapshot with 594 GB of GPU state and compared two storage backends for the checkpoint artifact: NFS and local NVMe (cold).
Test setup: On GLM-5.3 with NVIDIA HGX B300 on Crusoe Cloud. "Cold" means first read with no warm page cache.
What the numbers show
GPU restore is the big win. Moving 594 GB of GPU state from local NVMe took 15.0 seconds, about 40 GB/s. The same restore from NFS took 33.3 seconds, roughly 18 GB/s. That is a 55% reduction, and it accounts for most of the improvement in total restore time.
Agent restore time dropped 45%. The total went from 41.4 seconds to 22.9 seconds.
CRIU restore did not change. At about 5 seconds on both backends, CRIU's share of the work is independent of where the artifact lives. On NVMe it is now about a fifth of the agent's restore time.
Pods are ready 19 seconds sooner. Pod created to Ready fell from 50 seconds to 31 seconds, a 38% reduction. About 8 seconds of that 31 sits outside the agent's restore work, which makes it the next place to look.
The takeaway is simple. Once restore is no longer dominated by storage reads, the remaining cost is a few seconds of CRIU and some pod lifecycle overhead. Storage is the lever.
Availability on Crusoe Cloud
Customers can deploy NVIDIA Snapshot on Crusoe Cloud today with any NVIDIA GPU type.
Contributing back to ai-dynamo/snapshot
Snapshot is an open source project, and we want to make it better for everyone who runs it. While testing, we reported a restore failure that was hard to diagnose. NVIDIA opened it as issue #310 based on our Crusoe team's findings, and we fixed it in pull request #532, which is merged to main for everyone to use.
The symptom. A restore pod failed, and the UI reported binfmt_misc: Operation not permitted. That points at a permissions problem.
The real cause. The end of CRIU's restore.log showed Can't stat mountpoint .../cache/vllm: No such file or directory. The restore pod was missing the /cache/vllm mount that the source pod had, so CRIU could not rebuild the mount namespace. The binfmt_misc message was an earlier warning, not the cause.
A misleading pattern. The source used several NFS subPath mounts. Configurations with three or four mounts worked, and one with 10 failed. That looks like a mount limit, but the log shows a single missing target, not a count.
The fix. The issue lays out three changes:
- Compare the snapshot's external mount paths with the restore pod's mounts before starting CRIU.
- If a mount is missing, name the path in the Pod condition, event and CLI output, for example
Restore pod is missing mount/cache/vllm. - When reading
restore.log, report the last fatal error instead of an earlier warning.
Three related PRs from our team:
- ai-dynamo/snapshot#532 report the error that stopped a failed restore (Alon Kejzman)
- ai-dynamo/snapshot#497 make the agent runtimeClassName configurable in the Helm chart (Nir Levy)
- ai-dynamo/snapshot#499 remove an unused placeholder image target (Tavor Lavi)
The lesson applies to anyone building on checkpoint and restore. The restore pod's spec is part of the checkpoint contract, and a check before CRIU runs turns a cryptic low-level failure into a one-line fix.
Working with NVIDIA
Snapshot builds on NVIDIA cuda-checkpoint and runs as part of the NVIDIA DSX ecosystem, so progress depends on tight feedback between the teams running it and the teams building it. We focused on finding the issues and getting the bugs fixed, and NVIDIA team supported us with their engineering input and validation. Eventually accepting all of our PR and merged them into main for everyone to use. NVIDIA's Snapshot engineering team worked with Crusoe from the first test runs through performance tuning. When Crusoe encountered a restore failure, the NVIDIA team helped resolve the issue and merged a fix upstream. NVIDIA also reviewed and merged Crusoe's contributions to the open-source project. The teams jointly tuned the setup until GLM-5.3 on Crusoe's cluster matched the startup time achieved in NVIDIA's internal benchmark.
Try it yourself
Restore time is now mostly about storage. With the artifact on local NVMe, a GLM-5.3 worker with 594 GB of GPU state is ready in 31 seconds. The rest is a few seconds of CRIU and pod startup. For AI factory operators, faster restores mean available GPU capacity can begin serving workloads sooner, a practical example of the operational efficiency NVIDIA DSX aims to enable.
- Repository: ai-dynamo/snapshot
- Release: v0.2.0-rc.2 with our code in PR #602
- Issue: #310
Want to try it? Launch an NVIDIA HGX B300 instance on Crusoe Cloud.
Frequently
asked questions
What is checkpoint-based recovery?
Checkpoint-based recovery saves the complete state of a running process so it can be restored later instead of being rebuilt from scratch. For GPU inference, NVIDIA Snapshot captures a fully initialized worker, including CPU and GPU memory, and restores it on any compatible node. In Crusoe's benchmark on NVIDIA HGX B300, with the snapshot on local NVMe, a restored GLM-5.3 pod went from created to Ready in 31 seconds.
How does CRIU work?
CRIU (Checkpoint/Restore In Userspace) checkpoints a live Linux process and later restores it, and NVIDIA Snapshot's node agent invokes it together with NVIDIA cuda-checkpoint against live processes. In Crusoe's GLM-5.3 benchmark, CRIU restore took about five seconds on both NFS and local NVMe, roughly a fifth of the 22.9-second agent restore on NVMe. CRIU also rebuilds the process's mount namespace, so the restore pod must have every mount the source pod had.
Is local NVMe faster than NFS for restoring a GPU snapshot?
Local NVMe restored a GPU snapshot more than twice as fast as NFS in Crusoe's benchmark. Moving 594 GB of GPU state for GLM-5.3 took 15.0 seconds from local NVMe, about 40 GB/s, versus 33.3 seconds from NFS, roughly 18 GB/s, a 55% reduction. Total agent restore fell from 41.4 to 22.9 seconds, and pod created to Ready fell from 50 to 31 seconds, because storage reads dominate restore time once CRIU's share is fixed.
Why does a Snapshot restore fail with “binfmt_misc: Operation not permitted”?
A Snapshot restore that reports binfmt_misc: Operation not permitted can be failing for a different reason, because that message is an earlier CRIU warning rather than the fatal error. In the case Crusoe hit, the end of CRIU's restore.log showed a missing /cache/vllm mount: the restore pod lacked an NFS subPath mount the source pod had, so CRIU could not rebuild the mount namespace. Pull request #532 in ai-dynamo/snapshot now surfaces that last fatal error.
.png)