Before first boot: How Crusoe Pre-Deployment Automation readies every GPU server

We've written about how Crusoe Provisioner takes a bare-metal GPU node from power-on to a fully configured hypervisor, and how Crusoe Burn-in proves every node meets our production bar before a customer touches it. This post covers the stage that comes before both: Pre-Deployment, the first mile between a server being physically racked and being ready to provision at all. It's the least visible part of bringing a data center online, and in much of the industry, this is handled manually. Here's how we automated it.

Why the first mile is hard

Before a node can be provisioned, a surprising number of things have to be true. Its identity has to exist in our DCIM. The vendor-supplied credentials for its BMC have to be uploaded. The device has to be physically racked, cabled, powered, and visible on the network. Network interfaces need to have their IPs assigned. And a full suite of validation checks has to pass: is the BMC reachable, does the hardware match what we expect, are the cables connected where they should be.

None of these steps are hard on their own. The challenge is that they form a chain of order-dependent work, and the way the industry typically handles that chain is manually: server metadata gathered by hand and uploaded before it can be racked, racking recorded once the paperwork is in order, ad hoc validation runs manually kicked off. When a prerequisite is missed, the failure often surfaces at the moment of provisioning, when the node is already on the critical path to customers. A data or hardware issue that could have been fixed weeks earlier instead has to be diagnosed and resolved in an extremely short window.

At the scale of a modern GPU data center buildout, where hundreds of nodes can land on the dock in a week, that approach becomes a bottleneck that slows how fast capacity reaches customers, and a source of drift, because the physical state of a data center and the recorded state of its systems inevitably diverge when humans are the sync mechanism.

We built Pre-Deployment Automation so that the chain never exists: an event-driven pipeline where every server gets its own workflow the moment it's racked, and that workflow shepherds it through every prerequisite and validation needed for provisioning. By the time a server reaches Provisioning, its readiness has already been validated, not assumed.

What Pre-Deployment does

At its core, Pre-Deployment is a per-server orchestration workflow. When a server is racked, a workflow is created for it. That workflow knows what has to be true before the server can be provisioned, waits for each prerequisite to be met (efficiently, on events rather than polling loops), collects the data the rest of our stack needs, and runs pre-provisioning checks. Workflow state and the status of every validation check are tracked end-to-end in our infrastructure inventory service and surfaced in our admin tooling, so anyone can answer "why isn't this node ready yet?" in seconds rather than hours.

The pipeline has four stages.

Step 1: Racking

Racking means two things in a data center. Physical racking is the hardware work: a server bolted into position, cabled, and powered. Logical racking is the record of that work: the server existing, in the right place, in Crusoe's data center digital twin. Every gap between the two is a window where our systems don't fully know the building, so the design goal is to bring them as close together as possible.

That's why racking is the entry point for Pre-Deployment: site ops logically racks a server in our admin tooling immediately after physical installation, and that action triggers the workflow. The digital twin reflects on-prem reality in real time, and the missing prerequisites become the workflow's problem to wait on, not the technician's problem to sequence.

Step 2: Waiting stages

Two prerequisites have to be met before pre-provisioning can run: the vendor data that makes the BMC accessible, and the device coming online. These are parallel tasks, and it isn't important which completes first; the workflow waits on both and advances the moment the second one lands.

Vendor data

The first thing the workflow needs is the vendor-supplied data that makes the server's BMC accessible. On start, it checks whether that data already exists in the DCIM. If it does, the stage completes immediately. If it doesn't, the workflow simply waits. The moment the data is uploaded, the upload flow signals the waiting workflow and the pipeline advances. No polling, no scheduled re-checks, no person remembering to kick anything.

Device online

Next, the workflow needs the server to actually be reachable. Device detection is driven by our DHCP infrastructure: a job checks for new leases and publishes an event for each newly discovered device. The workflow checks whether a lease already exists for the server; if so, it advances immediately. If not, it subscribes to events for that specific server and sleeps until the device appears on the network.

Step 3: Pre-provisioning

Once the device is online, the workflow runs pre-provisioning, which does three things.

IP reservation. The workflow reserves the device's BMC IP if a reservation doesn't already exist, so the MAC address observed at first lease is the address the node keeps.

BMC data collection. With the BMC reachable, the workflow collects subcomponent inventory over Redfish (BMC, GPUs, SSDs, and PSUs) and upserts it into our infrastructure inventory.

Validation checks. Finally, the workflow runs the checks that gate Provisioning: DCIM asset checks, BMC access checks, Redfish hardware checks, and cable connection checks.

Step 4: Provision-ready

Once all required validation checks succeed, the Pre-Deployment workflow completes and the node's provision-ready state is recorded in our infrastructure inventory.

From there, our lifecycle agent takes over, moving the node into the Provisioning state and handing it to Crusoe Provisioner. The handoff is machine-verified: by the time the lifecycle agent moves a node forward, every prerequisite has been checked, recorded, and passed.

Validation that explains itself

Every check is either required, gating the pipeline, or optional, informing without blocking. And every check resolves to one of four statuses.

Pass means the check ran and the server met the bar.

Fail means the check ran and found a real problem with the server: hardware that doesn't match the DCIM record, a cable in the wrong port.

Not Run means the check doesn't apply to this server type. RoCE MAC ingestion on a non-RoCE device is recorded as Not Run rather than silently skipped, so the coverage picture stays honest across a heterogeneous fleet.

Blocked means the check can't meaningfully run yet because something upstream is missing. If the BMC password hasn't been uploaded, the "BMC reachable" check doesn't fail; it reports Blocked.

The distinction between the last two and Fail matters operationally: Fail means something is wrong with the server, while Not Run and Blocked mean nothing is wrong at all, just inapplicable or not yet ready. Each status is a labeled, attributable state that routes to the right fix.

If any required check reports Fail or Blocked, the validation workflow fails and a retry is scheduled on a configurable interval, so the fleet is continuously re-scanned until every node clears. This is the pipeline's quiet superpower: failure isn't terminal, it's a waiting state. A server with a missing cable fails its cable check today; a technician reseats it tomorrow; the next scheduled retry passes and the server advances, with no one needing to remember to re-run anything. Once all required checks succeed, the node is cleared for Provisioning.

What this means in practice

Bringing a data center online is a race between hardware arriving and capacity reaching customers, and Pre-Deployment is where that race is won. From the moment a server hits the rack, it's on an automated pipeline moving it toward customers as fast as the physical world allows.

Reliable hardware and fast capacity bring-up are two key factors in selecting an AI infrastructure partner, and at Crusoe both are engineered in, layer by layer. Pre-Deployment gets a server from racked to provision-ready. Provisioner takes it from provision-ready to a fully configured hypervisor. Burn-in proves it meets our production bar before a customer ever touches it. Three systems, one property: a node's path to the fleet is automated end-to-end, observable at every step, and designed to be safe to retry at every stage. That's what it takes to bring GPU capacity online on a schedule customers can plan around, and it's what it means to be infrastructure customers can grow on.

To explore Crusoe Cloud's GPU offerings, visit crusoe.ai/cloud, or talk to our team about the infrastructure your workloads need at crusoe.ai/contact-sales.