Tokenomics in the Age of Agentic Inference

A few months ago, NVIDIA introduced the idea of an inference iceberg to show why accurately evaluating AI infrastructure requires diving deeper than GPU/hour and FLOPS per dollar — the numerators visible above the surface. The metaphor is simple: many teams that buy AI inference focus on what’s above the waterline: peak GPU specifications, cost per GPU per hour, or FLOPS per dollar. But a good buying decision starts with looking beneath the surface, which represents key factors that determine real-world token output. Cost per token is the ultimate measure of inference efficiency because it determines whether that cost holds under real production conditions. Agentic AI demands more from inference. This piece explores the imperatives for every person making inference infrastructure decisions. 

Why agents are changing the equation

The infrastructure playbook for AI inference was built for a request-response world: prompt in, completion out, done. That pattern is fading with the rise of agentic workloads moving into production. Agentic AI systems don't send one request and wait for an answer. They execute multi-step plans, maintain stateful context across dozens of inference calls, invoke external tools and APIs mid-workflow, and adapt their approach when intermediate steps fail. 

Agents can use 10 to 100x more tokens per task than a standard chat exchange. This explosion of token throughput has consequences that ripple through every layer of the stack, including how we measure success.

Consider a customer support agent resolving a complex enterprise issue. It may invoke more than a dozen tools, perform multiple searches, retrieve documents, summarize findings, and finally generate a single response. Although the final answer contains only 800 tokens, the workflow may process more than 50,000 tokens across retrieval, reasoning, tool calls, and intermediate steps. It’s not hard to see how inefficiency can break any budget.

Agents also use tokens differently than single-turn chat. They maintain stateful context, generate bursty sequential requests, and require consistent low latency. These patterns stress the entire inference stack. Agentic workflows demand underlying infrastructure that can handle growing context windows, stateful multi-turn execution, sustained high utilization, and compounding latency because every step depends on the last. This shift from single-turn conversations to multi-step execution fundamentally changes the economics of AI inference. 

The cost for agentic workloads should take into account a variety of variables beyond the cost of compute. For example, tools can generate large outputs that turn into input tokens for the next turn, this drives up iterative prefill cost between turns and poses a long tail cached token burden on memory and decode memory bandwidth. Additionally, multi-agent depth is a major multiplier of token usage, if you want to accelerate the time-to-solution it will increase token usage across parallel agents. This is why you must evaluate the entire stack when evaluating infrastructure for agentic workloads. 

Beneath the surface: new metrics for measuring agentic workloads

For years, GPU/hour and FLOPS per dollar were the default metrics for evaluating inference infrastructure. They answered a simple question: how much compute can I rent, and at what price? Because these metrics are visible, they’re easy to compare. The problem is they measure input, not output.

Beneath the surface is the metric that brings the ground truth of AI economics to life: cost per token. It measures the actual cost to produce each delivered token instead of just the compute consumed producing it, and it’s why two cloud providers offering the same GPU at the same hourly rate can deliver wildly different results. Cost per million tokens is driven by the tokenomics denominator, not the numerator. It's what every provider quotes, what every evaluation compares, and it's the most important signal of measured AI factory performance. But cost per token alone doesn’t tell the full story.

A more complete tokenomics view looks at multiple metrics: tokens per watt, cost per token, cost per completed task, and goodput. What determines the total output value? Two layers: both invisible in any pricing comparison.

• The physical foundation - the foundation that determines sustained throughput, energy efficiency, and scale.
• The open software stack - the serving frameworks, open models, and optimization software that maximizes infrastructure utilization and reduces inference cost.

Together, these two layers form the foundation of modern AI tokenomics for agentic systems.

The physical foundation

Most conversations about inference economics start with software and end with silicon. But beneath both sits the widest layer of the iceberg: the physical infrastructure. How an AI factory is powered, cooled, and networked sets the ceiling that everything above it can reach. Under the sustained demand of agentic workloads, that ceiling becomes a binding constraint.

Efficient GPU utilization, dynamic power distribution to where it is needed, and hot water liquid cooling within an AI factory expands the tokenomics denominator and therefore, the tokens per megawatt: the metric that tells you how much intelligence a fixed power budget can produce. Modular designs determine how fast new GPU generations, like Blackwell to Vera Rubin, can be deployed without needing to rebuild a data center. 

Power: AI factories are usually constrained by power before they are constrained by demand. Every watt consumed by overhead is a watt that cannot be used to produce tokens. Every watt stranded by static allocation is capacity that exists physically but cannot be turned into useful output. Crusoe secures power, often below market rates, by sourcing energy directly and designing for long-term efficiency. Crusoe’s AI factories run on a diverse range of energy sources including solar, wind, battery storage, hydro, natural gas, and geothermal. And by building power and compute infrastructure in parallel, new capacity can be energized quickly.

Cooling: Supporting higher 45°C inlet temperatures through direct liquid cooling and system-level thermal design can improve facility efficiency by reducing the burden on chilled-water systems and other heat-rejection infrastructure. When less site power is spent on cooling overhead, more of the fixed power budget can be directed toward token-producing compute. For bursty chatbot traffic, thermal throttling during off-peak periods is manageable. For agentic workloads maintaining continuous high utilization, throttling translates directly into lost tokens per watt.

Networking: Evaluating physical infrastructure for agentic workloads also means asking whether the scale-up interconnect can handle the all-to-all traffic of large MoE models. In a MoE architecture, every token generation requires routing across distributed expert sub-networks. If the interconnect can't keep pace, experts sit idle regardless of how much raw compute sits behind them.

NVIDIA provides the full AI factory stack—from accelerated computing and reference architectures to NVIDIA Dynamo serving software and open Nemotron reasoning models. At the infrastructure layer, the NVIDIA DSX platform provides the reference architecture for purpose-built AI factories. Power, cooling, networking, and compute co-designed as a unified system. This is the operating frame behind DSX MaxLPS: maximize AI factory throughput and tokens per watt within a fixed power budget using intelligent optimizations and dynamic policies across GPU, rack, and workload levels. Crusoe is an early adopter of the DSX Platform, to increase performance and efficiency of Crusoe AI factories. 

Crusoe designs, manufactures, and operates vertically-integrated AI factories, which means decisions on everything from energy sourcing to responsible data center development to software optimizations powering Crusoe Cloud are made in-house; not inherited from a third-party. The ability to control memory and interconnect capacity directly, control cache placement instead of working through an abstraction layer, and onboard or allocate compute capacity where it’s needed are structural advantages that help deliver the sustained, high-throughput inference that agents demand. 

This co-design is how purpose-built infrastructure shows up in low cost per token.

The open-source stack

The open-source AI ecosystem is the fastest way to further optimize the largest layer of the inference iceberg. The physical layer provides the baseline performance, the software maximizes total performance.

For a growing set of production workloads, open models are redefining the economics of AI inference. They increasingly deliver frontier-level reasoning performance while giving organizations the flexibility to customize, optimize, and deploy models at a fraction of the inference cost of proprietary alternatives. The evidence is concrete: LangChain recently optimized its Deep Agents harness for NVIDIA Nemotron 3 Ultra, enabling it to match the top closed models on business tasks, with no model retraining required. Purpose-built for long-running agentic workflows, NVIDIA Nemotron 3 Ultra combines frontier reasoning performance with open weights that enable organizations to customize and deploy specialized AI agents for production. 

The gains came not from changing the model itself, but from optimizing the entire inference environment around it. At agent scale—where the same multi-step workflow may run thousands of times a day—that difference compounds rapidly. A lower cost per token isn’t simply a pricing advantage; it’s what separates sustainable unit economics from an inference bill that grows faster than business values.

NVIDIA Nemotron 3 Ultra and the tuned LangChain Deep Agents harness are available to run on Crusoe Managed Inference today, providing a direct path to deploy high-performing open agentic AI systems on infrastructure purpose-built for the workloads they generate. 

For teams that want to modify open models using their own data, open tooling makes it easier to turn real-world usage into continuous model improvement at scale. Crusoe Serverless Fine-Tuning gives teams a managed pipeline to customize open models on their proprietary data and deploy with self-serve inference on the same platform, or anywhere else—zero lock-in.

At the inference layer, NVIDIA Dynamo is becoming the open-source foundation for high- performance serving. Dynamo’s disaggregated serving architecture, which separates the compute-heavy prefill phase from the memory-bound decode phase, is especially suited to agentic prompts, which tend to be prefill-heavy by nature. Because Dynamo is built on and optimized for NVIDIA infrastructure, token output continues to improve and cost per token continues to decline long after the infrastructure is acquired.

Crusoe isn't just a consumer of NVIDIA open-source software, it's a contributor. Crusoe worked closely with the Dynamo team to develop fastokens, a drop-in BPE tokenizer that improves tokenization for every team building on Dynamo. For prompts above 50K tokens—common in agent workloads—fastokens delivers an average tokenization speedup of 17.4x, peaking at 31x, with end-to-end time to first token (TTFT) improvements of up to 40% in production. That improvement ships to the entire Dynamo community, not just Crusoe Cloud customers.

The opportunity for AI innovation in the open-source ecosystem is immense. Open models, open serving frameworks, and open tooling give teams the control, the economics, and the optionality that proprietary stacks structurally cannot.

Goodput and throughput matter for maximum value

The physical layer determines what your infrastructure can sustain. The open-source stack determines how efficiently you use it. But neither tells you whether the tokens your system produces are actually delivering value. That's the final layer of the iceberg, and it’s the easiest to measure wrong.

Most teams monitor throughput: total tokens generated per second. It's an intuitive metric but doesn’t paint the full picture, because it counts every token a system produces whether or not that token arrived in time to matter. The industry is converging on a better measure of success: goodput, the volume or percentage of token output that actually advances the task toward completion within acceptable latency.

For single-turn inference, throughput and goodput track closely. Agents expose the gap. Take a 15-step research agent: if step 8 times out — a cold KV cache forcing full recomputation, a long prefill causing head-of-line blocking, sustained load pushing hardware into thermal throttling — the entire task fails. Throughput says the system generated thousands of tokens across the first eight steps. Goodput says it delivered nothing, because every one of those tokens was wasted on compute for a task that never completed.

This is where cost per task is another valuable measure for agent-scale economics: tokens per task multiplied by cost per token. If goodput measures the overall efficiency of the underlying infrastructure, it can determine the cost to complete a certain task. A system with high throughput but low goodput drives cost per task up no matter how cheap each individual token is priced. A system with high goodput converts infrastructure spend into completed work.

Crusoe's inference engine powered by MemoryAlloy^TM technology is built specifically for this. Cluster-wide KV caching and context-aware routing mean an agent maintaining context across a multi-step workflow doesn't trigger redundant prefill recomputation when its requests land on different nodes. The cache follows the context, wherever the request goes. Combined with the NVIDIA Dynamo serving architecture underneath it, the result is fewer wasted tokens, more completed tasks, and a cost per task that reflects real output.

The full picture

In conclusion, tokenomics can't be judged on any single number. A fuller picture combines efficiency (tokens per watt), unit economics (cost per token), outcome economics (cost per completed task), and throughput quality (goodput). The right combination of hardware and software doesn’t just provide maximum performance, efficiency, and reliability at scale, it determines the real cost of intelligence.

Crusoe's stack, co-designed with NVIDIA from physical infrastructure to accelerated computing, open models, and open-source software innovations like fastokens and MemoryAlloy technology, maximizes the denominator: more tokens delivered, fewer wasted, on infrastructure that sustains it at scale.

In the agentic era, organizations won’t compete on who rents GPUs most cheaply. They’ll compete on who converts infrastructure into completed work most efficiently. The teams who choose infrastructure designed for the full agentic loop — constantly optimizing for cost per token, goodput, and valuable output — will compound a cost advantage with every agent they ship. 

This is what it means to look below the waterline.