Cloud
Engineering
November 18, 2026

A 9B model beats a 1T model on FinQA, trained with Crusoe Managed RL

With Crusoe Managed RL, a 9B open model beat a 1T model on a multi-turn FinQA task. GRPO took Qwen3.5-9B from 53.1% to 73.9% held-out, passing Kimi K2.6 at a fraction of the inference cost. Here is the API, the recipe and the ablations behind the result.

Matan Halfon snapshot
Matan Halfon
Staff Research Engineer
Itay Niv headshot
Itay Niv
Staff Research Engineer
Daniel Shen headshot
Daniel Shen
Machine Learning Engineer
Suman Debnath headshot
Suman Debnath
Director, Developer Relations
Yiwei Song headshot
Yiwei Song
Senior Director of Model Lifecycle
Omri Berkovitch Photo
Omri Berkovitch
Director, Research
November 17, 2026
Lines fanning out from a single point on black, representing rollouts sampled from one prompt during RL training

Crusoe Managed RL, now in private preview, is an API for RL post-training of open-weight models using verifiable rewards (RLVR). You write the training loop, the environment and the reward, and run them on your own machine; Crusoe manages the GPU fleet: the samplers, the trainer and the weight sync between them, tuned to get the most out of the hardware, from 9B dense models to 1T+-parameter mixtures of experts.

With Crusoe Managed RL you spend your time on reward shaping, loss adaptation and defining your own agent environment, while the managed system takes care of sampler throughput, trainer memory (no more out-of-memory (OOM) errors) and the rest of the infrastructure tuning, so you can focus on what matters for your product.

This post covers the API and one result. We trained three Qwen models (9B to 35B) with Group Relative Policy Optimization (GRPO) on FinQA, a multi-turn task where the model has to find and query financial tables through tools. The trained 27B model reaches 74.1% on held-out questions, up from 48.2% before training: a gain of more than 20 points from a short run that lets the model adapt to the harness it is used in. That puts it ahead of Kimi K2.6, a 1T-parameter model, GLM-5.2 (744B) and Claude Sonnet 5, at a fraction of their inference cost.

How Managed RL works: you write the loop, Crusoe runs the GPUs

RLVR (reinforcement learning with verifiable rewards) is behind much of the recent progress in reasoning and agentic models, and it is the most direct way to teach an open-weight model your task: write a program that checks the answer, and the model learns from its own attempts.

The challenging part of RL post-training is the infrastructure. Every RL step uses two groups of GPUs, a sampler fleet and a trainer fleet, working in lockstep. A sampler fleet generates rollouts from the current policy, often over many turns with tool calls in between. A trainer computes gradients on those rollouts with a policy-gradient algorithm of your choice, updates the weights, and pushes them back to the samplers for the next step.

RL post-training is inference-heavy by design. Each training step needs on the order of a hundred trajectories to produce a useful signal, even for simple tasks, and far more for multi-turn setups. In practice the bottleneck is high-throughput inference, not the gradient step. Managed RL runs the rollouts on Crusoe's production inference stack and overlaps them with training, so each step keeps the hardware fully used.

So that is where Managed RL draws the line. The client is a Python library that runs wherever you do: a laptop, a notebook, a CI job. It controls everything about the task. The backend runs on Crusoe Cloud and handles everything that needs a GPU.

Figure 1: what runs where
Figure 1. What runs where. Prompts, tool results and advantages go to Crusoe; rollouts and metrics come back. Your environment and reward function run on your machine.
Client (your machine)Backend (Crusoe Cloud)
The training loop and algorithm settings: GRPO, group size, sampling, clippingGPU capacity, cluster management and scheduling
Choice of data and promptsHigh-throughput rollout sampling
Environments, single-turn or multi-turn with toolsForward and backward passes and optimizer steps on tuned kernels
Reward functions and reward shapingWeight sync from trainer to samplers after each step
Base model and LoRA configurationScaling from 9B dense models to 1T-parameter mixture-of-experts models

Table 1. The split between client and backend.

CallRuns onWhat it does
sampleCrusoe GPUsGenerates rollouts from the current policy. In a multi-turn environment the sampler stops at each tool call, your environment returns the result, and generation continues.
forward_backwardCrusoe GPUsRecomputes token log-probabilities on the trainer, applies your advantages and computes gradients.
optim_stepCrusoe GPUsApplies the optimizer update and syncs the new weights to the samplers.
save_stateCrusoe GPUsSaves the adapter and optimizer state for evaluation, resuming or deployment.
Your reward functionYour machineScores each rollout. Any Python works: unit tests, SQL checks, an LLM judge.

Table 2. The four API calls.

Here is what that looks like in practice. The complete training script below trains a Qwen model on FinQA, the task described in the next section. Your environment and your reward function are ordinary Python; the three client.* calls inside the loop are the only lines that touch a GPU.

import crusoe_mill as mill client = mill.TrainingClient(    base_model="Qwen/Qwen3.6-35B-A3B",    lora_rank=128,) for step, batch in enumerate(finqa_train.batches(size=64)):    # 16 rollouts per prompt, sampled on Crusoe GPUs; tool calls run in your env    rollouts = client.sample(batch, env=FinQAToolEnv(max_turns=48), n=16)     # Score them with your own reward, on your machine    rewards = [finqa_reward(r) for r in rollouts]     # GRPO update on Crusoe GPUs    adv = group_normalize(rewards, n=16)    client.forward_backward(rollouts, adv)    client.optim_step(lr=1e-5)     if step % 10 == 0:        client.save_state(f"finqa-step{step}")

How we trained the Qwen models: GRPO

Crusoe Managed RL does not pick the algorithm for you. Rewards and advantages are computed in your code, so the same four calls run any policy-gradient method, and you can adjust the optimization or try new techniques as they appear.

For the Qwen models we used one of the most widely adopted of these methods: GRPO, introduced in DeepSeekMath [Shao et al., 2024] and popularized by DeepSeek-R1. It learns efficiently from the kind of sparse, verifiable reward that RLVR provides.

The idea is simple: for each prompt, sample a group of rollouts, score them, and push the model toward the ones that beat their group's average. The group mean serves as the baseline, which lets GRPO skip the separate critic model that proximal policy optimization (PPO) needs; for a 27B policy, that critic would cost another 27B model's worth of GPU memory.

Looking back at the training script above, one GRPO step is exactly the body of that loop:

  1. Sample a group. client.sample(batch, env, n=16) generates 16 rollouts per FinQA question on Crusoe GPUs.
  2. Score. Your reward function gives each finished rollout one number, for example 1 for a correct answer and 0 otherwise.
  3. Compare within the group. group_normalize turns the rewards into advantages using the formula below. Every token in a rollout shares that rollout's advantage.
  4. Update. forward_backward computes a clipped policy-gradient loss that raises the probability of above-average rollouts and lowers it for below-average ones; optim_step applies the update and syncs the new weights to the samplers.
GRPO advantage formula: reward minus group mean, divided by group standard deviation

One consequence: a group where every rollout scored the same has zero advantage everywhere, so it contributes nothing. That is the reason for dynamic sampling, which comes up in the ablations below.

Why it fits multi-turn agents

In a multi-turn episode the reward arrives exactly once, when the agent submits its final answer. GRPO hands that episode's advantage to every token in every turn, so a correct answer reinforces the whole trajectory, including the tool calls that got there. Nothing has to be graded per turn, which is what makes it practical for a task like FinQA, where the only thing you can check is the final number.

The task: an agent over your own tables

Most of the questions a company wants answered live in its own data, in a warehouse or a document store the model cannot see from its prompt. The model has to go and get it through the tools a given harness exposes, and that turns a one-shot answer into a multi-turn episode: call a tool, read what comes back, decide what to do next.

This setting is well suited to reinforcement learning. The correct sequence of tool calls cannot be specified in advance for every question, but the final answer can be verified, and rewarding episodes that end in a correct answer is enough for the agent to learn an effective policy: which tool to call first, how to recover from an empty result, and when enough evidence has been gathered to stop.

FinQA: financial analysis over company tables

FinQA is a dataset of financial-analysis questions answered by a SQL agent, such as "What was the year-over-year revenue growth?" or "How did net margin change?" Answering a question requires identifying the relevant table among the dozens a company reports, extracting the figures, and performing exact arithmetic; many questions draw on several tables. The model never observes the filing directly and accesses the data only through four tools:

  • get_descriptions(company_name): lists the company's tables
  • get_table_info(company_name, table_name): returns the columns, types and sample values of one table
  • sql_query(company_name, table_name, query): executes one read-only SQL SELECT on one table
  • submit_answer(answer): submits the final answer in \boxed{} and ends the episode

We train on templated questions about 165 companies and evaluate on two sets: an in-distribution test set covering 22 held-out companies, and SnorkelFinance, an external benchmark of 290 expert-verified questions with its own tables, tool descriptions and grader. The latter is the more informative test, since its questions were written by a different team in a different style.

A 9B Qwen model beats a 1T model on FinQA

We post-trained three open Qwen models with the training script above: Qwen3.5-9B and Qwen3.6-27B, both dense, and Qwen3.6-35B-A3B, a mixture-of-experts model with 3B active parameters. Each started from its public checkpoint and trained a LoRA adapter. For comparison we ran two much larger open models, Kimi K2.6 (1T parameters) and GLM-5.2 (744B), and Claude Sonnet 5. Every model went through the same harness: the same system prompt, tool descriptions and grader, up to 20 tool turns with a 4,096-token output cap per turn, thinking enabled for GLM-5.2 and Kimi K2.6, and Claude Sonnet 5 called through the Anthropic API with default settings. Scores are pass@1 on the full 290-question held-out set.

Figure 2. Held-out FinQA accuracy against the estimated cost of 1,000 tasks at list price. The trained Qwen models are priced on a Crusoe Self-Serve Deployment of two NVIDIA H200 GPUs ($6 per GPU-hour) at a steady 1,000 output tokens per second, assuming the deployment is fully used; the untrained Qwen base models at Crusoe Serverless Inference rates for their size class; GLM-5.2 and Kimi K2.6 at their publishers' API list prices ($1.40 / $4.40 and $0.95 / $4.00 per million input / output tokens); Claude Sonnet 5 at Anthropic API rates. Token counts per task come from each model's own FinQA runs: about five tool turns and 24,500 input tokens, with 2,500 output tokens for the Qwen models, 4,200 for GLM-5.2 and 4,500 for Kimi K2.6, both run with thinking on. Prices as of November 17, 2026. Dashed lines split each axis at its midpoint; better is top left.

* Moonshot AI's documentation notes that the Kimi K2 series will be retired at the end of December.

ModelID testHeld-out (SnorkelFinance)
Qwen3.5-9B base70.7 (83.5)53.1
Qwen3.5-9B + Managed RL90.1 (94.1)73.9
Qwen3.6-27B base76.048.2
Qwen3.6-27B + Managed RL91.274.1
Qwen3.6-35B-A3B base64.750.5
Qwen3.6-35B-A3B + Managed RL89.2 (92.1)70.9
Kimi K2.6 (1T)80.466.9
GLM-5.2 (744B)86.372.1
Claude Sonnet 585.971.4 ± 2.6

Table 3. FinQA accuracy (%), pass@1 with pass@8 in brackets where available. Best result per column in bold. Trained held-out scores are each run's peak checkpoint.

RL lifts every Qwen model by more than 20 points on the held-out SnorkelFinance questions: Qwen3.5-9B goes from 53.1% to 73.9%, Qwen3.6-27B from 48.2% to 74.1%, and Qwen3.6-35B-A3B from 50.5% to 70.9%. The dense models (74.1% and 73.9%) end 6 to 7 points above Kimi K2.6 (66.9%), a 1T-parameter model, and 2 to 3 points above GLM-5.2 (72.1%) and Claude Sonnet 5 (71.4 ± 2.6); the MoE model (70.9%) is on par with those two. Because the held-out questions come from a different team with different tool descriptions, the models learned a better search strategy rather than the quirks of the training harness.

In distribution, the trained models lead every reference model: the 27B model reaches 91.2%, the 9B model 90.1% and the 35B-A3B model 89.2%, against 86.3% for GLM-5.2, 85.9% for Claude Sonnet 5 and 80.4% for Kimi K2.6. Pass@8 moves in step with pass@1, so the improvement is in the model's first answer, not only in its coverage.

Serving the trained model

Accuracy is half the story. The other half is what each answer costs, and this is where a small model you trained yourself pulls away. At list prices as of November 17, 2026, 1,000 FinQA questions cost about $8 with a trained Qwen model, $41 with Kimi K2.6, $53 with GLM-5.2 and $74 with Claude Sonnet 5. That leaves the Qwen models alone in the top-left of Figure 2: ahead of GLM-5.2 and Claude Sonnet 5 on accuracy, well ahead of Kimi K2.6, at a quarter to a ninth of the price.

The pricing models differ because the deployments do. GLM-5.2, Kimi K2.6 and Claude Sonnet 5 are stock models you rent per token, so they are priced at their publishers' list prices. The trained model is dedicated to you, in this case the Qwen models, so it runs on a Self-Serve Deployment: two NVIDIA H200 GPUs at a flat $12 per hour, whatever the traffic. The $8 assumes steady traffic that keeps the deployment fully used. Training is managed the same way: jobs get dedicated GPUs, and you never touch the hardware.

How the cost per task is computed

Cost per task is each model's rate times the tokens a FinQA question actually consumes. GLM-5.2 and Kimi K2.6 are priced at their publishers' API list prices: $1.40 / $4.40 per million input / output tokens for GLM-5.2 (Z.ai pricing) and $0.95 / $4.00 for Kimi K2.6 (Moonshot AI pricing); the same rates are listed on OpenRouter. Claude Sonnet 5 is $2 / $10 through the Anthropic API (Anthropic pricing). The trained Qwen models are priced by the hour on their dedicated deployment: at a steady 1,000 output tokens per second, 1,000 tasks of 2,500 tokens each keep the two NVIDIA H200 GPUs busy for about 40 minutes, or $8 at Crusoe's Self-Serve rate of $6 per GPU-hour (Crusoe pricing). We assume no prompt caching for the API models; it would narrow the gaps without changing the order. All prices are list prices as of November 17, 2026, from the linked pages; Crusoe does not offer GLM-5.2, Kimi K2.6 or Claude Sonnet 5 as hosted models, and their prices are quoted only for comparison.

The token counts come from the FinQA runs themselves. A question takes about five tool turns, and because every turn re-sends the conversation, input adds up to about 24,500 tokens per task for every model. Output is what separates them: a trained Qwen model writes about 2,500 tokens per task, GLM-5.2 about 4,200 and Kimi K2.6 about 4,500, both with thinking on. The big models spend most of their budget thinking. The trained small model just answers.

Ablations: what the recipe depends on

Which parts of the recipe matter? We took Qwen3.6-35B-A3B and turned off one setting at a time. The full recipe is GRPO with 16 rollouts per prompt, LoRA rank 128 on the attention of all 40 layers (10 full-attention, 30 linear-attention) and 40 RL steps, scored at the best-validation checkpoint. Two parts of the recipe decide which prompts the model trains on, and they are the ones to know about:

  • Dynamic sampling [DAPO] drops groups in which every rollout got the same reward, since they carry no learning signal. An adaptive in-flight scheduler keeps enough groups generating to absorb the drop rate and cancels stragglers once the step is full.
  • Revisit pool replays prompts the model has not mastered. Half of each step's prompts are fresh. The other half come from prompts whose last group was mixed or all-fail, sampled in proportion to learnability p(1 − p) [Foster & Foerster, 2025], following the replay-or-fresh scheme of Prioritized Level Replay [Jiang et al., 2021]. Success-rate prioritization [Kimi k1.5] favors unsolved prompts; we instead down-weight prompts that fail every rollout, because an all-fail group has no GRPO signal and dynamic sampling would drop it anyway. A prompt is revisited at most once every five steps.
Figure 3: FinQA ablation of Qwen3.6-35B-A3B
Figure 3. Each row removes one component from the recipe: ID-test and held-out accuracy at the best-validation checkpoint, and the hours until validation reaches 90% of the run's gain. Single runs; run-to-run noise is about ±3 points on the ID test and ±7 on held-out; validation every five steps.
ConfigurationID testHeld-out (SnorkelFinance)Time to 90% of gain
Base model (no RL)64.750.5—
Full recipe89.270.90.84 h
Group size 16 → 1287.6 (−1.6)70.0 (−0.9)0.62 h
Group size 16 → 887.4 (−1.8)68.2 (−2.7)0.26 h
Dynamic sampling off86.1 (−3.1)*69.6 (−1.3)0.71 h
Revisit pool off87.0 (−2.2)66.3 (−4.6)2.59 h
LoRA on full-attention layers only87.6 (−1.6)67.4 (−3.5)0.85 h

Table 4. Single-setting ablations. Accuracy (%), pass@1 averaged over eight samples, at the best-validation checkpoint. LoRA rank 128, 40 RL steps. Time = hours until validation reaches 90% of the run's gain. * The only difference larger than run-to-run noise.

The full recipe takes the model from 64.7% to 89.2% in distribution and from 50.5% to 70.9% on SnorkelFinance, and removing any one part lowers both. One caveat: these are single runs, and run-to-run noise is about ±3 points in distribution and ±7 on held-out. Only the drop without dynamic sampling clears that bar; the rest are consistent directions rather than measured gaps.

  • Revisit pool. Without it, held-out accuracy drops 4.6 points and training slows: steps take 420 s instead of 290 s, and validation takes 2.59 h instead of 0.84 h to reach 90% of its gain. More fresh prompts come back all-solved, so dynamic sampling has to oversample to fill each step.
  • LoRA coverage. Adapters on the 10 full-attention layers alone match the recipe on validation but score 3.5 points lower on held-out.
  • Group size. Groups of 8 reach 90% of their gain in 0.26 h, under a third of the recipe's 0.84 h, but end 2.7 points lower on held-out.
  • Dynamic sampling. Turning it off is the one change whose effect, −3.1 on the ID test, is larger than run-to-run noise.

What's next

FinQA is one task, but the recipe is general: put your data behind tools, write a reward that checks the final answer, and let RL find the strategy. Everything task-specific stayed in a few hundred lines of Python on our side; the cluster work stayed on Crusoe's. A 9B model trained this way beat a model about 100 times its size and edged out GLM-5.2 and Claude Sonnet 5, at a quarter to an eighth of their cost per question.

Crusoe Managed RL is available in private preview. Request early access for Crusoe Managed RL.

References

Get Crusoe stories in your Google results

Frequently
asked questions

Is it worth fine-tuning a small model instead of using a larger one?

Fine-tuning a small open model is worth it when the task has a checkable answer. In Crusoe's FinQA experiment, RL post-training with a LoRA adapter took Qwen3.5-9B from 53.1% to 73.9% on held-out questions, seven points above Kimi K2.6, a 1T-parameter model, and two to three points ahead of GLM-5.2 and Claude Sonnet 5. Running 1,000 tasks on a trained Qwen model costs about $8 at list price, against $41 to $74 for the larger models.

How is RL different from fine-tuning?

Reinforcement learning learns from a reward you define rather than from example answers. The model generates its own attempts, a program scores each one, and training pushes the model toward the attempts that scored well. That matters for agent tasks, where the correct sequence of tool calls cannot be specified in advance but the final answer can be verified.

What is GRPO in simple terms?

GRPO (Group Relative Policy Optimization) trains a model by sampling a group of rollouts for each prompt, scoring them, and pushing the model toward the ones that beat the group's average. The group mean acts as the baseline, so GRPO skips the separate critic model that PPO needs. For a 27B policy, that critic would cost another 27B model's worth of GPU memory.

How does RLVR work?

Reinforcement learning with verifiable rewards (RLVR) scores each model attempt with a program that checks the answer, for example 1 for a correct final answer and 0 otherwise. In a multi-turn agent episode the reward arrives once, when the agent submits its answer, and GRPO hands that episode's advantage to every token in every turn, so a correct answer reinforces the tool calls that led to it.

Latest articles

Chase Lochmiller - Co-founder, CEO
November 18, 2026
A 9B model beats a 1T model on FinQA, trained with Crusoe Managed RL
Chase Lochmiller - Co-founder, CEO
November 17, 2026
Serving GLM-5.3 coding agents faster on NVIDIA GB200 NVL72
Chase Lochmiller - Co-founder, CEO
November 16, 2026
Crusoe's MLPerf Training v6.1 results on AMD Instinct MI355X

Are you ready to build something amazing?