Preserving the trace: a guide to fine-tuning chain-of-thought models

Reasoning models are trained to think before they answer. The data most businesses want to fine-tune them on has no thinking in it at all. Put the two together and the reasoning does not just fade, it disappears, faster and more completely than we expected when we started measuring. This blog walks through how that happens, compares four ways to fix it, and asks a question that usually gets skipped. Was the reasoning ever worth paying for?

Here’s how we’ll get there. First, a quick primer on what a reasoning model actually is, and why the data most teams fine-tune on has no reasoning in it. Then we’ll show what happens when you train on that data anyway, the fixes we compared (with a simple check to run before you train anything), the results, and the probes that test whether the surviving reasoning means anything. We’ll wrap up with a set of recommendations that turns all of it into concrete decisions.

Reasoning models

The most useful definition of reasoning, in the context of LLMs, is a deliberately unphilosophical one: a reasoning model generates intermediate tokens before it commits to a final answer. Those intermediate steps can surface in two ways. Sometimes the model writes them out in the open, as a worked explanation the user reads alongside the answer. More commonly, in most current reasoning models, they are wrapped in special tags such as <think>...</think>, and the chat interface hides them, so the user only ever sees the final answer.

Either way, the substance is the same: the model has been trained to spend tokens working through the problem before answering it, not to leap from question to answer in one hop. This generated middle part is what people mean by chain of thought. In this blog we will simply call it the trace.

So how is that different from a conventional, non-reasoning LLM? The difference is easiest to see with two prompts.

First, a pure recall question:

Prompt: Who wrote One Hundred Years of Solitude?

Answer: Gabriel García Márquez.

A conventional LLM gets this right, but it is not deducing anything. It is completing text from a strong statistical association it absorbed during training. This behavior is usually called pattern matching: the model reproduces correlations learned from data rather than applying explicit rules. Pattern matching is what pretraining buys, and at scale it is genuinely powerful.

Now a question whose answer is stored nowhere and has to be worked out:

Prompt: A notebook costs $4 and a pen costs $2. Ria buys 3 notebooks and 2 pens. How much does she spend in total?

There is no association to recall here. The answer has to be constructed. A conventional model must jump from the question straight to “$16” in a single hop, and as problems like this grow longer, that hop increasingly fails. A reasoning model instead combines several intermediate steps to arrive at the conclusion, and writes them out:

Step 1: identify the quantities (3 notebooks at $4 each, 2 pens at $2 each)

Step 2: notebooks: 3 × 4 = $12

Step 3: pens: 2 × 2 = $4

Step 4: total: 12 + 4 = $16

Answer: $16

Because each generated token becomes part of the context for the next one, the intermediate results accumulate, and the final answer only has to take the last small step.

Two things are worth being clear about, because the word “reasoning” invites more than the mechanism delivers.

The steps are not logic: There is no rule engine or theorem prover inside the model. Every step above is still next-token prediction, statistical to the core, which is why a trace can be fluent and wrong at the same time. LLM reasoning is probabilistic where a symbolic system is deterministic, and it likely works differently from human reasoning too. When practitioners say the model “thinks,” they mean it in this practical engineering sense, and nothing in this blog depends on the trace resembling human thought.

The steps are not free: The trace is ordinary autoregressive output, generated one token at a time, and every one of those tokens costs a full forward pass through the model. The same mechanical fact that gives the trace its power (intermediate results written into the context are available to every later token) is what makes it expensive. Both halves of that trade-off will come back with numbers attached.

Finally, it helps to know where the behavior comes from, because that is exactly where fine-tuning attacks it. Every model in this study went through the standard pipeline: pretraining, which teaches language by next-token prediction over trillions of tokens, then post-training, which teaches the model to follow instructions and to answer in ways people prefer. The reasoning behavior is installed on top of all that, typically with reinforcement learning against verifiable rewards, sometimes by distilling a larger reasoning model, following the recipe DeepSeek-R1 made public. (A third family of techniques improves reasoning at inference time without touching the weights; it is not our concern here.)

Your data doesn’t think

Think about the data a typical enterprise fine-tuning project starts with: a customer arrives with 10,000 labeled examples and wants to specialize a reasoning model on them. For example:

• A support ticket and the intent it was filed under
• A clinical note and its billing code
• A contract clause and its risk rating

And in each case, the data records only the end result. The label is there; the reasoning that produced it is not.

Nobody is doing anything wrong here. This is just how work gets recorded. When a support agent tags a ticket with an intent, the deliberation happens in their head and stays there, because no step in the workflow ever asks for it. The reasoning was real, it just never made it into the data.

That leads to the question this blog is based on: what happens when you fine-tune a thinking model on data that contains no thinking, and how much should you care?

The first half has a clear answer: the model stops reasoning entirely. Valid reasoning rate goes from 100% before fine-tuning to 0% after, across every seed and every adapter rank we tried.

That leaves the second half: how much should you care? The answer depends on what you wanted the reasoning for in the first place, a question we think gets skipped far too often. It got skipped in our own first attempt at this study, and correcting for it changed our conclusions more than once.

The collapse

Let’s start with what we observed. Fine-tune Qwen3-4B-Thinking on banking77, a customer-support intent classification dataset, using the answers alone. The fine-tuned model’s valid reasoning rate is exactly zero. Not a single generation contains a non-empty reasoning block.

This is not a gradual degradation; the reasoning disappears completely. And the training tokens explain why. The usual term here is catastrophic forgetting (a model losing previously learned capabilities when trained on new data), which makes it sound passive. What is actually happening is more direct.

Qwen’s chat template always emits an opening <think> tag in the assistant turn. When the training row has no reasoning content to put inside it, the rendered training target looks like this:

<think>\n\n</think>\n\nautomatic_top_up<|im_end|>

That target contains nine supervised tokens, three of which are the empty reasoning block. Cross-entropy, the loss function used in standard SFT, rewards the model for assigning high probability to every supervised token, and the empty block is one of them. The model is not losing the ability to reason; it is being trained, on every row, to predict that no reasoning comes next. The complete disappearance of the reasoning is the expected result of the training objective, not a side effect.

Gemma-4 loses its reasoning too, but for a different reason: its template emits no thought block at all when reasoning content is absent, so the fine-tuned model never learns to open one. Qwen is trained to emit an empty block; Gemma is never shown a block to begin with. That difference will matter later in the blog.

The Reasoning-Trace Collapse paper (arXiv:2605.21127) reports the same collapse across four open-weight reasoning models fine-tuned on a chemistry corpus, with the same core finding: task accuracy stays high while explicit reasoning disappears. That paper documents the problem. In this blog, we run experiments on what to do about it: which fixes work, on which models, for which tasks, and when the reasoning is even worth keeping.

Why keep the reasoning?

The collapse looks like a defect, and if it is a defect, repairing it seems like the obvious next step. But before reaching for a fix, it is worth asking a more basic question: what were you going to use the reasoning for?

In practice, two motives get bundled together, and they behave differently under every intervention we tested.

1. Accuracy: Chain-of-thought helps on some task families and not others. The largest meta-analysis to date (Sprague et al., 2024) finds large gains on math and symbolic reasoning and, in their words, “much smaller gains” everywhere else. If your task falls in the second category, preserving the reasoning is a cost with no return: you pay for thousands of trace tokens on every request and get nothing back.
2. Auditability: You keep the reasoning because you want to see why the model answered the way it did. In regulated settings this is often the stronger motive, a model that shows its work can be reviewed, debugged, and deployed where unexplained decisions are not acceptable. But this only holds if the trace reflects the computation that actually produced the answer, and a substantial body of research says it frequently does not (Turpin et al., 2023; Lanham et al., 2023). A fluent explanation and a faithful explanation are two different things.

Both motives are measurable, and as we will show later with numbers, they do not rank the fixes the same way. A method that wins on accuracy can lose on auditability, and the other way around.

Beyond accuracy and auditability, there is a third reason to preserve the trace: generalizability. When a model retains its chain-of-thought, it learns a reusable problem-solving procedure rather than a direct mapping from query to label. This procedural thinking allows the model to extrapolate far better when presented with out-of-distribution inputs that lie outside your fine-tuning dataset.

The five arms

Every method below can be described in terms of a single training row, written as {question, trace, answer}, where the trace is the part missing from the customer’s data. We compare five training configurations, or arms: a baseline that trains on the data as it is, and four candidate fixes. The fixes split into two families:

• One family fills in the missing trace using a stronger model.
• The other changes the training objective and leaves the data alone.

Here is each arm in turn, starting with the baseline.

1. Trace-free is that baseline: fine-tune on the data exactly as it arrives. Reasoning collapses completely, as we just saw. It is also, more often than the framing suggests, a perfectly reasonable choice, and the results will show exactly when.
2. Distillation asks a stronger model (the teacher) to supply the reasoning your data lacks, then trains on the completed triple. Sample traces from one or more teachers, discard any trace whose final answer disagrees with your gold label, and train on what survives with ordinary cross-entropy. In the terms of the distillation literature, this is hard, off-policy distillation. Hard, because the student trains on the text the teacher wrote, not on the teacher’s full token-probability distribution (the classic setup of Hinton et al., 2015). Off-policy, because the traces are generated once up front, with no live teacher scoring the student’s own outputs during training. It is just SFT where a teacher wrote the rows, the same recipe behind the DeepSeek-R1 distilled variants and standard practice since Magister et al. (2022) and Hsieh et al. (2023). One structural limitation, which we will quantify later: you can only train on prompts a teacher solved, so the method is bounded by how well the teachers do on your task.
3. Style-selected distillation changes only which trace you keep. Where several teacher candidates solve the same prompt, take the one with the lowest perplexity under the student, where perplexity is roughly a measure of how surprised the model is by a piece of text. In other words, keep the explanation phrased most like something the student would have written itself. This is not a second generation pass and not more data: same prompts, same row count, different pick.
4. Masking plus a KL anchor needs no teacher at all. It combines two interventions that address different parts of the failure. The mask removes the empty reasoning block from the loss, so cross-entropy never rewards emitting one; that goes directly at the empty-block supervision we walked through in the collapse section. The KL term penalizes divergence from the frozen base model: a penalty that grows as the fine-tuned model’s token probabilities drift away from the base model’s. This protects capabilities the fine-tuning data never exercises. The data contains no reasoning, so nothing in the loss preserves it, and the KL anchor holds it in place instead. This is the same KL-to-reference constraint used in RLHF to keep a tuned model close to its starting point (Ziegler et al., 2019), and under LoRA (Hu et al., 2021) the reference model comes for free: just detach the adapter.
5. Masking alone is the same loss mask with no KL anchor. Running it separates the two components of the previous arm, so we can see which one does the work. It corresponds to the response-only variant in Reasoning-Trace Collapse.

Here are all five, drawn as what the loss function actually sees:

The failure is the red block in the first row: trace-free SFT spends supervised tokens teaching the model to open a reasoning block and immediately close it. Every other row avoids that in one of two ways: replace the block with real reasoning, or remove it from the loss.

The pre-flight check

There is a catch in the masking rows, and it is the single most important implementation detail in this blog. Masking is only available if your chat template actually emits an empty reasoning block to mask. Whether it does is a property of the template, not of the method. Watch the same trace-free row render three different ways:

You can check which case you are in before spending a GPU-hour, with one rendered row:

 Pre-flight check: run this before you train

Render one trace-free training row through your chat template and look for the block. The row needs an assistant turn; the maskable block lives in the training target, not in the generation prompt.

row = [{"role": "user", "content": "Where is my card?"},
       {"role": "assistant", "content": "card_arrival"}] 
       
print(repr(tok.apply_chat_template(row, tokenize=False)[-80:]))

# Output 
Qwen3-4B-Thinking   ...assistant\n<think>\n\n</think>\n\ncard_arrival<|im_end|>   -> maskable
gemma-4-31B-it      ...<|turn>model\ncard_arrival<turn|>                          -> nothing to mask

If a block appears, the full mask + KL recipe applies. If nothing appears, “mask + KL” silently reduces to a KL anchor alone, and as we will see, that half of the recipe on its own is the weakest thing we measured. One line predicts which intervention you will actually get.

The setup

Everything in this blog comes from one grid: three model families, three tasks, and five training arms, evaluated under conditions chosen to catch the failure modes we describe below.

Models

We picked the three model families because their chat templates render a trace-free training row in three different ways. As the results will show, that mattered more than their size or their architecture:

Only one of the three gives you an empty reasoning block to mask. On the other two, mask + KL turns into a different intervention, and not the same one on both; we cover this in detail later in the post.

Two implementation notes on gpt-oss-20b, recorded here because both cost us debugging time:

1. Its MoE architecture forces attention-only LoRA: the fused expert tensor exposes no MLP projections, so it trains with less adapter capacity than the other two models.
2. Its template reads the reasoning field under a different key than the other families. If you write only the common key, your distillation arms silently become trace-free arms; we hit exactly that, and our harness now emits both keys.

All three models are fine-tuned with LoRA, with the same configuration everywhere:

config = {
    # LoRA adapter
    "lora_rank": 32,
    "lora_alpha": 64,          # alpha/rank held constant, so changing rank
                               # does not silently change the effective LR
    "lora_dropout": 0.0,

    # training
    "epochs": 1,
    "precision": "bf16",
    "gradient_checkpointing": True,
    "optimizer": "adamw_fused",   
    "learning_rate": 1e-4,
    "lr_schedule": "cosine",
    "warmup_ratio": 0.03,
    "effective_batch_size": 8,    # per-device batch x grad accumulation
}

We deliberately did not tune hyperparameters per arm: every arm on a given model trains with an identical configuration, so differences between arms are attributable to the data and the objective, not to tuning.

The trade-off is that no arm sits at its individual optimum; the comparisons are like-for-like rather than best-versus-best.

Tasks

The tasks vary two properties independently: whether reasoning is plausibly load-bearing, and whether the labels are guessable from their names.

A quick description of each task:

GSM8K (Cobbe et al., 2021) contains grade-school math word problems, for example:

Janet's ducks lay 16 eggs per day. She eats three for breakfast every
morning and bakes muffins for her friends with four…

Each problem takes a few steps of arithmetic and has a single numeric answer. This is the task in our grid where reasoning is most plausibly load-bearing.

banking77 (Casanueva et al., 2020) is intent classification for online-banking customer support. A short user message maps to one of 77 fine-grained intents:

message: "My phone is not on me. How can I use the app?"
intent:  lost_or_stolen_phone

We chose banking77 because its labels behave like real proprietary data. The 77 intents are one company’s internal conventions, and some pairs (such as card_payment_fee_charged vs transaction_fee_charged) differ in ways only that company’s taxonomy defines, so a model cannot get them right from general knowledge; it has to learn them from the data.

AGNews is the topic classification of news articles into four categories: World, Sports, Business, Sci/Tech. The category names describe themselves, so a model can do well from the label names alone, which makes it the opposite of banking77 by design. We subsample it to 8k rows to match banking77’s size, so cross-task comparisons are not confounded by dataset size.

For the reasoning task, we train one adapter and evaluate it on three test sets, which is the most direct way to test whether a method generalizes rather than memorizes:

No evaluation question appears anywhere in training.

Every classification task is evaluated under two conditions, and as the results will show, the same arm can rank first under one and last under the other:

• Fixed taxonomy: train on all categories, test on held-out examples. The label space never changes.
• Expanded taxonomy: train on a subset of categories, test on both seen and unseen ones. This is the realistic case where a customer adds categories after fine-tuning.

Teacher traces

The distillation arms need teacher traces, so the first step was to generate them. We used three teachers (DeepSeek-V4-Flash, GLM-5.2, and Kimi-K2.6), sampled two traces from each per prompt, and kept any trace whose final answer matches the gold label. That gives up to six candidate traces per prompt, from which the distillation and style-selected arms make their picks. Teacher coverage on GSM8K came out at 96.4%, with the three teachers contributing near-evenly.

All three teachers were called through Crusoe Cloud Serverless Inference, behind a single OpenAI-compatible endpoint, so swapping teachers meant changing one string. In practice, generating traces from three teachers took about the same effort as generating them from one.

Row matching

One design constraint we enforced everywhere: arms being compared train on the same rows wherever possible.

Distillation and style-selection draw from one shared pool over identical prompts and differ only in which candidate is selected, so any gap between them comes from trace selection, not data quantity. This matters more than it might sound: an earlier version of this study reported a style-selection win that turned out to be a data-budget artifact, and we had to retract it.

Mask + KL, masking alone, and trace-free all train on the same rows, so those three are mutually row-matched too. The one exception is distillation against trace-free: teacher coverage is incomplete, so the distillation arms train on slightly fewer rows. We flag that where it matters in the results.

Metrics

Four metrics recur in every table below. They are worth reading together: as the probes later in this blog will show, pass@1 and VR% can both be high while echo% reveals that the reasoning is a copy of the answer, not deliberation.

Generation budgets are calibrated per model and task until truncation falls below 2%, rather than fixed at one token count. A fixed budget systematically handicaps whichever arm reasons longest, and which arm that is changes across the grid. We know this bias is real because we hit it: an earlier version of this work reported a base GSM8K score of about 74, and we briefly concluded the base model was mediocre at grade-school math. The real number is 96.5. The gap was almost entirely a 2,048-token cap, and re-running eight over-target cells overturned a conclusion we had already written down.

All grading routes through one frozen scorer covered by 49 golden regression tests, and every generation is stored raw, so a scoring fix costs a re-parse rather than a re-run. Where the text below calls a difference between arms significant, that is a paired bootstrap (10,000 resamples) over the shared evaluation prompts, with per-prompt correctness averaged across seeds.

We ran three seeds on GSM8K and on banking77’s expanded condition. The fixed-taxonomy conditions are single-seed controls, and AGNews uses two held-out-class folds (Tech and World) whose spread substitutes for seeds. The untuned-base rows are single runs; we measured about 0.7 points of run-to-run variation on those, which is the practical error bar on any single-seed cell.

Results

This section reports what every method does to accuracy. The sections after it interpret the results through two variables:

• task: whether reasoning helps at all, and whether the surviving reasoning is real
• chat template: which fix you actually get

If you want the conclusions without the evidence, the recommendations near the end summarize both.

GSM8K

For the reasoning task, the adapter is trained once and evaluated on three test sets of increasing distance from the training data.

Bold marks the best value in each column within a model block. Two gpt-oss style-selected cells sit at 3.0 to 3.3% truncation and are slight underestimates. Gemma and gpt-oss have no masking-alone arm: nothing to mask.

Two things to take from this table:

1. The trace-free collapse is not a distribution-shift effect: The trace-free arm is the lowest line in every panel, including in-domain, where it loses 62 points on Qwen (96.5 down to 34.5), 59 on gpt-oss, and 24 on Gemma. The model did not merely fail to generalize; it lost a capability it had before fine-tuning, on the exact data it was fine-tuned on. Training on 900 correct answers made the model substantially worse at producing them.
2. The best method is different for each model: Mask + KL is the top arm on Qwen and the weakest trace-preserving arm on Gemma, with the widest error bars in the grid; we explain why in the template section. Distillation is the steady row in all three panels: never the best arm, never the worst, on any family. That profile becomes relevant in the recommendations.

banking77

The classification results follow a different pattern from GSM8K, and it depends entirely on whether the label space changes after training.

Bold marks the best per column within a model block. Masking-alone exists only on Qwen.

Trace-free is the best arm on all three models when the taxonomy is fixed (tied-best on Gemma), and loses that lead entirely when the taxonomy grows. On Qwen it goes from best (93.3) to worst (68.7, below the untuned base at 72.0), with the same adapter and the same weights; only the evaluation condition changed.

This is also the one setting in the study where fine-tuning delivers a clear accuracy gain over the untuned base: +24.0, +10.0, and +12.6 points on the three families. That makes sense, because installing internal conventions is exactly what SFT is for: card_payment_fee_charged versus transaction_fee_charged is a distinction no model can get right from general knowledge, and the training data is the only place it exists.

AGNews

AGNews repeats the banking77 setup with self-describing labels instead of arbitrary ones.

The two “unseen” columns are separate folds; the named class is the one held out of training. Bold marks the best per column within a model block.

The same pattern as banking77, with smaller numbers: trace-free is best under a fixed taxonomy on all three models and worst or near-worst when a class is held out.

The two unseen columns also show how unstable single-fold numbers are. Holding out Tech costs Qwen’s trace-free arm 50 points; holding out World costs it 8. Same method, same model, same dataset, and the choice of held-out class moves the effect size by 6x. That fold-to-fold gap is the realistic error bar on any single number in these columns, and it is why we ran two folds and report both rather than averaging them.

  Do not model-select on in-distribution accuracy: In this grid, in-distribution accuracy was anti-correlated with out-of-distribution accuracy. Qwen’s masking-alone arm posts the best in-distribution number in the study (95.0 on seen classes in the expanded condition) and the worst out-of-distribution one (36.0). Nothing in the training logs warns you, because that arm also has the best loss curve. If you select checkpoints on seen-class accuracy, you can select for the wrong thing.

Does the task need reasoning?

The results tables report what happened. The next two sections explain it through the first of the two variables: the task.

  Probe provenance: The probes in this section and the next (suppression, filler substitution, early answering, corrupted-step injection) were run on an earlier iteration of this grid: reasoning task MATH rather than GSM8K, Qwen and Gemma only. They have not been re-run on the final grid or on gpt-oss, so treat their findings as general patterns about the methods and tasks, not as measurements of the exact models in the results tables.

Take the reasoning away

The most direct test of whether reasoning is load-bearing is to remove it. At inference time, we forced the reasoning block closed and required a direct answer, then measured what that did to accuracy.

‍

Accuracy deltas under reasoning suppression. (*Qwen’s base refuses to answer when denied room to think, so -80.0 measures a refusal, not a capability. † Gemma’s trace-free arm never emits a thought block anyway, so its +17.8 is a format artifact, not a benefit of silence.)

The spread across tasks is the finding. Suppressing reasoning costs around 46 points on math and close to nothing on classification, occasionally even helping slightly. In our experiments, the task, not the method, determined whether reasoning was load-bearing. This matches what Sprague et al. found at prompting time, now showing up at fine-tuning time, and it simplifies planning: the same question, whether the task is CoT-shaped, decides both whether to prompt for reasoning and whether to pay to preserve it.

Content, not compute

Suppression alone cannot distinguish between two explanations:

• the model used the content of the trace, or
• the model just needed extra tokens to compute over

Each generated token is a full forward pass, so even a content-free token buys the model one more round of computation over its context, and Pfau et al. (2024) showed that filler tokens can substitute for chain-of-thought on some algorithmic tasks. To separate the two, we replaced each trace with matched-length, content-free filler.

Filler never recovers the loss, and is usually worse than answering immediately. Qwen’s distillation arm on MATH scores 24.7 with filler versus 30.3 with no reasoning at all. This might look like it contradicts Pfau et al., but it does not: they note that exploiting filler tokens requires specific, dense supervision to converge, which our models never received. The 46 points come from the content of the trace, not the extra room.

The value arrives early

Following the early-answering probe of Lanham et al. (2023), we truncated each model’s own trace at various fractions and forced an answer:

‍

The first quarter of the trace delivers roughly 95% of the benefit; the remaining three quarters of inference tokens buy two or three points. We do not have a good explanation for why saturation is this fast, so we report the curve without one. (Gemma’s non-monotone mask + KL curve is consistent with that arm’s general instability, covered in the template section, rather than with anything about reasoning.)

This curve matters for inference cost, and we come back to it in the recommendations: if you are paying for preserved reasoning on every request, capping the trace length is a cheaper lever than switching training methods.

There is a critical caveat to the trace-free victory on classification: distribution drift. While stripping the trace achieves peak accuracy on the exact evaluation benchmark, it converts the model into a rigid pattern matcher. If your production traffic encounters novel phrasing, edge cases, or unseen user intents, trace-free models break easily. Preserving the reasoning trace acts as an insurance policy—the explicit decision path anchors the model to generalizable logic rather than brittle surface-level correlations, giving it significantly higher out-of-distribution (OOD) resilience.

Is the reasoning real?

Back in Why Keep the Reasoning? We named two motives: accuracy and auditability.

The previous section settled the accuracy question. Auditability needs something stronger than a visible trace: the answer has to depend on the trace, and that dependence has to be tested rather than assumed, because Turpin et al. (2023) showed models can produce fluent explanations that systematically misrepresent what drove the output. We ran two tests:

The echo test

What this probe checks: Real deliberation should contain something the answer alone does not. If the trace merely restates the answer, no deliberation happened, and the trace is decoration.

Consider the masking-alone arm on banking77. Its valid reasoning rate is 99.4%, which by the structural metrics is a full repair of the collapse. Reading the generations shows otherwise:

unable_to_verify_identity</think>unable_to_verify_identity

The model emits the answer, closes the reasoning block, and emits the answer again. 99.7% of its “reasoning” is the answer restated. It learned the shape of thinking without the substance, and every structural metric we had scored this as a complete success.

This bears directly on Reasoning-Trace Collapse, which reports masking as restoring valid reasoning from 0 to 84. Our mask-only arm reproduces that recovery, and it is almost entirely echo. Their measurement is sound, but what it means is narrower than it looks.

The corruption test

  What this probe checks: If the trace is a faithful account of the model’s reasoning, the answer should depend on it: change the trace, and the answer should change too. If the answer ignores the change, the trace is decoration.

The test follows the corrupted-step probe of Lanham et al. (2023): take the model’s own trace, inject an error into a decisive step, resume generation, and see whether the answer follows the injected error.

Corrupted-step results. Parentheses: change against the uncorrupted run. Trace-free is absent: no trace to corrupt.  († Gemma’s mask + KL reasons on only about a third of inputs, so read its follow rate, not its delta. ‡ Explained later.)

The pattern from the suppression probe reappears here, one level deeper: faithfulness is a property of an arm-task pair, not of an arm.

• On banking77, the untuned bases and mask + KL emit fluent reasoning whose content the answer almost never consults, with follow rates of 0.3 to 1.3%.
• On MATH, the same untuned bases follow a corrupted step 45 to 54% of the time, losing 16 to 33 points of accuracy in the process, and Qwen’s mask + KL goes from 0.3% to 47.8%.

The identical weights produce decorative reasoning on one task and load-bearing reasoning on another.

The distilled arms sit in between on both tasks, at 14 to 33% follow, which cuts both ways. Their traces are read everywhere, and they are the only arms with that property. But even at their most faithful, two thirds of their answers ignore a corrupted premise. Reasoning can be load-bearing in aggregate without the written trace being a faithful account of any individual answer, and aggregate accuracy cannot tell you which one you have.

The masking-alone row is a measurement artifact, and we initially misread it. Its follow rate is the highest on banking77 (30.7%)‡, which looks like faithfulness. But its traces are 99.7% echo, so corrupting the trace changes the string being copied. The high follow rate measures copying, not reasoning. No single probe is sufficient, and the three metrics only mean something together:

The verdicts attach to arm-task pairs, not arms. Both untuned bases and Qwen’s mask + KL are decorative on banking77 and dependent on MATH. Any audit-readiness claim has to name the task it was measured on.

If you wanted an audit trail

Suppose the whole reason you preserved the reasoning was compliance, where a reviewer needs to see why the model decided what it decided. What do these probes say about that plan?

• A visible trace vs an audit trail: On our classification tasks, an expert reviewing the model’s reasoning would be reviewing a plausible story, not the cause of the decision. The model wrote something that looks like a justification, and the corruption test shows the answer never consulted it.
• What the standard metrics miss: Valid reasoning rate scores the two worst offenders in our grid, one decorative and one copying, as complete successes. We only caught them by reading generations and injecting errors.
• Accuracy and auditability pick different methods: On classification, distillation’s traces are the only ones the answers actually consult: 14 to 23% follow an injected error, and accuracy drops 9 to 20 points under corruption. Mask + KL on the same task scores better on accuracy and produces traces that look excellent, but they are almost entirely decorative. If a reviewer needs to trust the trace, distillation is the right pick, and accuracy alone would have pointed you the other way.

None of this makes partially-faithful traces worthless. They surface alternatives the model considered and expose the working taxonomy, which has real review value. They are just weaker evidence than they appear, so test them on your own task before they carry compliance weight.

The template decides

The task sections explain when reasoning is worth paying for. They do not explain a pattern in the GSM8K table that the task cannot account for: the same recipe finishing first on one model and last on another.

As the results showed, the best method is different for each model, and the winner on one is the loser on another.

• On Qwen, mask + KL is the strongest arm on all three test distributions and statistically matches the untuned base (96.0 / 95.5 / 77.9).
• On gpt-oss it is also at or above base everywhere (95.5 / 95.3 / 81.1).
• On Gemma it is the worst trace-preserving arm, 66.5 in-domain, below trace-free, and this is not noise: its cells carry ±7.8 and ±5.6, the widest spreads in the grid, and its valid reasoning rate only reaches 30%.

The reason is the one we flagged in the pre-flight check. Gemma has nothing to mask, so its “mask + KL” is a bare KL anchor. gpt-oss masks a final-channel header instead, which turns the arm into response-only training plus a KL anchor, an intervention that happens to work well. The same nominal recipe is three different interventions across three families.

Decomposing the recipe

Mask + KL bundles two interventions, so before trusting it we wanted to know which one does the work. Our grid happens to contain the answer: Qwen ran the full recipe and the mask alone, and Gemma’s version, thanks to its template, is effectively the KL anchor alone.

Decomposing the recipe. Neither component works on its own.

The mask by itself recovers nothing. It scores 32.8, statistically indistinguishable from trace-free at 34.5, and valid reasoning stays at zero. The KL anchor by itself is not much better: it keeps reasoning alive on only 30% of inputs, and its ±7.8 is the widest error bar in the grid. You need both, and you can only have both if the template gives you something to mask.

Which brings us back to what the mask actually does on each family:

The same configuration flag, three different interventions. 

  A mask rate of 100% proves nothing: On gpt-oss the mask rate reads 100%, which looks like the mask is working. But what got masked was a channel header, not a reasoning block. The number tells you masking happened; it says nothing about whether the right thing was masked.

This also sharpens the strategy-dependent and model-dependent finding in Reasoning-Trace Collapse: in our runs, that dependence traces cleanly to the chat template. If you take one operational thing from this blog, take the pre-flight check.

The fine-tuning tax

One more pattern in the GSM8K table needs an explanation. Even the best arms can trail the untuned base on MATH-500. This is expected, not a failure of the methods. Training on 900 grade-school problems pulls the model toward grade-school mathematics, and some loss at the far end of the difficulty range is the price of any narrow fine-tune. So when judging an intervention, the fair reference is trace-free, which scores 24.4, 34.4, and 19.2 on the three families, not the base.

Measured against that reference, the picture changes.

• Distillation recovers 39 to 58 points on the hardest held-out test.
• Mask + KL on Qwen and gpt-oss lands statistically at the base itself (77.9 vs 78.3 and 81.1 vs 77.7).
• Where the base leaves headroom, the teacher arms actually beat it, with gains of +6.7 and +7.8 in-domain on Gemma and +5.5 for style selection on gpt-oss, all significant under the paired bootstrap.

Gemma makes the fine-tuning tax easiest to see because its base is the strongest MATH-500 model in the study at 84.3. The more far-transfer capability a model starts with, the more a narrow fine-tune has to lose. There is a real decision hiding in that. If capability far from the training distribution matters more to you than the fine-tuning task itself, do not fine-tune.

Distillation

Distillation never reads the chat template, which is why it is the only arm that behaves the same way on all three families. What it cannot ignore is the teacher.

First, what the teacher traces delivered.

• Full reasoning restoration on every family. Valid reasoning sits at 99 to 100% on Qwen, Gemma, and gpt-oss alike. It is the only intervention that never needed a caveat about the chat template.
• Real gains where the base had headroom. The teacher arms add +6.7 and +7.8 over the untuned base in-domain on Gemma and +5.5 for the style-selected variant on gpt-oss, all significant under the paired bootstrap, so this is improvement rather than mere recovery.
• The only traces the answers consult on every task we probed. Distilled traces show causal dependence everywhere, with 14 to 33% following an injected error and 9 to 20-point accuracy drops under corruption. Every other arm’s faithfulness is task-bound.
• No pre-flight dependence. It works the same regardless of what the template renders, which makes it the one recipe you can adopt without first auditing the model family.
• A free upgrade path. Style selection reuses the same traces for an average +1.1 points (more on this below).

Generating the traces is a one-time, training-time cost. The structural limit is a different matter, and no budget makes it go away. You can only train on prompts some teacher got right.

On banking77, 1,002 of 7,809 prompts were solved by no teacher in six attempts across three frontier models. And those 1,002 are not randomly scattered. They are the convention-defining cases, card_payment_fee_charged versus transaction_fee_charged, encoding exactly what a general-purpose model cannot guess about a private taxonomy. Coverage also degrades as the label space grows, from 87.2% at 60 intents to 84.6% at 77. This is the uncomfortable part of the method. The cases you most need a teacher for are the ones teachers are worst at.

Style selection

Selecting the lowest student-perplexity trace costs nothing extra. It is a filter over traces already generated. Across the full grid it beats plain distillation in 16 of 24 matched comparisons, with a mean gain of +1.1 points and a best case of +7.0 (Qwen, AGNews with Tech held out). It is not uniformly safe. Its worst case is -4.8 (Qwen, MATH-500). The fair summary is a free option with positive expected value and occasional losses, not a strict improvement.

We do not have a good explanation for why it works. Lower-perplexity traces may help because they match the student’s style, or simply because they tend to be longer, and longer traces alone improve accuracy on these tasks. Separating those hypotheses requires a length-matched control we did not run, so we report the effect and leave the mechanism open.

Recommendations

Everything above compresses to five lines, and then to six decisions.

Each intervention in one line. The sections above carry the numbers and the caveats.

And as a decision flow:

The same decisions, spelled out with the evidence behind each:

1. Fixed label space, classification-shaped task: Train trace-free. It was best or tied-best on all three models, it is the cheapest at inference, and the reasoning it destroys was neither helping accuracy nor faithful.
2. Label space will grow after fine-tuning: Preserve traces, and treat the result as insurance. No arm beat the untuned base significantly on unseen categories. What trace preservation buys is holding roughly base-level accuracy where trace-free falls far below it, while keeping the large seen-category gains. Which trace arm held best varied by family, so run the pre-flight check first.
3. Reasoning-heavy task: Distill. Trace-free cost 24 to 62 points in-domain, and teacher traces recovered 39 to 58 of them on the hardest held-out test, on every family.
4. Auditability required: Distill, then verify on your own task. Suppress the reasoning and check whether accuracy moves. Corrupt a step and check whether the answer follows. Measure echo before believing either. If nothing moves the answers, the trace is a narrative, not an explanation.
5. No teacher available: Mask + KL, after the pre-flight check. It was the best arm on Qwen and gpt-oss and the worst trace-preserving arm on Gemma. The same configuration is three different interventions across our three families, so reproduce the result on your own model before relying on it.
6. Unsure: Start from distillation. Never the best arm and never the worst on any family, the only faithful traces, and no dependence on the chat template. Treat the other methods as optimizations to reach for once the pre-flight check and a faithfulness pass have said they apply.
7. Evaluate your data drift: If your task has a strictly fixed domain where inputs are static and predictable, trace-free SFT offers the best speed-to-accuracy ratio. If your production environment faces high input variance and out-of-distribution (OOD) prompts, pay the fine-tuning tax to keep the trace. The reasoning steps are exactly what keep the model adaptable in the wild.

The recurring cost

One more lens, because every decision above is also a compute decision, and the two costs land at different times.

Distillation costs you once, at training time. Three teachers at two samples each is a fixed amount of compute spent before the adapter exists, plus the structural gap that 10 to 15% of your prompts will not be covered no matter how much you spend. Preserved reasoning costs you on every request, forever, and the mechanics of why are worth spelling out. An LLM generates one token at a time, and every token requires a full forward pass through the model, so an answer ten times longer costs roughly ten times the compute and latency. Our reasoning arms emit thousands of characters of trace per answer where trace-free emits none.

On fixed-taxonomy classification, that per-request cost returns nothing. Suppressing the teacher arms’ reasoning moves banking77 accuracy by only -1.2 to +2.1 points, so the trace tokens multiply latency and cost with no gain in accuracy. And even where reasoning is load-bearing, the early-answering curve applies. The last three quarters of the trace adds two or three points. If inference cost matters, capping trace length is a more efficient lever than switching training methods.

Closing

The data a business produces records outcomes, not the reasoning behind them, and fine-tuning a reasoning model on outcomes alone trains the reasoning away. That much is mechanical, visible in nine rendered tokens, and reproducible across every family we touched.

Whether it matters depends entirely on what the reasoning was for. On a fixed task, the collapse works in your favor, since the trace tokens were adding cost without adding accuracy. Under distribution shift, it is a real loss, and a teacher can repair most of it. For trust, it is a property to measure rather than assume, because the same weights that reason faithfully on one task will produce decorative reasoning on another, and every structural metric will score the decoration as a success. Not every task needs a thinking model, and matching the tool to the job is a decision, not a default.

The pre-flight check costs one line. The faithfulness probes cost one evaluation pass. Run both before committing a fine-tuning budget, and spend according to which regime you are actually in.

All of the teacher traces in this study were generated through Crusoe Cloud Serverless Inference, behind a single OpenAI-compatible endpoint, swapping between three frontier teachers meant changing one string. If you're planning a fine-tuning project of your own, get started with Crusoe Cloud and run the pre-flight check before you commit the budget.

References

Papers

• DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948.
• Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the Knowledge in a Neural Network. arXiv:1503.02531.
• Hsieh, C.-Y., Li, C.-L., Yeh, C.-K., Nakhost, H., Fujii, Y., Ratner, A., Krishna, R., Lee, C.-Y., & Pfister, T. (2023). Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes. arXiv:2305.02301. Findings of ACL 2023.
• Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685. ICLR 2022.
• Lanham, T., Chen, A., Radhakrishnan, A., et al. (2023). Measuring Faithfulness in Chain-of-Thought Reasoning. arXiv:2307.13702. Source of the early-answering and corrupted-step probes used here.
• Magister, L. C., Mallinson, J., Adamek, J., Malmi, E., & Severyn, A. (2022). Teaching Small Language Models to Reason. arXiv:2212.08410. ACL 2023.
• Pfau, J., Merrill, W., & Bowman, S. R. (2024). Let’s Think Dot by Dot: Hidden Computation in Transformer Language Models. arXiv:2404.15758.
• Reasoning-Trace Collapse: Evaluating the Loss of Explicit Reasoning During Fine-Tuning. arXiv:2605.21127.
• Sprague, Z., Yin, F., Rodriguez, J. D., Jiang, D., Wadhwa, M., Singhal, P., Zhao, X., Ye, X., Mahowald, K., & Durrett, G. (2024). To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning. arXiv:2409.12183. ICLR 2025.
• Turpin, M., Michael, J., Perez, E., & Bowman, S. R. (2023). Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. arXiv:2305.04388. NeurIPS 2023.
• Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., & Irving, G. (2019). Fine-Tuning Language Models from Human Preferences. arXiv:1909.08593. The KL-to-reference constraint follows Jaques et al. (2017; 2019).

Models

• Qwen3-4B-Thinking-2507
• gemma-4-31B-it
• gpt-oss-20b
• DeepSeek-V4-Flash
• GLM-5.2
• Kimi-K2.6

Datasets and benchmarks

• GSM8K (Cobbe et al., 2021)
• banking77 (Casanueva et al., 2020),
• AGNews,
• SVAMP,
• MATH,
• MATH-500

‍

‍