← Back to research

Small-Model Distillation — Part 3: Student-State Hard-Token Correction SFT for a 0.8B SQL Agent

2026-06-15

TL;DR

Before this experiment, the best 0.8B SQL agent came from top-k soft-label distillation and reached 55/220 on the fixed eval. That student was already much better than the base model, but it still failed in very agentic ways: wrong submissions, SQL errors, repeated actions, and bad recovery after its own tool calls.

This post tests a different question: what happens if the student generates the states, but GPT 5.5 medium takes over from selected difficult states and finishes the trajectory? This is DAgger-style expert correction SFT: the states come from the student, the teacher gives hard-token continuations, the harness verifies them, and the student trains with supervised fine-tuning. There are no teacher probabilities or logprobs here. Post 04's token-level probability distillation is a separate method.

The trajectory of results tells the real story. The first correction model moved the top-k soft-label student from 55/220 to 58/220. Mixing verified wrong-submit repair rows with successful-retention rows pushed the best direct repair model to 64/220. Repeating that recipe from the 64/220 parent reached 63/220 and stopped, so the loop did not compound. The final hard-token result, 67/220, came from a retention-heavy anti-forgetting extension: I mixed successful traces from complementary earlier students on tasks the 64/220 parent missed with four copies of parent-success retention, trained with normal hard-token SFT, and selected the best checkpoint.

The useful lesson is not "student-state correction solved it." It is more specific: hard-token correction helped the 0.8B agent recover from states it actually visits, retention anchors helped with forgetting, and a small amount of complementary successful behavior pushed the best hard-token checkpoint to 67/220, but SQL judgment and loop stability still dominate the failures.

What I Wanted To Test

The hard-token and soft-label experiments before this were both off-policy. The student learned from teacher-created trajectories. That is clean, but it has a limitation: during evaluation, the student does not always follow the teacher's path. It makes its own mistakes, receives different SQL observations, repeats actions, reaches bad intermediate states, and then has to recover.

The question for this post is:

Can GPT 5.5 medium improve the best 0.8B SQL agent by taking over from the states that the student actually reaches during rollout?

The important constraint is that I want to use hard-token supervision only. I do not use a probability-exporting teacher in this post. GPT 5.5 medium is the correction teacher, and the training target is the canonical action text that the harness already expects.

The SQL-Agent Problem

The task is the same deterministic SQL tool-use harness used for the off-policy hard-token and top-k soft-label experiments. The model receives a user issue and buggy SQL. It can inspect schema, run SQL, receive SQLite observations, and finally submit corrected SQL. A hidden deterministic scorer checks whether the submitted SQL passes the task.

The action interface is defined with BAML, or Boundary Markup Language, which I use as the structured-output layer around the model. BAML renders the chat request, defines the allowed SQL-agent action shape, and parses the model response into an executable object for the harness. I am not changing that layer in this post. The point is not to make the benchmark easier; the point is to change where the training states come from.

This matters because the model is not just writing a final answer. It is controlling a small loop. It has to decide when to inspect, when to query, how to use observations, when to stop, and what final SQL to submit. A model can improve at harness control while still submitting wrong SQL, so success rate and failure decomposition both matter.

Here is the kind of task the agent sees. The user has a bookstore database and a buggy query that tries to find orders where order_id jumps by at least 20% compared with the previous order. The original SQL uses a window function in the wrong direction and then filters in a way that returns too many rows. The agent may first inspect schema, then run a query, then receive an SQLite error such as misuse of aliased window function lg, and only then decide what corrected SQL to submit.

The canonical action shape is intentionally small:

{"draft": "short plan", "output": {"action": "inspect_schema"}}
{"draft": "short plan", "output": {"action": "run_sql_query", "sql": "SELECT ..."}}
{"draft": "short plan", "output": {"action": "submit_sql", "sql": ["SELECT ...;"]}}

That small schema is important. It means the training target is not a vague assistant answer. It is the exact structured action the harness can parse and execute.

The Training Idea

Student-state data collection means the training states come from the current student behavior. A policy is just the model's behavior: what action it tends to take from a given context. In off-policy distillation, the data comes from another policy, usually the teacher. In this experiment, the student acts first, and then GPT 5.5 medium gives hard-token corrections from the states the student actually visited.

This is close to the classic DAgger idea: run the learned policy, collect the states it visits, ask an expert what should happen there, and aggregate those expert labels into supervised training data. The same idea has started to show up in language-agent expert-correction work, where a rollout starts with the student and then switches to an expert model partway through the trajectory.

For this SQL agent, the difference looks like this:

Off-policy:
teacher history -> teacher next action
---
Student-state correction:
student history -> teacher continuation

That small change is the point of the post. A clean teacher trace may never contain the student's weird bad state. But if the student runs a bad SQL query, receives an error, and starts repeating itself, we can ask GPT 5.5 medium to take over from exactly that context.

The verified continuation can then become several SFT rows:

student_bad_history -> GPT action 1
student_bad_history + GPT action 1 + observation -> GPT action 2
student_bad_history + GPT suffix -> GPT submit_sql

This is still supervised fine-tuning. It is still hard-token distillation. The difference is where the input state comes from and how I verify the teacher output. I do not want to train on a correction just because it looks plausible. GPT has to run inside the real SQL environment and reach a final submission that passes the deterministic train-task tests.

There are two methods people often blur together. The first is black-box expert correction: the student creates the state, a teacher writes a better action or suffix, the environment verifies it, and the student trains on those hard tokens. That is the method in this post. The second is token-level probability distillation: the student samples from its current policy, a probability-exporting teacher scores those sampled tokens, and the trainer updates the student with a KL-style probability-matching objective. That second version needs teacher probabilities and a trainer built for probability matching, so I keep it separate from this hard-token correction experiment and compare against it in post 04.

Which Student-State Methods Work With Hard Tokens?

Yes, student-state data collection with hard-token teacher correction is possible. It is probably the cleanest version for this post because GPT 5.5 medium can produce the corrected next action even if we do not have its token probabilities.

The methods that fit this constraint are:

| Method | Uses student rollouts? | Uses GPT 5.5 medium? | Uses probabilities? | Fit for this post | | --- | --- | --- | --- | --- | | Teacher correction / DAgger-style expert-correction SFT | Yes | Yes | No | Best fit | | Recovery expert correction | Yes | Yes | No | Best fit | | Verifier-filtered teacher correction | Yes | Yes | No | Good fit | | Best-of-N teacher selection | Yes | Yes | No | Possible, but more expensive | | Hard-token self-training / self-distillation control | Yes | No | No | Later post | | Token-level probability distillation | Yes | Needs probability-exporting teacher | Yes | Post 04 comparison |

The main method should be teacher continuation from student-created states, similar in spirit to DAgger and recent expert-correction work for multi-turn LM agents. The student rolls out. I select difficult states. GPT 5.5 medium takes over in the same SQL environment. If the continuation passes the hidden train-task tests, the GPT suffix becomes hard-token SFT data for the 0.8B student.

Recovery distillation is a focused version of the same idea. Instead of labeling every state, we target failure states: repeated actions, SQL/tool errors, wrong-submit contexts, and near-max-turn states. This may be especially useful here because the best current student already knows the basic loop. The remaining problem is often recovery and decision quality.

Verifier-filtered teacher correction is the safety rail. I can ask GPT 5.5 medium for corrections, but I only keep rows that pass structural checks, fit the context limit, produce canonical actions that the harness can parse, and belong to a continuation that eventually passes the hidden train-task tests.

Best-of-N teacher selection is possible without probabilities too. The student can sample several candidate next actions or short continuations. GPT 5.5 medium or the deterministic harness can choose the best one, and we train on that selected action. I would treat this as a later ablation unless the basic correction setup is not enough.

The methods that do not fit this post are probability-based methods, because they need token probabilities, and self-distillation, because this post should use an external teacher. Post 04 can test the probability version: the student creates the states, a probability-exporting teacher scores those same sampled tokens, and the update uses a token-level distillation loss instead of only hard-token imitation.

A Small Educational Code Walkthrough

The canonical training row stays close to the earlier hard-token SFT rows. The model input is the full rendered conversation and environment history. The target is one GPT assistant action from a verified continuation.

row = {
    "messages": [
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": task_prompt},
        {"role": "assistant", "content": student_action_1},
        {"role": "user", "content": environment_message(sqlite_observation_1)},
        {"role": "assistant", "content": student_action_2},
        {"role": "user", "content": environment_message(sqlite_observation_2)},
    ],
    "target": gpt_5_5_medium_action_from_verified_suffix,
    "source": "student_rollout_teacher_correction",
}

For the bookstore example above, one student prefix looked like this in simplified form:

user: "Why does my lag query return almost every order?"
assistant: inspect_schema
environment: tables include cust_order(...)
assistant: run_sql_query with the student's buggy window-function query
environment: {"ok": false, "error": "misuse of aliased window function lg"}

GPT 5.5 medium then continued from that exact state inside the same SQL environment. It first ran a corrected ascending-lag query, observed the result, and then submitted a final SQL statement using LAG(order_id) OVER (ORDER BY order_id ASC) and order_id >= previous_order_id * 1.2. Because that continuation passed the deterministic train-task scorer, the suffix became training data.

The loss is normal assistant-token cross-entropy. Prompt tokens and previous history tokens are masked. Only the GPT 5.5 medium action target is trained.

labels = input_ids.copy()
labels[:prompt_and_history_token_count] = -100
loss = cross_entropy(student_logits, labels)

This is why student-state hard-token correction is practical with GPT 5.5 medium. I do not need GPT token probabilities. I only need the verified action text in the same canonical format the harness already parses.

The training data is still one-action-at-a-time supervised fine-tuning (SFT). If GPT takes three turns to recover from a student state, that successful suffix becomes three SFT rows, not one giant target. Each row asks the student to imitate the next verified GPT action from the exact history available at that point.

Teacher suffix to SFT rows sketchnote

How I Built The Correction Dataset

The data generation pipeline had eight stages:

  1. Start from the best top-k soft-label distillation checkpoint.
  2. Run it on the train split with the same harness, parser, tools, stop rules, and context budget.
  3. Save full student trajectories, not just final submissions.
  4. Bucket states by outcome: successful, wrong submit, repeated action, SQL/tool error, parse failure, and max-turn stop.
  5. Ask GPT 5.5 medium to continue from selected student-created states.
  6. Run the GPT suffix in the real SQL environment.
  7. Keep only successful continuations whose final submit_sql passes the deterministic train-task tests.
  8. Build a hard-token SFT dataset from the verified GPT suffix rows.

The key design decision was which states to label. Labeling every state is simple but expensive and includes many easy rows. Labeling only failure or near-failure states is cheaper and more targeted, but it may over-focus the training on recovery. For the first pass, I ran the best top-k soft-label student on the full 879-task train split and selected difficult states from that full rollout distribution.

One small detail mattered here. If I kept only the first candidate from each failed task, wrong-submit states dominated because many failed episodes ended with a submitted-but-wrong SQL query. That was useful, but it could hide earlier SQL-error or repeated-action recovery states inside the same episode. For the full correction pass, I used an all-buckets policy: at most one wrong-submit state, one SQL/tool-error state, one repeated-action state, one parse-failure state, and one near-max-turn state per failed task. This cost more GPT calls, but it matched the research question better because I wanted to test recovery from several kinds of student-created bad states.

I also kept the teacher call simple: one selected state got one GPT 5.5 medium continuation attempt. There was no best-of-N search and no retry loop in this experiment. GPT could take multiple tool-use turns inside that one continuation, but if the final submit_sql did not pass the deterministic train-task scorer, that candidate was discarded.

The first smoke test used two train tasks where the student submitted wrong SQL. GPT 5.5 medium produced continuations from both pre-submit states, but neither continuation passed the deterministic train-task scorer, so the filter kept zero SFT rows. That was not a result yet, but it was a useful pipeline check: unverified GPT output did not enter the dataset.

The first full train rollout finished on all 879 train tasks. The warm-started student solved 229 of them before any student-state correction training. From the failed and difficult trajectories, I selected 807 candidate states: wrong submits, SQL/tool-error states, repeated-action states, parse-failure states, and near-max-turn states. GPT 5.5 medium solved 345 of those candidate states under the real train-task scorer, which created 560 SFT rows. This final dataset keeps the mechanics strict: one GPT continuation attempt per state, environment execution at every tool step, deterministic verification at submit_sql, and only verified suffixes becoming training rows. The verified rows came from several buckets, not only wrong submits: 193 wrong-submit states, 76 SQL-error states, 62 repeated-action states, nine max-turn states, and five parse-failure states.

Final student-state correction dataset by failure bucket

The Method, Step by Step

Every experiment in this post is the same hard-token loop with a different data mix. The student rolls out, GPT 5.5 medium continues from selected states, the unchanged SQL harness executes and verifies the continuation, and only accepted assistant actions become SFT rows.

Algorithm: Student-state hard-token expert correction

Input: current student checkpoint, fixed 879-task train split, unchanged BAML SQL-agent harness, GPT 5.5 medium correction teacher.

Output: verified hard-token SFT rows and a LoRA adapter, scored on the fixed 220-task eval split.

  1. Run the current student on all train tasks and save full trajectories.
  2. Select difficult states from failed trajectories: wrong-submit, SQL/tool error, repeated-action, parse-failure, and near-max-turn.
  3. Replay each state so the teacher sees exactly what the student reached.
  4. Ask GPT 5.5 medium for one continuation, and let the harness execute every action.
  5. Keep the continuation only if the final submit_sql passes the deterministic train-task scorer.
  6. Turn each accepted teacher action into one hard-token SFT row and train a LoRA adapter from the current checkpoint.

No teacher probabilities or student-token logprobs are used, so this is DAgger-style SFT, not post 04's probability distillation.

From that one loop, the post walks a short progression. Each step changes only the data mix or the parent checkpoint it starts from:

The per-run mechanics, dataset counts, and archived negative variants are recorded in this folder's README and experiment scripts, so the rest of the post can read these results as research questions instead of re-listing every run.

Implementation Note: The Training-Code Boundary

This experiment is a clean example of where to draw the boundary between task-specific agent code and reusable training code. The SQL-agent side owns rollout collection, state selection, the GPT 5.5 medium correction call, BAML parsing, SQLite execution, deterministic verification, and turning accepted suffixes into rows. Those pieces depend on the environment and should stay close to the harness.

The reusable training side is simpler and method-generic: validate that the row source matches the method label, enforce assistant-action-only masking, audit token lengths and dropped rows, check LoRA settings, use a standard SFT trainer, write enough metadata to reproduce the checkpoint choice, and compare full eval against the fixed baseline. The rows come from student-created states, but the target is still a hard teacher action and the loss is still supervised fine-tuning, so the training path must not label this as token-level probability distillation.

Training Setup

The student remains unsloth/Qwen3.5-0.8B. The first correction run starts from the best top-k soft-label checkpoint, not the base model, because the initial question is whether student-state hard-token correction can improve the best student I already have. Later ablations start from the first hard-token correction model. The iterative repair test starts from the 64/220 repair-plus-retention parent, and the final specialist-success anti-forgetting run also starts from that 64/220 parent. The teacher for correction suffixes is GPT 5.5 medium.

The data split is unchanged: 879 train tasks and 220 held-out eval tasks from the same birdsql/six-gym-sqlite benchmark. For the rollout collection step, the student runs with the same agent harness, a maximum of 8 tool-use turns, an 8192-token input/history budget, and up to 512 generated tokens per action. That larger rollout/eval budget lets the agent carry longer histories at inference time. The first correction run used a 4096-token SFT filtering budget for the rendered training sequence because that matched the earlier LoRA recipe. Later retention runs either kept that filter or explicitly moved the SFT limit to 8192, which I call out where it matters.

Here is the core setup in one place:

| Setting | Value | | --- | --- | | Student | unsloth/Qwen3.5-0.8B | | Initialization | Best top-k soft-label checkpoint for the first correction run; hard-token correction adapters for later ablations; 64/220 repair-plus-retention parent for the final anti-forgetting run | | Teacher | GPT 5.5 medium | | Train / eval split | 879 train tasks / 220 held-out eval tasks | | Rollout and eval budget | 8192 input/history tokens, 512 generated tokens per action | | Max tool-use turns | 8 | | Initial correction SFT filter | 4096 rendered prompt-plus-target tokens | | Later retention SFT filter | 4096 or 8192, called out per ablation | | Training backend | CUDA LoRA SFT, bf16 when supported, no 4-bit loading | | LoRA modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj | | LoRA rank / alpha | 32 / 32 | | Batch / grad accumulation | 1 / 8 | | Initial correction learning rate | 5e-5 | | Retention repair learning rates | 2e-5 and 1e-5 | | Validation split | 5% | | Optimizer/scheduler | Trainer defaults, held fixed across comparable SFT runs |

The training recipe stayed close to the hard-token and soft-label runs before it: LoRA fine-tuning with rank 32 and alpha 32 on the Qwen projection and MLP modules, batch size 1, gradient accumulation 8, learning rate 5e-5, and a 5% validation split. The first correction dataset had 560 source rows. Under the 4096-token training budget, 508 rows were kept and 52 long rows were dropped; the final split was 483 train rows and 25 validation rows. The rendered training sequences had p50 2881 tokens, p90 3997, p95 4244, and max 7125 before filtering. Training ran for 183 optimizer steps, which was three epochs over the kept train rows. It took 5941 seconds on the RTX 3090, reached train loss 0.2835, and ended with validation loss 0.4151.

For the later hard-token SFT ablations, I kept the same model family, harness, action schema, eval split, LoRA target modules, batch size, gradient accumulation, and assistant-token-only SFT contract. The runs are not perfectly single-variable experiments, because some later tests intentionally change the data mix and the continuation setup together. Still, each run has one clear question: second-round correction data, aggregate correction data, wrong-submit repair rows, successful-retention rows, retention ratio, update budget, learning rate, adapter capacity, or one more gated repair-plus-retention iteration. This is why I treat the follow-up runs as ablations, not as new benchmark settings.

The retention repair family is still part of the student-state hard-token correction method family. It does not use teacher probabilities. The repair rows are verified hard-token GPT 5.5 medium continuations from student-created wrong-submit states. The retention rows are hard-token successful traces used to keep behavior the current student already had. For example, the 64/220 2:1 successful-retention run trained from the first correction adapter on 878 source rows, kept all 878 under an 8192-token SFT limit, used rank-32 LoRA with alpha 32, batch size 1, gradient accumulation 8, learning rate 2e-5, and one epoch of TRL SFTTrainer.

The final 67/220 checkpoint used the same hard-token SFT contract, but a different data mix. It started from the 64/220 parent and trained on 1260 rows: 1008 parent-success retention rows plus 252 specialist-success rows from earlier complementary students. The SFT sequence limit was 8192, no rows were dropped, the split was 1197 train rows and 63 validation rows, LoRA stayed rank 32 / alpha 32, batch size stayed 1 with gradient accumulation 8, and the learning rate was 5e-6 for one epoch. The final adapter scored 65/220, while checkpoint-100 scored 67/220 and is the checkpoint I treat as the best hard-token correction-family model.

One wording point is worth making explicit. Context length means the input/history budget available to the model during rollout or eval. Max new tokens means the generation budget for the next assistant action. The SFT max sequence length is different: it is the rendered training example length after combining the prompt/history and one target action. The first correction run filtered SFT rows at 4096 tokens; later retention ablations either kept that filter or explicitly moved the SFT limit to 8192.

In the repo, the algorithms above map to the rollout/eval harness, candidate selection, teacher-correction generation, retention-dataset builder, SFT training, and chart scripts. Reproduction commands live in the README; the blog teaches the method and the decisions.

Results As Research Questions

Did hard-token correction beat the top-k soft-label student?

Yes. The top-k soft-label student scored 55/220. After one round of GPT 5.5 medium hard-token correction from student-created states, the same unsloth/Qwen3.5-0.8B student scored 58/220. That first result was only a 3-task gain, so I would not oversell it.

The direct repair-plus-retention result is stronger: wrong-submit repair plus a 2:1 successful-retention mix reached 64/220 on the same fixed 220-task eval. That is a 9-task gain over the top-k soft-label starting point and a 6-task gain over the first correction model. The final anti-forgetting extension pushed the best checkpoint in this hard-token family to 67/220 by adding complementary specialist-success rows with heavy parent retention. It is still not a breakthrough, but it changed my read of the experiment: more correction alone was not enough, while correction plus explicit retention anchors was useful, and complementary successful trajectories could add a few more solved tasks when the parent was strongly anchored.

Student-state correction performance chart

The important comparison is not against the base model anymore. This experiment starts from the best top-k soft-label checkpoint, so the honest question is whether hard-token correction training improves that warm-started student. The answer is yes: first to 58/220, then 64/220 with repair plus retention, and finally 67/220 with specialist-success anti-forgetting. The cost is also clear: we had to run students over the full train split, verify candidate trajectories in the environment, train SFT adapters, evaluate checkpoints on the fixed 220-task split, and keep the method labels strict.

What changed inside the loop?

The failure decomposition is more interesting than the headline score. The first hard-token correction model submitted 180 times, up from 171 in the top-k soft-label checkpoint. Repeated-action stops went down from 39 to 33. Max-turn stops went from 3 to 0. Average turns also dropped from 3.39 to 3.20. That is the part of the first result I like: the correction data seems to teach the student to move through the loop more decisively.

Student-state correction failure decomposition chart

The negative side is that SQL/tool errors went up from 12 to 27 in that first correction run. So the model did not simply become better everywhere. It became more willing to act and submit, and some of that turned into solved tasks, but some of it turned into bad SQL.

The 64/220 retention model improved that balance but did not make it clean. It submitted 179 times, had 18 SQL/tool errors, 4 parse failures, 37 repeated-action failures, and no max-turn or runtime failures. Compared with the first correction model, it solved more tasks and made fewer SQL/tool and parse mistakes. Compared with the 61/220 1:1 successful-retention run, it solved three more tasks but had more repeated-action failures.

The 67/220 specialist-success checkpoint had a different shape again. It submitted 174 times, had 8 SQL/tool errors, 5 parse failures, 40 repeated-action failures, one max-turn failure, and no runtime failures. So the score improved, and SQL/tool errors dropped a lot compared with the 64/220 parent. But repeated-action failures rose slightly from 37 to 40. This is the basic tradeoff: better final score does not mean every loop category improved.

Which correction states became training data?

The correction dataset came mostly from wrong-submit states because many failed trajectories eventually submit an incorrect query. GPT solved 193 wrong-submit candidates and those produced 279 SFT rows. SQL-error states were also useful: 76 verified continuations produced 139 rows. Repeated-action states gave 62 verified continuations and 118 rows. Max-turn states were much smaller: 9 verified continuations and 18 rows. Parse-failure states were tiny, with 5 verified continuations and 6 rows.

This does not prove that each bucket caused a marginal eval gain. It only says which buckets produced verified rows that entered the first correction run. Still, the distribution matters because the training signal was not balanced by failure type. The first correction adapter mostly learned from wrong-submit and SQL-error recovery, with some repeated-action correction. The later retention experiments did not change the benchmark or scorer; they changed the hard-token SFT mix to test whether repair rows needed successful behavior anchored beside them.

Was one correction round enough?

One round helped, but it was not enough to change the character of the model. The score moved from 55 to 58, and the loop behavior improved, but wrong submissions still dominated the failures. So I tried the natural next question: if the corrected student rolls out again, can GPT correct the new states and keep improving it?

Correction-only training was not enough. After the first correction model reached 58/220, I ran it over the full 879-task train split again. It solved 254/879 train tasks and produced 619 candidate failure states: 481 wrong-submit states, 131 repeated-action states, and 7 parse-failure states. GPT 5.5 medium verified 241 of those states and produced 391 second-round SFT rows. Training on only those rows reached 56/220. Aggregating round 1 and round 2 tied the first correction model at 58/220, but did not beat it. Starting again from the top-k soft-label checkpoint with the same aggregate data fell to 47/220. Extra correction rows were real, but they were not a clean upgrade signal by themselves.

The next useful turn was to treat the problem as forgetting repair, not just more correction. Wrong-submit repair alone reached 54/220. Targeted replay for tasks the top-k soft-label model solved and the first correction model forgot also reached 54/220. But mixing verified wrong-submit repair rows with successful-retention rows from the first correction model finally moved the held-out result: 1:1 successful retention reached 61/220, and a 2:1 successful-retention mix reached 64/220.

This belongs in the hard-token correction family because the training objective is still SFT on hard targets. The teacher never provides a probability distribution over student tokens. There is no KL-style probability update here; that is the post 04 comparison. The repair rows are accepted GPT 5.5 medium action text from verified continuations, and the retention rows are accepted hard-token traces that preserve behavior the student already had.

The retention curve was not monotonic. Half of the 1:1 retention update reached 59/220. A larger effective batch on the same 1:1 data fell to 54/220. Lowering the learning rate on the 1:1 data tied the first correction model at 58/220. A 1.5:1 retention mix fell to 52/220, while 2:1 retention became the best direct repair-plus-retention run at 64/220. A stepwise learning-rate continuation from the half-budget 1:1 checkpoint reached 60/220: clean repeated-action behavior, but worse SQL/tool and parse behavior than the best run. Merging the 64/220 adapter into the base and training a fresh rank-64 LoRA on the same 2:1 retention data reached 63/220, close but not better. Repeating the 2:1 repair-plus-retention loop from the 64/220 parent also reached 63/220, so the gated chain stopped after one iteration.

The specialist-success anti-forgetting run was the final step in this story, not a separate method bolted on later. It did not ask GPT 5.5 medium for another repair batch. Instead, it took successful train trajectories from complementary earlier students on parent-missed tasks and mixed them with four copies of parent-success retention. That dataset had 1260 rows: 1008 parent-retention rows and 252 specialist-success rows. Training from the 64/220 parent with a smaller 5e-6 learning rate produced a final 65/220 adapter, but checkpoint-100 reached 67/220. That makes 67/220 the best hard-token correction-family checkpoint before post 04's probability-distillation experiments.

The earlier ablations still matter as negative evidence. Broad replay and non-submit replay hurt. Critical-state filtering made the model too eager to submit and reached 50/220. Earlier teacher intervention was mechanically clean: GPT took over after the first environment observation, verified 111 of 300 candidates, and produced 178 SFT rows. But the trained model became fast and confidently wrong, reaching 47/220. Adapter soups sometimes reduced one failure mode while creating another. The broader pattern is not "more rows always help"; it is "the data mix and update path decide whether correction becomes learning or forgetting."

There was one useful extra-compute clue from the first correction checkpoint. The top-k soft-label checkpoint and the first hard-token correction model solved different tasks: 38 overlap, 17 solved only by the soft-label checkpoint, and 20 solved only by the first correction model. A simple label-free selector over those two rollouts, using only trace signals like submitted-vs-not, SQL/tool errors, repeated-action stop, parse failure, max-turn stop, and turn count, reached 63/220. But that did not scale cleanly. Across broader sets of weaker runs, the oracle union rose as high as 97/220, while the same simple selector fell back to 55-56/220. So I treat selector/oracle analysis as a research clue, not as a headline model result.

The 64/220 and 67/220 checkpoints also show why the final gain is not just three extra tasks. They solve overlapping but not identical eval tasks: 53 tasks are solved by both, 11 are solved only by the 64/220 parent, 14 are solved only by the 67/220 checkpoint, and 142 are solved by neither. The oracle union is 78/220. I do not use eval complementarity to build training rows, but it is useful for interpretation: the anti-forgetting update added more new wins than it forgot, even though it still moved the agent into a slightly different failure shape.

Second-round student-state correction ablation chart

My read is that the first student-state correction round fixed an obvious behavior problem: the student was getting stuck or repeating itself too often. The second round was harder. It mostly saw wrong-submit states from a model that already had better loop control. GPT could rescue 241/619 of those states, but imitating those rescues alone did not make the held-out rollout better. The later retention results changed the answer: one correction round was not enough, but the useful second step was repair plus retention, not correction-only SFT.

Key numbers behind the charts

| Model / stage | Success | Submitted | SQL/tool errors | Repeated-action stops | Runtime errors | | --- | ---: | ---: | ---: | ---: | ---: | | Best top-k soft-label student | 55/220 | 171 | 12 | 39 | 0 | | First student-state hard-token correction | 58/220 | 180 | 27 | 33 | 0 | | Round-2 correction rows only | 56/220 | 174 | 17 | 38 | 0 | | Aggregate first- and second-round correction rows | 58/220 | 175 | 25 | 37 | 1 | | Wrong-submit repair only | 54/220 | 179 | 26 | 35 | 0 | | Wrong-submit repair + 1:1 successful-retention mix | 61/220 | 180 | 17 | 30 | 0 | | Stepwise learning-rate continuation from half-budget 1:1 retention | 60/220 | 183 | 24 | 28 | 0 | | Wrong-submit repair + 2:1 successful-retention mix | 64/220 | 179 | 18 | 37 | 0 | | Merge best into base, fresh rank-64 LoRA on 2:1 retention | 63/220 | 184 | 23 | 31 | 0 | | Iterative wrong-submit repair + 2:1 retention from 64/220 parent | 63/220 | 179 | 14 | 36 | 0 | | Specialist success + 4:1 parent-success retention, checkpoint-100 | 67/220 | 174 | 8 | 40 | 0 | | Targeted replay variants | 54/220 | 185-187 | 28-36 | 27-31 | 0 | | Early intervention | 47/220 | 212 | 62 | 1 | 0 |

This is the reading path. The full per-run ledger, including the archived replay, adapter-interpolation, and selector variants, lives in this folder's README so the post stays focused on the method rather than the audit trail.

A short note on test-time ensembling: the best top-k soft-label checkpoint and the first correction checkpoint have an oracle union of 75/220, and a label-free trace-signal selector over their rollouts reaches 63/220. Across many weaker runs the oracle union rises as high as 97/220 while the selector falls back to 55-56/220, so I treat selector and oracle analysis as a research clue, not a headline result.

Failure Analysis

The main failure after hard-token correction is still wrong submitted SQL. The final 67/220 checkpoint submitted 174 times and solved 67, which means 107 submitted attempts still failed. That is better than the first correction model's 122 unsuccessful submissions and the 64/220 parent's 115 unsuccessful submissions, but it is still the dominant failure mode. The student learned to control parts of the tool loop better, and retention repair preserved more useful behavior, but it still often ended with the wrong final query.

Repeated-action behavior is mixed. The top-k soft-label checkpoint had 39 repeated-action failures. The first correction model improved that to 33. The 1:1 retention repair run improved further to 30, and the stepwise learning-rate continuation reached 28. But the 64/220 parent rose back to 37 repeated-action failures, and the final 67/220 checkpoint had 40. That is why I do not read the best score as a clean loop-control fix. It is the best held-out score, not the cleanest trajectory shape.

SQL/tool errors tell the other side. They jumped from 12 in the top-k soft-label checkpoint to 27 in the first correction model, then came down to 18 in the 64/220 retention model and 8 in the final 67/220 checkpoint. My interpretation is that the first correction model became more action-oriented, but not always more careful. Retention and specialist-success consolidation reduced SQL/tool errors, but they did not solve SQL judgment. This is why I do not want to judge agent training only by loss, submission rate, or one failure category. The rollout decomposition tells the real story.

The later ablations made that clearer. The failure was not simply "we need more corrected rows." Correction-only rows, aggregate rows, replay rows, retention-ratio changes, earlier intervention, learning-rate changes, higher adapter capacity, adapter mixing, and one more gated repair-plus-retention iteration all changed the rollout distribution. Some variants lowered one failure category while making another worse. Early intervention almost eliminated repeated-action stops, but it made the model submit quickly and wrongly. The rank-64 continuation reduced repeated-action failures compared with the 64/220 model, but it made more SQL/tool errors and lost one solved task overall. The extra gated iteration reduced SQL/tool errors to 14, but it still lost one solved task and kept repeated-action failures high at 36. The broad ensemble analysis showed that failed runs contain some unique wins, but visible trace signals were not reliable enough to select those wins across many runs. That is the uncomfortable pattern: the student has pockets of recoverable behavior, but naive extra training moves the agent into new bad states instead of cleanly adding capability.

Hardware And Infrastructure Lessons

This post looked like it should keep paid GPU time low, but the full workflow still needed the GPU in three places: rolling out the current student on all 879 train tasks, training the corrected student, and running the full 220-task eval. The expensive teacher-correction phase was mostly API-bound because GPT 5.5 medium was the teacher. Since there was no probability-exporting teacher in this post, the Mac did not need to load a large local teacher just to prepare soft labels.

The practical workflow was split: use the GPU for local-model rollout and training work, use GPT 5.5 medium to generate verified correction suffixes, then return to the GPU for SFT and evaluation. The train rollout was the slow first stage because every task is an agent episode, not a single forward pass. The LoRA training itself took 5941 seconds on the RTX 3090 for 183 optimizer steps. The full 220 eval was also slow because each eval item can include multiple model/tool turns.

The main infrastructure lesson is that agent experiments are not priced like plain SFT. Rollout and eval can dominate because each task may require several model calls plus SQLite execution. Training is only one piece of the cost. Student-state correction adds another layer: you first pay for the student to create states, then pay the teacher to recover from some of those states, then pay again to train and evaluate the new student.

What I Learned

First, student-state hard-token correction can help, but the first gain was modest. Moving from 55/220 to 58/220 matters because the student was already warm-started from the best top-k soft-label model, but it is not a breakthrough. The cleanest initial improvement was loop behavior: fewer repeated-action stops, no max-turn stops, and slightly fewer turns on average.

Second, verifier filtering is the difference between useful correction data and wishful thinking. GPT 5.5 medium solved 345 out of 807 selected states, which means more than half of the attempted corrections were discarded. I like that. It means the dataset is not "GPT said something plausible." It is "GPT continued from this student-created state and passed the deterministic train-task scorer."

Third, retention mattered more than correction-only repetition. Wrong-submit repair alone reached 54/220, but repair plus a 1:1 successful-retention mix reached 61/220 and repair plus a 2:1 successful-retention mix reached 64/220. Repeating that 2:1 recipe from the stronger 64/220 parent reached 63/220, so the gain did not compound automatically. That makes the method lesson sharper: the student needs corrected actions from bad states, but it also needs enough successful behavior anchored so the update does not forget what already worked.

Fourth, the final improvement came from retention-heavy specialist success, not another correction batch. The 67/220 checkpoint mixed a small amount of complementary successful behavior with a lot of parent-success retention. That makes it part of the same hard-token SFT family, but it changes the emphasis: after correction teaches recovery, anti-forgetting consolidation decides whether the new behavior survives.

Fifth, better recovery can expose weaker reasoning. The first corrected model submitted more often and repeated less, but it also produced more SQL/tool errors. The follow-up experiments made the same point from different angles: more rows, gentler learning rates, earlier teacher intervention, higher adapter capacity, and adapter mixing did not cleanly add capability. For this SQL agent, preserving loop control while improving final SQL judgment is the hard part.

Conclusion

This experiment answered the narrow question I cared about. Yes, DAgger-style hard-token expert correction from student-created states can improve the best 0.8B SQL agent from the top-k soft-label experiment. The first improvement was small: 55/220 to 58/220. The stronger result came from the same method family plus explicit forgetting repair: wrong-submit repair with a 2:1 successful-retention mix reached 64/220 on the fixed 220-task eval.

The final best result in this post is 67/220. It came from the same hard-token SFT family: start from the 64/220 parent, add complementary train-split successes from earlier students only where the parent failed, mix those rows with four copies of parent-success retention, and select the best checkpoint after full eval. The final adapter itself scored 65/220, but checkpoint-100 reached 67/220, so checkpoint selection mattered more than validation loss alone.

That ending makes the lesson cleaner. Correction-only training did not keep improving the model. Round-2 correction rows alone reached 56/220, aggregate first- and second-round correction rows tied 58/220, wrong-submit repair alone reached 54/220, and repeating the best 2:1 repair-plus-retention loop from the 64/220 parent reached 63/220. What worked was correction plus retention, then retention-heavy consolidation of complementary successful behavior.

Off-policy distillation gave the student a strong starting behavior. Soft-label distillation improved it further. DAgger-style expert correction then added recovery ability from states the student actually visits, and anti-forgetting SFT helped preserve behavior that correction-only SFT was prone to losing. Post 04's token-level probability distillation is a different method, so I report it separately from this hard-token correction family.