Small-Model Distillation — Part 1: Offline Teacher-Trace SFT for a 0.8B SQL Agent

TL;DR

I wanted to see whether a very small language model could learn to act inside a real SQL tool-use loop. The model was not asked to only write a final answer. It had to inspect a SQLite database, run SQL probes, read tool observations, and submit corrected SQL that passed hidden deterministic tests.

The first method in this series is the simplest useful one: offline teacher-trace hard-token SFT. A stronger teacher runs the fixed harness first. I keep only successful train trajectories. Then each teacher decision becomes a supervised fine-tuning (SFT) row of the form conversation so far -> next structured decision.

The result was useful but also humbling. The main 0.8B student improved a lot, but it stayed far below the stronger teachers and baselines.

| Role | Result | | --- | --- | | Main 0.8B student | unsloth/Qwen3.5-0.8B moved from 1/220 to 44/220 with GPT 5.5 medium rows, and to 46/220 with Qwen3.5-35B-A3B 8-bit rows. | | Comparison students | unsloth/Qwen3.5-2B reached 58/220 at best; LiquidAI/LFM2.5-8B-A1B reached 50/220 at best. | | Strong baselines | Qwen3.5-35B-A3B 8-bit solved 96/220, GPT 5.4 mini solved 105/220, and GPT 5.5 medium solved 115/220. |

The answer is not "the small model became as good as the teacher." It did not. The better answer is: hard-token trajectory SFT transferred the agent protocol, but not teacher-level SQL judgment. The students learned to operate the harness and submit much more often, but most of the remaining failures moved downstream into wrong SQL submissions and loop-control mistakes.

The Method Contract

The reusable idea in this post is a hard-token SFT contract. The SQL-agent code owns the benchmark-specific work: running the teacher in the harness, keeping successful traces, and turning each assistant action into a messages -> assistant target row. The training flow checks that the data and method still match the claim: this is supervised fine-tuning on teacher-written hard targets, with assistant-only labels and LoRA adapters.

Conceptually, the preflight looks like this:

contract = load_experiment_contract("post1_hard_token_sft")
rows = load_jsonl(contract.train_rows)
audit = summarize_rows(rows)
assert audit.row_count > 0
assert audit.max_sequence_tokens <= contract.max_length
assert all(row.method == "hard_token_sft" for row in rows)
training_examples = []
for row in rows:
    example = tokenize_with_chat_template(row.messages)
    example.labels = mask_everything_before_assistant_target(example)
    training_examples.append(example)
adapter = train_lora_sft(
    model=contract.student_model,
    examples=training_examples,
    lora=contract.lora_config,
    optimizer=contract.optimizer_config,
)

That is the important shape: define the method, audit the rows, mask the prompt/context tokens, train the adapter, and evaluate the result back in the unchanged SQL-agent harness. The exact operational commands for this use case live in the README, because those include machine setup, model paths, and artifact locations.

What I Wanted To Test

I am interested in small language models that become good at a narrow, useful task. Not a general chat model. Not a leaderboard toy. A small model that can sit inside a real workflow and do one thing well enough to matter.

For this first post, I started with a SQL repair agent. Each task gives the model a user issue, a buggy SQL query, and a SQLite database. The model can inspect the schema, run SQL queries, read observations, and eventually submit corrected SQL. The final score is deterministic: the submitted SQL either passes hidden tests or it does not.

The question was:

Can a tiny model learn the behavior of a stronger SQL tool-use agent from saved teacher trajectories?

That wording matters. I was not only asking whether the model can write SQL-looking text. I wanted the model to learn the loop: choose an action, respect the schema, read the observation, decide what to do next, and stop when it has enough evidence.

I kept the benchmark, action schema, parser, stop rules, and eval split fixed. Small-agent results are easy to distort by changing the harness, so I did not tune the environment per model.

The SQL-Agent Problem

Here is the kind of task the model sees:

Database: chinook

User issue: I want to find the latest track_id and use that id to filter records in the track table.

Buggy SQL:

WITH vars AS (SELECT COUNT(*) AS vars_id FROM track)
SELECT * FROM track WHERE track_id = vars_id

The bug is small but meaningful. COUNT(*) is not the latest id. The intended repair is closer to MAX(track_id). The buggy SQL gives useful clues about tables and columns, but it can also anchor the model to the wrong operation.

That gives the model two jobs:

• Operate the agent protocol correctly.
• Choose the right SQL repair.

A weak model can fail before doing SQL reasoning at all. It can inspect schema, inspect schema again, emit invalid JSON, or repeat an unproductive action until the harness stops it. A better model can drive the harness but still submit SQL that fails the hidden tests. Those are different failure modes, and I wanted the experiment to separate them.

The benchmark source was birdsql/six-gym-sqlite. I used only Query tasks from four SQLite databases: netflix, movie_3, books, and chinook.

Concretely, the benchmark filter was category == "Query" and db_id in {"netflix", "movie_3", "books", "chinook"}. I chose this fixed slice to keep the first experiment narrow while preserving several domains, then split inside each database so train and eval kept the same domain mix.

| Setting | Value | | --- | --- | | Task category | Query | | Databases | netflix, movie_3, books, chinook | | Source rows scanned | 5000 | | Candidate rows after filtering | 1099 | | Train split | 879 tasks | | Eval split | 220 tasks | | Split seed | 42 |

The split by database was:

| Database | Candidate tasks | Train tasks | Eval tasks | | --- | ---: | ---: | ---: | | books | 282 | 226 | 56 | | chinook | 251 | 201 | 50 | | movie_3 | 273 | 218 | 55 | | netflix | 293 | 234 | 59 | | Total | 1099 | 879 | 220 |

The model can choose only three structured actions:

{"action": "inspect_schema"}
{"action": "run_sql_query", "sql": "SELECT ..."}
{"action": "submit_sql", "sql": ["SQL statement 1", "SQL statement 2"]}

BAML, or Boundary Markup Language, is the structured-output layer around the prompt contract. It defines the action schema and renders the model request. Hosted teacher and baseline calls use the BAML structured parsing path; local HF/PEFT student eval decodes model text and the harness validates it with parse_decision. In both paths, the accepted target is canonical decision JSON, not free-form teacher prose.

The loop is simple:

1. Build messages from the task.
2. Ask the model for one structured action.
3. Parse and validate the action.
4. Execute the action in SQLite when needed.
5. Append the observation.
6. Repeat until submit, parse failure, repeated action, max turns, or runtime failure.

The stop categories are part of the result, not just logging noise.

| Stop reason | What it means | | --- | --- | | submitted | The model submitted final SQL; the SQL either passed or failed hidden tests | | parse_failure | The model did not produce a valid structured action | | repeated_action | The model repeated an action that the harness considered unproductive | | max_turns | The model kept acting but never reached a valid final submission | | runtime_error | The model call or harness call failed during the task |

For this task, formatting is not cosmetic. A human can understand "I should inspect the schema first," but the harness needs {"action":"inspect_schema"}. I did not use keyword matching or an LLM judge to rescue malformed actions. The environment expects structured actions, executes them, and scores the final SQL against hidden tests.

The Training Idea

This post is about post-training, not pretraining. Pretraining is the broad next-token stage that gives a model language, code, and world-pattern priors. Post-training is where we shape behavior for a use case: instruction tuning, supervised fine-tuning, preference optimization, reinforcement learning, safety tuning, tool-use training, or domain adaptation.

Knowledge distillation means using a stronger teacher model to transfer useful behavior into a smaller student model. For agents, the important question is not only "which teacher?" It is also:

Which part of the teacher behavior becomes supervision?

There are several possible signals.

| Distillation signal | What the student learns from | Why it matters | | --- | --- | --- | | Hard labels | The teacher's chosen output tokens | Simple SFT: given this input, produce this exact target. | | Soft labels / logits | The teacher's probability distribution over next tokens | Preserves uncertainty over alternatives instead of only the winning token. | | Feature distillation | Internal teacher activations or hidden states | Tries to align representations, but needs direct teacher internals. | | Final-answer distillation | Only the completed teacher answer | Useful for answer-only tasks, weak for agents because it discards the process. | | Trajectory distillation | Intermediate actions, observations, and final answer | Teaches the policy: inspect, query, react, and submit. | | Reward or RL-style methods | A scalar outcome after rollout | Optimizes behavior through environment feedback instead of direct imitation. |

This first post uses offline hard-token trajectory distillation. Offline means the teacher runs first and training happens later from saved traces. Hard-token means the student trains on the exact tokens the teacher chose, not the teacher's probability distribution. Trajectory means the training rows come from intermediate tool-use decisions, not only final SQL.

So the unit of learning is:

Full conversation state before the teacher decision -> canonical next executable decision.

This is still a local imitation objective. It does not directly optimize task success under the student's own rollouts. That limitation becomes important in the results.

A Small Educational Code Walkthrough

The core training object is easier to understand with small code-shaped pieces than with a long command line. This is not meant to be a drop-in script. It is the mental model of the pipeline: how the harness state becomes rows, how rows become masked SFT examples, how LoRA training uses them, and how the adapter is evaluated again in the same loop.

A task starts as a conversation state:

# Simplified task state before the first model call.
state = {
    "system": "You are a SQL repair agent. Return one structured action.",
    "database": "chinook",
    "user_issue": "Find the latest track_id...",
    "buggy_sql": "WITH vars AS (...) SELECT ...",
    "history": [],
}

The model is not allowed to write arbitrary prose. BAML wraps the response in a structured decision with a short draft and exactly one executable action. The target that reaches SFT is the canonical JSON form of that decision.

# Three possible decisions from one successful teacher trace.
decision = {
    "draft": "Need schema first.",
    "output": {"action": "inspect_schema"},
}

decision = {
    "draft": "Check the latest id.",
    "output": {
        "action": "run_sql_query",
        "sql": "SELECT MAX(track_id) FROM track",
    },
}

decision = {
    "draft": "Use max track id.",
    "output": {
        "action": "submit_sql",
        "sql": ["SELECT * FROM track WHERE track_id = (SELECT MAX(track_id) FROM track)"],
    },
}

The harness calls the model once per turn, executes the action, and appends the environment observation back into the conversation. When the teacher run succeeds, each turn can become one training row. The important part is that each row uses the conversation state before the action, not the full solved task as one flat answer.

# Each successful teacher decision becomes one SFT row.
rows = []
state = initial_state(task)

for teacher_decision, observation in successful_teacher_trace:
    rows.append({
        "messages": messages_before + [
            {"role": "assistant", "content": teacher_decision.raw_text}
        ],
        "teacher_draft": teacher_decision.draft,
        "teacher_action": teacher_decision.output,
    })
    state = append_decision_and_observation(state, teacher_decision, observation)

Before training, each row is rendered with the same BAML prompt shape the harness uses at eval time, then filtered by the full rendered sequence length. The full sequence includes the prompt/history and the target decision, so a row can be dropped even when the target JSON itself is short.

# Keep only rows that fit the training sequence budget.
kept_rows = []

for row in rows:
    canonical = canonical_sft_row(row)
    rendered = chat_template(
        canonical.messages,
        add_generation_prompt=False,
        enable_thinking=False,
    )

    if token_count(rendered) <= 4096:
        kept_rows.append(canonical)

train_rows, valid_rows = split_train_validation(kept_rows, validation_fraction=0.05)

Training and eval used the same prompt shape but different budgets. SFT rows were capped at 4096 tokens for the full prompt-plus-target sequence. Local student rollout eval used an 8192-token total sequence cap, with up to 512 tokens reserved for generation on each model call. For Qwen chat templates, I set enable_thinking=False in filtering, tokenization, and local eval so hidden thinking scaffolding was not part of the SFT/eval format.

During SFT, the student should learn only the assistant action target. It should not learn to predict the system prompt, task text, schema observations, or SQL query results. Those tokens are context, not labels, so prompt tokens become -100, the ignore index used by the loss.

# Mask prompt/context tokens; train only on the assistant decision.
prompt = chat_template(
    messages[:-1],
    add_generation_prompt=True,
    enable_thinking=False,
)
full = chat_template(
    messages,
    add_generation_prompt=False,
    enable_thinking=False,
)

prompt_ids = tokenize(prompt)
full_ids = tokenize(full)
labels = [-100] * len(prompt_ids) + full_ids[len(prompt_ids):]

loss = cross_entropy(
    student_logits[:, :-1, :],
    labels[:, 1:],
)

That is hard-token distillation. The teacher says, "this was the action I took," and the student is trained to put probability mass on that exact target sequence.

# Hard-token distillation is ordinary cross-entropy on teacher tokens.
hard_token_loss = cross_entropy(
    logits_at_assistant_positions,
    teacher_decision_token_ids,
)

The training part is normal LoRA SFT. LoRA means low-rank adaptation: instead of updating all model weights, I attach small trainable matrices to selected modules and train only those adapter parameters. The base model provides the capability; the adapter learns the SQL-agent behavior.

# Load the student and attach trainable LoRA adapters.
tokenizer = load_tokenizer(student_model)

train_examples = [
    tokenize_sft_row(tokenizer, row, max_length=4096)
    for row in train_rows
]

valid_examples = [
    tokenize_sft_row(tokenizer, row, max_length=4096)
    for row in valid_rows
]

student = load_base_model(student_model, max_seq_length=4096)

student = add_lora_adapters(
    student,
    rank=32,
    alpha=32,
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ],
)

Then the trainer sees already-tokenized examples with input_ids, attention_mask, and masked labels. There is no packing, because each row is one agent decision state and I wanted to keep that boundary clear.

# Train for a fixed number of comparable optimizer steps.
steps_per_epoch = ceil(len(train_examples) / (batch_size * grad_accum))
max_steps = 3 * steps_per_epoch

trainer = SFTTrainer(
    model=student,
    train_dataset=train_examples,
    eval_dataset=valid_examples,
    data_collator=DataCollatorForSeq2Seq(
        tokenizer,
        label_pad_token_id=-100,
    ),
    args=SFTConfig(
        max_length=4096,
        max_steps=max_steps,
        per_device_train_batch_size=1,
        gradient_accumulation_steps=8,
        learning_rate=5e-5,
        bf16=True,
        packing=False,
        dataset_kwargs={"skip_prepare_dataset": True},
    ),
)

trainer.train()
adapter = save_lora_adapter(student)

The batch math is also part of the training story. Batch size 1 means one SFT row per microbatch. Gradient accumulation 8 means the optimizer updates after eight rows. Three epochs over about 990 GPT-teacher train rows gives 372 optimizer updates; three epochs over about 1151 Qwen-teacher train rows gives 432 updates for the Qwen students. That is why the Qwen-teacher runs took longer even before model-size differences.

Evaluation is not teacher forcing. The trained adapter goes back inside the same SQL-agent harness and has to act on the 220 held-out tasks. On each turn, the harness renders the current state, the model generates one decision, the harness parses and validates it, SQLite executes it when needed, and the next observation is appended.

# Evaluation is a real rollout through the SQL-agent harness.
def generate_action(messages):
    rendered = render_baml_sql_agent_messages(messages)
    prompt = chat_template(rendered, add_generation_prompt=True)

    if token_count(prompt) + 512 > 8192:
        raise RuntimeError("no room left for this turn")

    new_tokens = model_with_adapter.generate(
        prompt,
        max_new_tokens=512,
        temperature=0.0,
    )

    return decode(new_tokens)

results = []

for task in held_out_eval_tasks:
    result = run_sql_agent_task(
        task,
        generate=generate_action,
        max_turns=8,
        timeout_seconds=180,
    )
    results.append(result)

summary = summarize_results(results)

That last snippet is the most important difference between a training loss and an agent result. During training, the model predicts teacher decisions under teacher-style contexts. During eval, it sees its own previous actions and its own observations. If it makes one bad probe, the next state may be unlike anything in the teacher data.

This is why hard-token trajectory distillation is both useful and limited. It is simple, and it teaches the action protocol. But it throws away teacher uncertainty and does not directly optimize recovery under the student's own rollout. That is why the second post in the series moves to soft-label distillation.

Dataset And Filtering

The teacher and student live inside the same harness. The teacher is not writing synthetic explanations for a detached dataset. It is using the actual SQL environment the student will later be evaluated in.

The data-generation flow was:

1. Split train and eval tasks.
2. Run the teacher on train tasks.
3. Keep only successful trajectories.
4. Turn each successful teacher decision into an SFT row.
5. Canonicalize the assistant decision target.
6. Render the full training sequence.
7. Filter rows that do not fit the training context budget.
8. Split kept rows into train and validation.

I kept only successful trajectories. That is conservative, but it keeps the dataset's meaning clean. A failed trajectory can contain locally reasonable actions followed by a wrong final submit. If I kept those earlier actions, I would be claiming I know which pieces of a failed plan deserve credit. For the first experiment, I avoided that ambiguity.

Both teacher-data rounds used the same 879 train tasks. In the next table, success, submitted count, parsed actions, and average turns are measured over all 879 attempted train tasks. Source SFT rows count only assistant decisions from successful trajectories, so they are not simply success multiplied by average turns.

| Teacher source | Success | Submitted | Other stops | Parsed actions | Avg turns / attempted task | Source SFT rows | | --- | ---: | ---: | --- | ---: | ---: | ---: | | GPT 5.5 medium | 446/879 = 50.7% | 879 | 0 parse / 0 repeat / 0 max-runtime | 2262 | 2.57 | 1046 | | Qwen3.5-35B-A3B 8-bit | 394/879 = 44.8% | 816 | 0 parse / 32 repeat / 31 max-runtime | 3064 | 3.49 | 1232 |

The SFT row count is larger than the number of successful tasks because the unit of supervision is one decision point inside a successful trajectory.

The length filter was based on the full rendered training sequence, not only the output length. A row includes the system message, user task, buggy SQL, prior assistant decisions, tool observations, and the target assistant decision. The target decision is usually short. The context can be long because schema text and query observations accumulate across turns.

| Dataset / training path | Source rows | Kept at 4096 | Dropped | Final train / validation rows | In-loop validation | Token length min / P50 / P90 / P95 percentiles | | --- | ---: | ---: | ---: | ---: | --- | ---: | | GPT 5.5 rows, Qwen SFT | 1046 | 1042 | 4 | 990 / 52 | enabled | 605 / 1786 / 2948 / 3208 | | Qwen3.5-35B-A3B 8-bit rows, Qwen SFT | 1232 | 1211 | 21 | 1151 / 60 | enabled | 604 / 2014 / 3180 / 3531 | | Qwen3.5-35B-A3B 8-bit rows, LFM SFT | 1232 | 1226 | 6 | 1165 / 61 | disabled for LFM | 569 / 1907 / 2984 / 3335 |

The longest source rows before filtering were 15836 tokens for the GPT-row Qwen path, 5908 for the Qwen-row Qwen path, and 5297 for the Qwen-row LFM path. The longer rows were not "bad examples." They just did not fit the training infrastructure.

The LFM train/validation split was still written, but trainer validation was disabled because validation compilation was unreliable. The 220-task harness eval remained the comparison point.

I also estimated how much of each SFT row was prompt/history versus assistant target.

| SFT row token estimate | Teacher rows | Mean | P50 | P90 | P95 | | --- | --- | ---: | ---: | ---: | ---: | | Prompt/history before target | GPT 5.5 | 1339 | 1345 | 2367 | 2552 | | Target action JSON | GPT 5.5 | 90 | 69 | 192 | 259 | | Prompt plus target | GPT 5.5 | 1431 | 1447 | 2516 | 2745 | | Prompt/history before target | Qwen3.5-35B-A3B 8-bit | 1558 | 1556 | 2578 | 2862 | | Target action JSON | Qwen3.5-35B-A3B 8-bit | 83 | 71 | 150 | 191 | | Prompt plus target | Qwen3.5-35B-A3B 8-bit | 1643 | 1663 | 2693 | 3013 |

This is the shape I expected for agents. The model reads a relatively large state and emits a compact action.

Training Setup

All benchmark results used the same 220-task held-out eval split and the same SQL-agent harness. Teacher traces were generated only on train tasks. I did not train on hidden tests, reference SQL, failed teacher trajectories, or teacher free-form prose.

| Piece | Setup | | --- | --- | | Local workbench | Mac with 128 GB unified memory for dataset prep, notebooks, MLX serving/eval, GPT evals, charts, and blog artifacts | | Training machine | Rented NVIDIA RTX 3090 CUDA server for LoRA SFT and direct student evals | | First teacher | GPT 5.5 medium | | Same-family teacher | mlx-community/Qwen3.5-35B-A3B-8bit | | Hosted smaller baseline | GPT 5.4 mini | | Students | unsloth/Qwen3.5-0.8B, unsloth/Qwen3.5-2B, LiquidAI/LFM2.5-8B-A1B |

LoRA means low-rank adaptation. Instead of updating every weight in the model, I trained small adapter matrices on selected attention and MLP modules. That made it practical to run several student experiments quickly on the GPU machine.

| Setting | Value | | --- | --- | | Backend | CUDA / Unsloth-style SFT | | Epochs | 3 | | Optimizer updates | 372 for GPT rows; 432 for Qwen-row Qwen students; 438 for Qwen-row LFM | | Batch / grad accumulation / effective batch | 1 / 8 / 8 | | Learning rate / seed | 5e-5 / 42 | | LoRA rank / alpha | 32 / 32 | | Precision | bf16, 16-bit LoRA, not 4-bit | | Qwen target modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj | | LFM target modules | in_proj, out_proj, q_proj, k_proj, v_proj, w1, w2, w3 |

| Student | Trainable LoRA parameters | | --- | ---: | | Qwen3.5-0.8B | 12.78M of 865.77M, about 1.48% | | Qwen3.5-2B | 21.82M of 2.24B, about 0.98% | | LFM2.5-8B-A1B | 11.40M of 8.48B, about 0.13% |

I used this as a boring first recipe, not a hyperparameter search. Three epochs, learning rate 5e-5, rank/alpha 32, seed 42, and bfloat16 LoRA kept the runs comparable. I avoided 4-bit training here because the models fit with 16-bit LoRA on the GPU path, and I wanted the first result to be less entangled with quantization.

The local student eval cap includes both the input/history tokens and the generated action tokens. In practice, the system prompt, user task, prior actions, schema text, query results, structured-output instructions, and reserved output budget all had to fit inside the sequence limit. Max new tokens is still shown separately because it controls the maximum size of one assistant action.

| Run family | Context / sequence cap | Output budget per turn | Max turns | Timeout | Temperature | | --- | ---: | ---: | ---: | ---: | ---: | | CUDA local Hugging Face PEFT students and bases | 8192-token total sequence cap | 512 new tokens | 8 | 180s/task | 0.0 | | GPT 5.5 medium teacher eval | model/provider context | 1024 new tokens | 8 | 180s/task | 0.0 | | GPT 5.4 mini hosted eval | model/provider context | 2048 new tokens | 8 | 180s/task | 0.0 | | Qwen3.5-35B-A3B 8-bit MLX eval | MLX model context | 2048 new tokens | 8 | 180s/task | 0.0 | | LiquidAI/LFM2.5-8B-A1B-MLX-8bit base eval | MLX model context | 2048 new tokens | 8 | 180s/task | 0.0 |

Results As Research Questions

I find the results easier to read as questions. The main question is about the 0.8B student, but I also include Qwen3.5-2B and LiquidAI/LFM2.5-8B-A1B as comparison students. They help answer whether the behavior is a tiny-model accident or a broader effect of the training setup.

The charts use the same 220 held-out tasks. A base row means the model ran in the harness without a LoRA adapter. An SFT row means the base model plus the trained adapter ran in the same harness. Success means the final submitted SQL passed the hidden tests.

First: how good are the strong baselines on the same harness?

That gives the scale. Qwen3.5-35B-A3B 8-bit solved 96/220, GPT 5.4 mini solved 105/220, and GPT 5.5 medium solved 115/220. The GPT baselines submitted on every task and had no parse, repeat, max-turn, or runtime stops. Their failures were almost all wrong SQL submissions, which is a healthier failure shape than failing to operate the loop at all.

Second: what does hard-token trajectory SFT do to small students?

It teaches them to act. Qwen3.5-0.8B moved from 1/220 to 44/220 on GPT 5.5 medium rows, and to 46/220 on Qwen3.5-35B-A3B 8-bit rows. Qwen3.5-2B moved from 0/220 to 57/220 and 58/220. LiquidAI/LFM2.5-8B-A1B-MLX-8bit started higher at 9/220, then reached 47/220 with GPT rows and 50/220 with Qwen rows.

The important behavior change is submission. The base Qwen models submitted only once. After SFT, they submitted on most tasks. That means the training transferred the protocol: inspect, query, read, submit. But most remaining failures were wrong submissions, so SQL judgment did not transfer at the same level.

Third: does a same-family Qwen teacher help?

The same-family teacher helped slightly on success. Qwen3.5-0.8B moved from 44 to 46, Qwen3.5-2B from 57 to 58, and LiquidAI/LFM2.5-8B-A1B from 47 to 50. The surprise is the cost. Repeat stops rose from 10 to 54 for Qwen3.5-0.8B, from 9 to 65 for Qwen3.5-2B, and from 27 to 38 for LiquidAI/LFM2.5-8B-A1B.

In that chart, the left side is solved eval tasks, where higher is better. The right side is repeated-action stops, where lower is better. I think of that right side as the loop-control cost. Qwen3.5-35B-A3B 8-bit produced more SFT rows because its successful trajectories were longer. Those rows gave small success gains, but they also transferred a loopier policy.

Fourth: what changed in the harness behavior?

This is the chart that explains the experiment. The base Qwen students are mostly repeat-stop bars; they are not consistently reaching the final answer step. GPT-row SFT turns those bars into mostly submissions. Same-family Qwen rows add a few correct submissions, but also more repeated-action stops.

The cleanest summary is:

SFT changed the students from "does not really operate the harness" into "operates the harness, but often submits wrong SQL."

Exact Numbers Behind The Charts

The charts are the reading path. This table is the audit path. Success, wrong submits, parse stops, repeat stops, and max/runtime stops are mutually exclusive final task outcomes, so they sum to 220. SQL execution errors are diagnostic events inside a run and can overlap with any of those outcomes.

| Strong baseline | Success | Wrong submits | Parse stops | Repeat stops | Max/runtime stops | SQL-error tasks (non-exclusive) | | --- | ---: | ---: | ---: | ---: | ---: | ---: | | Qwen3.5-35B-A3B 8-bit local MLX eval | 96/220 = 43.6% | 106 | 0 | 9 | 9 | 4 | | GPT 5.4 mini hosted baseline | 105/220 = 47.7% | 115 | 0 | 0 | 0 | 4 | | GPT 5.5 medium teacher/baseline | 115/220 = 52.3% | 105 | 0 | 0 | 0 | 1 |

| Student run | Success | Wrong submits | Parse stops | Repeat stops | Max/runtime stops | SQL-error tasks (non-exclusive) | Avg turns | | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | Qwen3.5-0.8B base | 1/220 = 0.5% | 0 | 11 | 208 | 0 | 0 | 2.39 | | Qwen3.5-0.8B SFT on GPT 5.5 rows | 44/220 = 20.0% | 160 | 6 | 10 | 0 | 45 | 2.50 | | Qwen3.5-0.8B SFT on Qwen3.5-35B-A3B 8-bit rows | 46/220 = 20.9% | 109 | 7 | 54 | 4 | 5 | 3.48 | | Qwen3.5-2B base | 0/220 = 0.0% | 1 | 0 | 219 | 0 | 0 | 2.00 | | Qwen3.5-2B SFT on GPT 5.5 rows | 57/220 = 25.9% | 149 | 5 | 9 | 0 | 49 | 2.59 | | Qwen3.5-2B SFT on Qwen3.5-35B-A3B 8-bit rows | 58/220 = 26.4% | 91 | 3 | 65 | 3 | 2 | 3.60 | | LiquidAI/LFM2.5-8B-A1B-MLX-8bit base | 9/220 = 4.1% | 22 | 0 | 52 | 137 | 8 | 1.64 | | LiquidAI/LFM2.5-8B-A1B SFT on GPT 5.5 rows | 47/220 = 21.4% | 139 | 7 | 27 | 0 | 34 | 2.59 | | LiquidAI/LFM2.5-8B-A1B SFT on Qwen3.5-35B-A3B 8-bit rows | 50/220 = 22.7% | 120 | 6 | 38 | 6 | 12 | 3.52 |

Failure Analysis

Success rate alone hides too much for agents. I also looked at turns, generated tokens, prompt estimates, validation loss, and stop reasons.

The student generated-token chart makes the same-family teacher effect visible. Students trained on Qwen3.5-35B-A3B 8-bit rows did not mainly become more verbose per action. Their actions were still short structured JSON. They took more turns. More turns means more generated output per task and more chances to drift into a state the teacher never supervised.

The LiquidAI/LFM2.5-8B-A1B-MLX-8bit base row is approximate because many invalid BAML generations were not saved as clean assistant messages. I used saved output-token counters for invalid generations and saved parsed outputs where available. The pattern is still useful: the base LFM output was raw and verbose, while fine-tuned LFM output was closer to short executable actions.

The teacher and strong-baseline token chart uses its own scale. Qwen3.5-35B-A3B 8-bit often produced shorter action text per call than GPT 5.5 medium, but it took more turns, so its generated output per task ended up higher. GPT 5.4 mini was the best efficiency point among the strong baselines I measured: close to GPT 5.5 medium in task success, with fewer turns and fewer generated tokens per task.

The prompt side is usually larger than the generated side. Each extra turn adds one assistant action, but it also forces the next prompt to resend the system instructions, task, prior actions, schema observations, query results, and structured-output instructions. That is why turns matter twice.

For the initial GPT-row and baseline runs where prompt reconstruction was available, the prompt and total estimates looked like this:

| Run | Prompt/call mean | Prompt/call P90 | Total/task mean | | --- | ---: | ---: | ---: | | Qwen3.5-0.8B base | 1118 | 1993 | 2881 | | Qwen3.5-0.8B SFT, GPT 5.5 medium rows | 1568 | 2545 | 4130 | | Qwen3.5-2B base | 1399 | 2420 | 2878 | | Qwen3.5-2B SFT, GPT 5.5 medium rows | 1611 | 2586 | 4400 | | LiquidAI/LFM2.5-8B-A1B SFT, GPT 5.5 medium rows | 1620 | 2600 | 4412 | | Qwen3.5-35B-A3B 8-bit local eval | 1872 | 2939 | 7000 | | GPT 5.4 mini hosted baseline | 1471 | 2522 | 3354 | | GPT 5.5 medium teacher/baseline | 1590 | 2601 | 4287 |

Validation loss shows the teacher-forcing gap. For Qwen3.5-0.8B, final validation loss improved from 0.3494 on GPT 5.5 rows to 0.2553 on Qwen3.5-35B-A3B 8-bit rows. For Qwen3.5-2B, it improved from 0.2966 to 0.2051. Under teacher-forced SFT, the same-family rows were clearly easier for the Qwen students to predict.

The rollout improvement was much smaller. Same-family rows gave only small success gains and made repeated-action stops worse. Validation loss asks, "Can the model predict the teacher's next action given a teacher-style context?" The harness asks, "Can the model recover from its own previous actions, choose useful SQL probes, and submit SQL that passes hidden tests?" Those are related questions, but they are not the same measurement.

Hardware And Infrastructure Lessons

The Qwen3.5-35B-A3B 8-bit teacher-row runs took longer because they kept more rows, ran more optimizer updates, and had longer median context.

Wall time moved from 15.9m to 22.4m for Qwen3.5-0.8B, from 22.1m to 29.4m for Qwen3.5-2B, and from 57.4m to 71.8m for LiquidAI/LFM2.5-8B-A1B.

The Mac with 128 GB unified memory was useful for capacity-heavy local work: preparing data, generating charts, running notebooks, and serving quantized models. The RTX 3090 CUDA server was better for LoRA training. Training is repeated forward, backward, and optimizer math, where CUDA kernels, tensor cores, FlashAttention, and GPU-local VRAM matter more than total host memory.

This is the RAM versus VRAM lesson I learned the slow way. The Mac can fit things because it has a lot of unified memory. The NVIDIA GPU trains faster because the tensors stay near the compute units and the kernel stack is built for this workload. A 24 GB VRAM GPU can be much faster for adapter training than a 128 GB unified-memory Mac, even though the Mac has more total memory.

LiquidAI/LFM2.5-8B-A1B took the longest because it is a much larger model and used an eager expert-execution path in this training setup. I disabled in-loop validation for the LFM runs because validation compilation was unreliable in that environment. The deterministic 220-task harness eval stayed as the comparison point.

What I Learned

The first lesson is that trajectory SFT can transfer the loop. The base Qwen students mostly failed before they became useful agents. After SFT, they inspected, queried, and submitted far more often. That is real progress.

The second lesson is that the failure simply moved downstream. A wrong submit is better than no submit, but it is still a failed task. The students learned the protocol faster than they learned SQL judgment under their own rollouts.

The third lesson is that successful-only filtering made the first dataset easier to reason about. I probably threw away some locally good actions from failed trajectories, but I avoided teaching from traces whose credit assignment I could not defend.

The fourth lesson is that a same-family teacher can be easier to imitate and still transfer bad habits. Qwen3.5-35B-A3B 8-bit gave slightly better student success and lower validation loss for the Qwen students, but it also transferred more repeated-action behavior.

The fifth lesson is that teacher-forced loss and rollout success measure different things. Validation loss dropped sharply on same-family rows, but the eval gain was tiny and loop control got worse. I learned not to trust offline loss alone as a proxy for agent quality.

Conclusion

Can a 0.8B model learn to act like a stronger model inside a real tool-use loop?

Yes, but only in the specific sense this experiment measured. For the 0.8B student itself, offline teacher-trace hard-token SFT taught the agent protocol: unsloth/Qwen3.5-0.8B went from 1/220 to 44/220 on GPT 5.5 medium teacher rows and 46/220 on Qwen3.5-35B-A3B 8-bit rows.

The larger student runs are comparison points, not the answer to the 0.8B question. unsloth/Qwen3.5-2B reached 57/220 with GPT rows and 58/220 with Qwen rows. LiquidAI/LFM2.5-8B-A1B reached 47/220 and 50/220. Those runs show the same broad pattern: SFT teaches the loop, but it does not close the gap to the stronger teachers.

But the students did not inherit teacher-level SQL judgment. The stronger baselines remained far ahead, and most remaining student failures were wrong SQL submissions or loop-control failures. That is the honest result of this first post: protocol transfer, yes. Teacher-level decision making, no.

The natural next step is to keep the traces and the harness fixed but give the student a richer signal than one hard token per position. That is what the next post does: off-policy top-k soft-label distillation over these same teacher traces.